-
Notifications
You must be signed in to change notification settings - Fork 476
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
MCO: admin defined reboot policy enhancement
- Loading branch information
1 parent
e06505a
commit 38f527e
Showing
1 changed file
with
264 additions
and
0 deletions.
There are no files selected for viewing
264 changes: 264 additions & 0 deletions
264
enhancements/machine-config/admin-defined-reboot-policy.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,264 @@ | ||
| --- | ||
| title: admin-defined-reboot-policy | ||
| authors: | ||
| - "@yuqi-zhang" | ||
| reviewers: | ||
| - TBD | ||
| approvers: | ||
| - TBD | ||
| api-approvers: | ||
| - TBD | ||
| creation-date: yyyy-mm-dd | ||
| last-updated: yyyy-mm-dd | ||
| tracking-link: | ||
| - TBD | ||
| see-also: | ||
| - https://issues.redhat.com/browse/OCPSTRAT-380 | ||
| - https://issues.redhat.com/browse/MCO-507 | ||
| replaces: | ||
| superseded-by: | ||
| --- | ||
|
|
||
| # Admin Defined Reboot Policy | ||
|
|
||
| ## Summary | ||
|
|
||
| This enhancement outlines an API/mechanism for users to define what actions to take upon a MachineConfigOperator driven change (e.g. a file change via a MachineConfig object). By default, all changes to MachineConfig fields require a drain and reboot. The user can use this new API, a RebootPolicy, to specify which MachineConfig changes to not disrupt their workloads. | ||
|
|
||
| ## Motivation | ||
|
|
||
| Historically, workload disruption (drains and reboots) can be very costly for OCP users, especially in bare-metal environments where the stopping and restarting of a node can take up to an hour. For small incremental changes to the host OS, many of them can take effect with a service reload or no action at all, and thus the ability to perform a non-disruptive update can be very beneficial to many users without any downside. | ||
|
|
||
| The MCO since OCP 4.7 has added the ability to perform disruptionless updates on select changes, see: [MCO doc on disruptionless updates](https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfigDaemon.md#rebootless-updates). This list however is by no means exhaustive, and it is not feasible for the MCO to hard-code all use cases for all users into the MCO. In addition, different users may have different expectations on what will cause a reboot or not. Thus there is a need for a user-defined, MCO operated list of "reboot policies" on what to do after an OS update has been applied. | ||
|
|
||
| ### User Stories | ||
|
|
||
| * As an Openshift cluster administrator, I have an environment with thousands of ICSP/IDMS mirror entries. These entries will be added and deleted constantly on a weekly basis. I understand the potential dangers of removing mirrors (can cause an image to no longer be accessible if no other mirror hosts it), but I accept that risk and would like my changes to apply without disrupting my workloads and only by reloading the crio service. | ||
|
|
||
| * As an application developer, I have certificates for my applications I need on-disk, which can be rotated safely and hot-loaded into the applications without any additional action. I do not wish to disrupt other applications when these update via the MCO. | ||
|
|
||
| ### Goals | ||
|
|
||
| * Create a new RebootPolicy CRD to allow users to define how OS updates are actioned on after application | ||
| * Have users be able to define no-action and service reloads to specific MachineConfig changes | ||
| * Have users be able to easily see existing cluster non-disruptive update cases | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| * Have RebootPolicy apply to non-MCO driven changes (e.g. SRIOV can still reboot nodes) | ||
| * Remove existing non-disruptive update paths (the user will be able to override cluster defaults) | ||
| * Design for image-based updates (live apply and bootc, will be considered in the future) | ||
| * Have the MCO validate whether a change can be successfully applied with the given RebootPolicy (i.e. it is up to the responsibility of the user to ensure the correctness of their defined actions) | ||
|
|
||
| ## Proposal | ||
|
|
||
| Create a new rebootpolicy.machineconfiguration.openshift.io CRD, allowing specify user-provided rebootpolicies, and view current version cluster defaults, as follows: | ||
|
|
||
| ```console | ||
| apiVersion: machineconfiguration.openshift.io/v1 | ||
| kind: RebootPolicy | ||
| metadata: | ||
| name: cluster | ||
| spec: | ||
| userPolicies: | ||
| - type: file | ||
| value: /etc/my-conf | ||
| Actions: | ||
| - type: drain | ||
| - type: reload | ||
| target: myservice | ||
| - type: reload | ||
| target: myotherservice | ||
| - type: kernelArgument | ||
| value: myarg | ||
| actions: | ||
| - type: none | ||
| - type: service | ||
| value: myservice | ||
| actions: | ||
| - type: restart | ||
| target: myservice | ||
| clusterDefaultPolicies: | ||
| - type: file | ||
| value: /etc/mco/internal-registry-pull-secret.json | ||
| actions: | ||
| - type: none | ||
| - type: file | ||
| value: /var/lib/kubelet/config.json | ||
| actions: | ||
| - type: none | ||
| - type: file | ||
| value: /etc/machine-config-daemon/no-reboot/containers-gpg.pub | ||
| actions: | ||
| - type: reload | ||
| target: crio.service | ||
| - type: file | ||
| value: /etc/containers/policy.json | ||
| actions: | ||
| - type: reload | ||
| target: crio.service | ||
| - type: file | ||
| value: /etc/containers/registries.conf | ||
| actions: | ||
| - type: special | ||
| ``` | ||
|
|
||
| Which, when applied, will define actions for all future changes to these MachineConfig fields based on the diff of the update (rendered MachineConfigs). | ||
|
|
||
| For this initial implementation we will support MachineConfig changes to: | ||
| - Files | ||
| - Units | ||
| - kernelArguments | ||
| - sshKeys | ||
| And actions for: | ||
| - None | ||
| - Draining a node | ||
| - Reloading a service | ||
| - Daemon-reload and restarting of a service | ||
| - Reboot (default) | ||
| - Special (see workflow description) | ||
|
|
||
| In the future, we can extend this in two ways: | ||
| Supporting more changes/actions (e.g. user defined script to run after a certain change) | ||
| Supporting diffing between ostree images (for image based updates) and bootc managed systems (for bifrost) | ||
|
|
||
|
|
||
| ### Workflow Description | ||
|
|
||
| An admin of an OCP cluster can at any time create/update/delete RebootPolicy objects to reconfigure how they wish updates to be applied. There isn’t any immediate effect so long as the RebootPolicy CR itself is a valid object. | ||
|
|
||
| The MachineConfigController’s drain controller will process any update to the object and validate the RebootPolicy for duplication, action validity (but not correctness), etc. If the object is valid, the drain controller will update the object’s status and observedGeneration to indicate success, and if not, update those and degrade the controller, example: | ||
|
|
||
| ``` | ||
| status: | ||
| conditions: | ||
| - lastTransitionTime: "xxx" | ||
| Message: “” | ||
| status: "False" | ||
| type: RebootPolicyValidationFailing | ||
| - lastTransitionTime: "xxx" | ||
| Message: “Validation of the rebootpolicy has succeed.” | ||
| status: "True" | ||
| type: RebootPolicyValidated | ||
| observedGeneration: x | ||
| ``` | ||
|
|
||
| The MachineConfigDaemon will process and apply the current RebootPolicy if it’s valid, indicated by generation == observedGeneration (MCC has processed the latest spec) and RebootPolicyValidated == true/RebootPolicyValidationFailing == false. If not, the daemon will also degrade (and queue a re-sync) waiting for the controller to validate the object. | ||
|
|
||
| For now, we only support one cluster-wide rebootpolicy, and as such any rebootpolicy not named “cluster” will be ignored and the MCO will report errors via degrading the operator. | ||
|
|
||
| When the user first enables RebootPolicy via featuregate (or at some version, by default), the “cluster” RebootPolicy object will be generated with the existing MachineConfigDaemon defaults today, so the user can also view the default policies that came with the cluster. The user can update these fields if they wish to use a different workflow via userPolicies, but are not allowed to modify clusterDefaultPolicies, which is reserved for MCO to update these as we wish in the future. The fields that exist today are as highlighted above. | ||
|
|
||
| Note that /etc/containers/registries.conf has a “Special” keyword. This is due to the extra processing the MCO does today. No user defined policy may use the “Special” keyword. | ||
| User-provided changes will overwrite the cluster defaults | ||
|
|
||
| The special MCO cert paths ("/etc/kubernetes/kubelet-ca.crt", "/etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem") are not listed in the clusterDefaultPolicies as they are no longer MachineConfig changes | ||
|
|
||
| In terms of the policy itself, in case a change has multiple matching policies (e.g. two files changed, both with a different policy), all policies will be applied in order (e.g. reload multiple services). If any change in the list does not match any defined policy, the MCO will default to rebooting and skip over all the policies. | ||
|
|
||
| The MachineConfigNodes CR will track the status of the upgrade, and will also be updated to indicate what the RebootPolicy used for the update was. | ||
|
|
||
| RebootPolicies are only processed in-cluster and cannot be used to skip the bootstrap pivot during cluster installation. | ||
|
|
||
| #### Variation and form factor considerations [optional] | ||
|
|
||
| This will work by default on standalone OCP and any other OCP variants that run the MCO in its entirety. (e.g. works for RHEL) | ||
|
|
||
| HyperShift deployments do not run the MCO pods directly and will not have RebootPolicy objects. | ||
|
|
||
| ### API Extensions | ||
|
|
||
| - Adds the RebootPolicy CRD (only under MCO namespace and only affects MCO) | ||
|
|
||
| ### Implementation Details/Notes/Constraints [optional] | ||
|
|
||
| This aims more to be a user-facing API to essentially map MachineConfig fields to what action should be taken on them, which aligns with existing MCO functionality to perform a hard-coded list of these policies. However, as we evolve the MCO to fit in with an image based OS update vision, we would need this to work with: | ||
| - Existing MachineConfig based entries | ||
| - User-provided base-image upgrade cases (live-apply) | ||
| - A more unified update interface with bootc | ||
|
|
||
| Which we can definitely translate under-the-hood to adapt to the system, but file/unit specific entries like this may not work as effectively when it comes to image-to-image updates, especially at the daemon level. | ||
|
|
||
| As there is a need for this in 4.16, we will proceed with the format that works best with the existing MCO architecture, while leaving it open to change for the future. | ||
|
|
||
| #### Hypershift [optional] | ||
|
|
||
| As noted above, Hypershift cannot directly tap into this. In a bit more detail, Hypershift has 2 modes of operation for nodepools: | ||
| Replace upgrades - since all machines are deleted and recreated in this model, there is no method of live-application today | ||
| Inplace upgrades - these actually use a special form of the daemon, and we can leverage that to update and perform necessary RebootPolicies. However we probably do not want hosted cluster admins to have direct control over this, much like MachineConfigs, so if we do want to add this we will need a new API in Hypershift to handle this. As such, we are not considering this at this time | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| As noted in the non-goal sections, any RebootPolicy is not verified in the MCO, minus formatting. This means that if you accidentally define a change that needed a reboot to take effect, the cluster would “finish the update” without being able to use the new configuration. | ||
|
|
||
| The user thereby needs to understand that no additional verification is taking place, and that they accept any responsibility for RebootPolicies causing configuration not to take effect. | ||
|
|
||
| Given the popularity of using pause on MachineConfigPools, it is possible some users will exploit this to skip upgrade reboots. That will likely not break the cluster but will put the configuration in an awkward spot, as we can no longer guarantee that the configuration a node claims to be on is actually applied. | ||
|
|
||
| The MCO rebootless apply and reboot apply actually does have 1 major difference still: the application of proper selinux labels. The MCO doesn’t currently have any ability to properly label files it writes, and some users have worked around this by relabelling via a service, which would trigger during the early boot process. With this, we will break that workflow unless the MCO explicitly has the ability to handle selinux labels. | ||
|
|
||
| ### Drawbacks | ||
|
|
||
| There shouldn’t be many drawbacks (this is a pretty lightweight sub-feature that we already in-part ship today) other than what was outlined in Risks and Mitigations. | ||
|
|
||
| ## Design Details | ||
|
|
||
| ### Open Questions [optional] | ||
|
|
||
| The biggest question is the format and constraints we would like for the UX to be. Conceivably, we have the options to: | ||
|
|
||
| - Have the user specify a global policy | ||
| - Allow users to wildcard paths | ||
| - Allow users to define policies per pool instead of globally | ||
| - Provide users more flexibility in defining the actions (e.g. provide a shell script to run to change selinux context) | ||
| - Provide users more granularity on changes within a file (e.g. adding vs deleting image mirrors within registries.conf) | ||
|
|
||
| This current outline is a bit more restrictive in UX but hopefully a bit more clearly defined. | ||
|
|
||
| The question of image based updates is also important, but discussed in the above Implementation Details section. | ||
|
|
||
| Selinux labels are discussed in risks and mitigations. | ||
|
|
||
| How we would like to handle certificat | ||
|
|
||
| ### Test Plan | ||
|
|
||
| The MCO e2e tests will cover functionality for checking if the policy has taken effect. | ||
|
|
||
| ### Graduation Criteria | ||
|
|
||
| We will add this behind a featuregate during the 4.16 timeframe. This at most only needs a tech preview phase, and then can be GA’ed since the feature is relatively isolated. If needed, we can likely ship as a feature directly in 4.16 and only featuregate during development. | ||
|
|
||
| #### Removing a deprecated feature | ||
|
|
||
|
|
||
|
|
||
| ### Upgrade / Downgrade Strategy | ||
|
|
||
|
|
||
|
|
||
| ### Version Skew Strategy | ||
|
|
||
|
|
||
|
|
||
| ### Operational Aspects of API Extensions | ||
|
|
||
|
|
||
| #### Failure Modes | ||
|
|
||
|
|
||
|
|
||
| #### Support Procedures | ||
|
|
||
|
|
||
| ## Implementation History | ||
|
|
||
|
|
||
| ## Alternatives | ||
|
|
||
| We could ship a general flag for 4.16 that allows users to do as they wish for any update, and implement a better method of this after image-based MCO is implemented. This might not be a very safe design, though. | ||
|
|
||
| ## Infrastructure Needed [optional] | ||
|
|
||
|
|
||
|
|