OCPBUGS-34586: Auto-recover from MC with invalid extension

[isabella-janssen]

Fixes: OCPBUGS-34586

- What I did
Add extension validation to controller's machine config validation function to catch the extension error before the daemon attempts to start updating nodes.

- How to verify it
Example MC with an invalid extension 1-worker-extension.yaml:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 80-worker-extensions
spec:
  config:
    ignition:
      version: 3.2.0
  extensions:
    - badName

Auto-recovery by deleting MC

1. Apply MC

$ oc apply -f 1-worker-extension.yaml

2. Wait for MCP to be in degraded state

$ oc get mcp
     NAME     CONFIG                 UPDATED   UPDATING   DEGRADED 
     worker   rendered-worker-XXXX   True      False      True

3. Delete MC

$ oc delete mc/80-worker-extensions

4. MCP should no longer be in a degraded state & nodes should update to new rendered config.

$ oc get mcp
     NAME     CONFIG                 UPDATED   UPDATING   DEGRADED 
     worker   rendered-worker-XXXX   True      False      False

Auto-recovery by reapplying MC with valid extension

1. Apply MC with invalid extension
2. Wait for MCP to be in degraded state
3. Update the MC to have a valid extension (example: change badName to usbguard) & apply it
4. MCP should no longer be in a degraded state & nodes should update to new rendered config with valid extension
5. Check that new extension was installed on the node

$ oc debug node/ip-10-0-91-65.us-west-2.compute.internal
     sh-5.1# chroot /host
     sh-5.1# rpm -q usbguard
     usbguard-1.0.0-15.el9.aarch64

- Description for the changelog
OCPBUGS-34586: Auto-recover from MC with invalid extension

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@isabella-janssen: This pull request references Jira Issue OCPBUGS-34586, which is invalid:

• expected the bug to target the "4.19.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Fixes: OCPBUGS-34586

- What I did

- How to verify it
Example MC with an invalid extension 1-worker-extension.yaml:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
 labels:
   machineconfiguration.openshift.io/role: worker
 name: 80-worker-extensions
spec:
 config:
   ignition:
     version: 3.2.0
 extensions:
   - badName

Auto-recovery by deleting MC

1. Apply MC

$ oc apply -f 1-worker-extension.yaml

2. Wait for MCP to be degraded state

$ oc get mcp
    NAME     CONFIG                  UPDATED UPDATING DEGRADED 
    worker   rendered-worker-XXXX   True      False      True

3. Delete MC

$ oc delete mc/80-worker-extensions

4. MCP should no longer be in a degraded state & nodes should update to new rendered config.

- Description for the changelog
OCPBUGS-34586: Auto-recover from MC with invalid extension

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: isabella-janssen
Once this PR has been reviewed and has the lgtm label, please assign rishabhsaini for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[isabella-janssen]

/jira refresh

[openshift-ci-robot]

@isabella-janssen: This pull request references Jira Issue OCPBUGS-34586, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@isabella-janssen: This pull request references Jira Issue OCPBUGS-34586, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.0) matches configured target version for branch (4.19.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

Fixes: OCPBUGS-34586

- What I did
Add extension validation to controller's machine config validation function to catch the extension error before the daemon attempts to start updating nodes.

- How to verify it
Example MC with an invalid extension 1-worker-extension.yaml:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
 labels:
   machineconfiguration.openshift.io/role: worker
 name: 80-worker-extensions
spec:
 config:
   ignition:
     version: 3.2.0
 extensions:
   - badName

Auto-recovery by deleting MC

1. Apply MC

$ oc apply -f 1-worker-extension.yaml

2. Wait for MCP to be degraded state

$ oc get mcp
    NAME     CONFIG                 UPDATED   UPDATING   DEGRADED 
    worker   rendered-worker-XXXX   True      False      True

3. Delete MC

$ oc delete mc/80-worker-extensions

4. MCP should no longer be in a degraded state & nodes should update to new rendered config.

$ oc get mcp
    NAME     CONFIG                 UPDATED   UPDATING   DEGRADED 
    worker   rendered-worker-XXXX   True      False      False

- Description for the changelog
OCPBUGS-34586: Auto-recover from MC with invalid extension

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[isabella-janssen]

/test all

[openshift-ci]

@isabella-janssen: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-ovn-upgrade-out-of-change	6a53bc9	link	false	/test e2e-azure-ovn-upgrade-out-of-change
ci/prow/e2e-gcp-op-ocl	6a53bc9	link	false	/test e2e-gcp-op-ocl

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[isabella-janssen]

/retest-required