MCO-585: MCO-569: MCO-563: MCO-586: Introduces BuildController

[cheesesashimi]

- What I did
This introduces the concept of the BuildController for handling on-cluster layering OS builds. This provides a state machine for determining whether a given MachineConfigPool has opted in to layered OS builds by use of the label machineconfiguration.openshift.io/layering-enabled. Included with this PR are two interchangeable image builder backends. These provide a common interface to the BuildController to allow building a layered OS image in one of two ways: 1) Makes use of the OpenShift Image Builder API. 2) Makes use of a custom build pod for building and pushing the containers.

The way the build process works is this:

1. The on-cluster-build-config ConfigMap is used to identify which push / pull secrets to use and where to push the final OS image to. So this ConfigMap must exist; though there is some experimental work going on in [DO NOT MERGE] - integrates BuildController and machine-os-builder stub #3765 to provide some sensible defaults.
2. Once a pool has opted-in, the BuildController identifies that a build is required.
3. A Dockerfile containing the base OS image (from the machine-config-osimageurl ConfigMap) is rendered from a template and stored in a ConfigMap specific to this build.
4. The rendered MachineConfig is pulled from the API server, compressed and base64-encoded, and stored in a ConfigMAp specific to this build.
5. Depending on whether the a custom build pod or an OpenShift Image Build is used, both of the aforementioned artifacts are injected into the build process.
6. The Dockerfile is crafted in such a way that the MachineConfig is decoded and extracted, then fed into Ignition. Ignition reads the config portion of the MachineConfig and performs the necessary operations to have the desired effect.
7. As the custom build pod or OpenShift Image Build begins the build, the MachineConfigPool status is updated in lockstep with the status of the build pod or OpenShift Image Build.
8. Upon success, the MachineConfigPool gets an annotation machineconfiguration.openshift.io/newestImageEquivalentConfig which contains the new layered OS image pullspec. The aforementioned Dockerfile and MachineConfig ConfigMaps are deleted upon a successful build.

OpenShift Image Builder:

• This is fairly self-contained. It creates an OpenShift Build object which knows how to create a build pod, inject the ConfigMaps, and knows where to push the container when the build is completed.

Custom Pod Builder:

• This creates a build pod which uses quay.io/buildah/stable:latest to perform the build / push process.
• Since we need the SHA256 digest produced by the build process, we use the --digestfile option on Buildah to write that value to the filesystem someplace.
• Then, we use the oc CLI contained within the base OS image to create a new ConfigMap with this digest.
• The PodBuildController reads the ConfigMap and extracts the SHA256 from it.

- How to verify it

This PR includes a unit test suite for the new BuildController which also covers the two interchangeable image builders. Running the unit tests will exercise all of the major code paths therein. In this current form, once merged, this PR will not become part of the MCO's execution code path, meaning that this code will not yet be built nor included in the e2e test suites.

- Description for the changelog
Introduces BuildController

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[cheesesashimi]

/test unit
/test verify

[cheesesashimi]

/test unit
/test verify

[cheesesashimi]

/test unit
/test verify

[cheesesashimi]

/test unit
/test verify

[openshift-ci-robot]

@cheesesashimi: This pull request references MCO-565 which is a valid jira issue.

In response to this:

- What I did
This introduces the concept of a common interface across Image Builder backends.

- How to verify it

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@cheesesashimi: This pull request references MCO-585 which is a valid jira issue.

In response to this:

- What I did
This introduces the concept of a common interface across Image Builder backends.

- How to verify it

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@cheesesashimi: This pull request references MCO-585 which is a valid jira issue.

In response to this:

- What I did
This introduces the concept of the BuildController for handling on-cluster layering OS builds. This provides a state machine for determining whether a given MachineConfigPool has opted in to layered OS builds by use of the label machineconfiguration.openshift.io/layering-enabled. Included with this PR are two interchangeable image builder backends. These provide a common interface to the BuildController to allow building a layered OS image in one of two ways: 1) Makes use of the OpenShift Image Builder API. 2) Makes use of a custom build pod for building and pushing the containers.

- How to verify it

This PR includes a unit test suite for the new BuildController which also covers the two interchangeable image builders. Running the unit tests will exercise all of the major code paths therein. In this current form, once merged, this PR will not become part of the MCO's execution code path, meaning that this code will not yet be built nor included in the e2e test suites.

- Description for the changelog
Introduces BuildController

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@cheesesashimi: This pull request references MCO-585 which is a valid jira issue.

In response to this:

- What I did
This introduces the concept of the BuildController for handling on-cluster layering OS builds. This provides a state machine for determining whether a given MachineConfigPool has opted in to layered OS builds by use of the label machineconfiguration.openshift.io/layering-enabled. Included with this PR are two interchangeable image builder backends. These provide a common interface to the BuildController to allow building a layered OS image in one of two ways: 1) Makes use of the OpenShift Image Builder API. 2) Makes use of a custom build pod for building and pushing the containers.

The way the build process works is this:

1. The on-cluster-build-config ConfigMap is used to identify which push / pull secrets to use and where to push the final OS image to. So this ConfigMap must exist; though there is some experimental work going on in [DO NOT MERGE] - integrates BuildController and machine-os-builder stub #3765 to provide some sensible defaults.
2. Once a pool has opted-in, the BuildController identifies that a build is required.
3. A Dockerfile containing the base OS image (from the machine-config-osimageurl ConfigMap) is rendered from a template and stored in a ConfigMap specific to this build.
4. The rendered MachineConfig is pulled from the API server, compressed and base64-encoded, and stored in a ConfigMAp specific to this build.
5. Depending on whether the a custom build pod or an OpenShift Image Build is used, both of the aforementioned artifacts are injected into the build process.
6. The Dockerfile is crafted in such a way that the MachineConfig is decoded and extracted, then fed into Ignition. Ignition reads the config portion of the MachineConfig and performs the necessary operations to have the desired effect.
7. As the custom build pod or OpenShift Image Build begins the build, the MachineConfigPool status is updated in lockstep with the status of the build pod or OpenShift Image Build.
8. Upon success, the MachineConfigPool gets an annotation machineconfiguration.openshift.io/newestImageEquivalentConfig which contains the new layered OS image pullspec. The aforementioned Dockerfile and MachineConfig ConfigMaps are deleted upon a successful build.

OpenShift Image Builder:

• This is fairly self-contained. It creates an OpenShift Build object which knows how to create a build pod, inject the ConfigMaps, and knows where to push the container when the build is completed.

Custom Pod Builder:

• This creates a build pod which uses quay.io/buildah/stable:latest to perform the build / push process.
• Upon completion of the build, it uses quay.io/skopeo/stable:latest to get the SHA256 for where the image was pushed to. At the time I wrote this code, I was unaware of the --digestfile option contained within Buildah, so some future work can be done to make this a bit better.

- How to verify it

This PR includes a unit test suite for the new BuildController which also covers the two interchangeable image builders. Running the unit tests will exercise all of the major code paths therein. In this current form, once merged, this PR will not become part of the MCO's execution code path, meaning that this code will not yet be built nor included in the e2e test suites.

- Description for the changelog
Introduces BuildController

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[cheesesashimi]

/test e2e-gcp-op
/test okd-scos-e2e-aws-ovn
/test e2e-aws-ovn

[cheesesashimi]

/test e2e-gcp-op
/test okd-scos-e2e-aws-ovn
/test e2e-aws-ovn

[cheesesashimi]

/test unit
/test verify

[cheesesashimi]

PR is rebased now.

[cheesesashimi]

/test verify
/test unit
/test bootstrap-unit
/test images

[cheesesashimi]

/retest-required

[cheesesashimi]

Fixed lint issue.

[jkyros]

Those test failures look like flakes. We've had green runs until this point and the latest changes wouldn't have affected anything. I say we go for it and let the bot have at it.
/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheesesashimi, jkyros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cheesesashimi,jkyros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jkyros]

/retest-required

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD df96614 and 2 for PR HEAD 48ac23e in total

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 6694a28 and 1 for PR HEAD 48ac23e in total

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD ee08d44 and 0 for PR HEAD 48ac23e in total

[openshift-ci]

@cheesesashimi: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	48ac23e	link	false	/test okd-scos-e2e-aws-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-ci-robot]

/hold

Revision 48ac23e was retested 3 times: holding

[jkyros]

/test e2e-hypershift

[jkyros]

/hold cancel

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD e91b573 and 2 for PR HEAD 48ac23e in total