MCO-1017: MCO-1018 MCO-1019: MCO-1020: MCO-1021: OCPBUGS-32583: OCPBUGS-32679: Implement PinnedImageSets

[hexfusion]

This PR adds the PinnedImageSet feature to MCO behind TechPreview.

This PR adds.

• PinnedImageSet Controller
• PinnedImageSet Manager
• Per node status reporting via MachineConfigNode
• CRI gRPC client implementation

Implements: openshift/enhancements#1481

details

• CRI gRPC client: This PR implements the CRI client which allows for MCD to speak directly to CRI-O.
• PinnedImageSet controller: This controller located in the MCC watches on PinnedImageSets manages the MachineConfigPool spec pinnedImageSets based on observations of the PinnedImageSet CR's. If a PinnedImageSet includes labels matching a pool for example machineconfiguration.openshift.io/role: "worker" the pinnedImageSet will be applied to the worker pool and any custom pools. This works in the exact same way as MachineConfig is applied today.

A custom pool would allow declaring a subset of nodes to pull and pin a targets image set. See docs on custom pools

spec:
  pinnedImageSets:
  - name: worker-pinned-image-set

PinnedImageSet manager: This controller located in the MCD performs the following actions given a list of PinnedImageSets

• the size of all the images in all the sets to conclude is there enough local storage.
• Speaking with CRI-O over the CRI client ask if the image exists locally if it does not add schedule the image to be pulled.
• After all the images are pulled we perform a systemctl reload crio which pinns the images. When an image is pinned it is not able to be deleted by the kubelet due to GC.
• To mitigate the the cost of pulling images on a control-plane node we requires images to be pulled in series. While on worker nodes we by default allow 5 workers to pull images in parallel.

Considerations

Because we are pulling possibly a large number of images there is a concern about how that could affect the control-plane. For this reason only a single worker is deployed on a master node. Each image is pulled serial with a 1s cool down period. But this still results in noticeable I/O. This is a basic idle AWS cluster. While this latency on its own is not an issue under load it should be a consideration. Current proposed mitigations include exposing knobs around concurrency and the throttle duration.

examples

• mcd logs showing reconciliation.

I0416 20:11:40.798505   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 3
I0416 20:11:40.798706   43478 pinned_image_set.go:764] Completed scheduling 24% of images
I0416 20:11:40.798724   43478 pinned_image_set.go:764] Completed scheduling 49% of images
I0416 20:11:40.798734   43478 pinned_image_set.go:764] Completed scheduling 74% of images
I0416 20:11:40.798745   43478 pinned_image_set.go:764] Completed scheduling 98% of images
I0416 20:11:40.882583   43478 file_writers.go:233] Writing file "/etc/crio/crio.conf.d/50-pinned-images"
I0416 20:11:40.916349   43478 update.go:2253] Running: systemctl reload crio

• mcd logs showing an update applied to a pinned image set while reconciling.

I0416 19:52:17.312344   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 1
I0416 19:52:50.517202   43478 pinned_image_set.go:310] Cancelling workload: PinnedImageSet update
I0416 19:52:50.561012   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 2

• example CR

apiVersion: machineconfiguration.openshift.io/v1alpha1
kind: PinnedImageSet
metadata:
  name: test-pinned
  labels:
    machineconfiguration.openshift.io/role: "master"
spec:
  pinnedImages:
   - name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86d26e7ebcccd6f07a75db5b1e56283b25c2ee1c6a755d6ffc5a4d59beb9c504
   - name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e43b2ef4fbc42dbcbea5d67f57f3feed38f6b45fb712c99acb06490103e277a9

• node level status via MachineConfigNode

oc get machineconfignodes ip-10-0-9-106.us-west-1.compute.internal -o json | jq .status.pinnedImageSets
[
  {
    "currentGeneration": 3,
    "desiredGeneration": 3,
    "name": "test-pinned"
  }
]

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[yuqi-zhang]

Some initial questions

[openshift-ci-robot]

@hexfusion: This pull request references MCO-1017 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This PR adds the PinnedImageSet feature to MCO behind TechPreview. The feature includes the following components.

This PR adds.

• PinnedImageSet Controller
• PinnedImageSet Manager
• Per node status reporting via MachineConfigNode
• CRI gRPC client implementation

details

• CRI gRPC client: This PR implements the CRI client which allows for MCD to speak directly to CRI-O.
• PinnedImageSet controller: This controller located in the MCC watches on PinnedImageSets manages the MachineConfigPool spec pinnedImageSets based on observations of the PinnedImageSet CR's. If a PinnedImageSet includes labels matching a pool for example machineconfiguration.openshift.io/role: "worker" the pinnedImageSet will be applied to the worker pool.

spec:
 pinnedImageSets:
 - name: worker-pinned-image-set

PinnedImageSet manager: This controller located in the MCD performs the following actions given a list of PinnedImageSets

• the size of all the images in all the sets to conclude is there enough local storage.
• Speaking with CRI-O over the CRI client ask if the image exists locally if it does not add schedule the image to be pulled.
• After all the images are pulled we perform a systemctl reload crio which pinns the images. When an image is pinned it is not able to be deleted by the kubelet due to GC.
• To mitigate the the cost of pulling images on a control-plane node we requires images to be pulled in series. While on worker nodes we by default allow 5 workers to pull images in parallel.

Considerations

Because we are pulling possibly a large number of images there is a concern about how that could affect the control-plane. For this reason only a single worker is deployed on a master node. Each image is pulled serial with a 1s cool down period. But this still results in noticeable I/O. This is a basic idle AWS cluster. While this latency on its own is not an issue under load it should be a consideration. Current proposed mitigations include exposing knobs around concurrency and the throttle duration.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[hexfusion]

@sergiordlr @rioliu-rh @ptalgulk01 we are in a pretty good place to start testing. Please let me know if you have any questions.

[openshift-ci-robot]

@hexfusion: This pull request references MCO-1017 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This PR adds the PinnedImageSet feature to MCO behind TechPreview. The feature includes the following components.

This PR adds.

• PinnedImageSet Controller
• PinnedImageSet Manager
• Per node status reporting via MachineConfigNode
• CRI gRPC client implementation

details

• CRI gRPC client: This PR implements the CRI client which allows for MCD to speak directly to CRI-O.
• PinnedImageSet controller: This controller located in the MCC watches on PinnedImageSets manages the MachineConfigPool spec pinnedImageSets based on observations of the PinnedImageSet CR's. If a PinnedImageSet includes labels matching a pool for example machineconfiguration.openshift.io/role: "worker" the pinnedImageSet will be applied to the worker pool.

spec:
 pinnedImageSets:
 - name: worker-pinned-image-set

PinnedImageSet manager: This controller located in the MCD performs the following actions given a list of PinnedImageSets

• the size of all the images in all the sets to conclude is there enough local storage.
• Speaking with CRI-O over the CRI client ask if the image exists locally if it does not add schedule the image to be pulled.
• After all the images are pulled we perform a systemctl reload crio which pinns the images. When an image is pinned it is not able to be deleted by the kubelet due to GC.
• To mitigate the the cost of pulling images on a control-plane node we requires images to be pulled in series. While on worker nodes we by default allow 5 workers to pull images in parallel.

Considerations

Because we are pulling possibly a large number of images there is a concern about how that could affect the control-plane. For this reason only a single worker is deployed on a master node. Each image is pulled serial with a 1s cool down period. But this still results in noticeable I/O. This is a basic idle AWS cluster. While this latency on its own is not an issue under load it should be a consideration. Current proposed mitigations include exposing knobs around concurrency and the throttle duration.

examples

• mcd logs showing reconciliation.

I0416 20:11:40.798505   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 3
I0416 20:11:40.798706   43478 pinned_image_set.go:764] Completed scheduling 24% of images
I0416 20:11:40.798724   43478 pinned_image_set.go:764] Completed scheduling 49% of images
I0416 20:11:40.798734   43478 pinned_image_set.go:764] Completed scheduling 74% of images
I0416 20:11:40.798745   43478 pinned_image_set.go:764] Completed scheduling 98% of images
I0416 20:11:40.882583   43478 file_writers.go:233] Writing file "/etc/crio/crio.conf.d/50-pinned-images"
I0416 20:11:40.916349   43478 update.go:2253] Running: systemctl reload crio

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@hexfusion: This pull request references MCO-1017 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This PR adds the PinnedImageSet feature to MCO behind TechPreview. The feature includes the following components.

This PR adds.

• PinnedImageSet Controller
• PinnedImageSet Manager
• Per node status reporting via MachineConfigNode
• CRI gRPC client implementation

details

• CRI gRPC client: This PR implements the CRI client which allows for MCD to speak directly to CRI-O.
• PinnedImageSet controller: This controller located in the MCC watches on PinnedImageSets manages the MachineConfigPool spec pinnedImageSets based on observations of the PinnedImageSet CR's. If a PinnedImageSet includes labels matching a pool for example machineconfiguration.openshift.io/role: "worker" the pinnedImageSet will be applied to the worker pool.

spec:
 pinnedImageSets:
 - name: worker-pinned-image-set

PinnedImageSet manager: This controller located in the MCD performs the following actions given a list of PinnedImageSets

• the size of all the images in all the sets to conclude is there enough local storage.
• Speaking with CRI-O over the CRI client ask if the image exists locally if it does not add schedule the image to be pulled.
• After all the images are pulled we perform a systemctl reload crio which pinns the images. When an image is pinned it is not able to be deleted by the kubelet due to GC.
• To mitigate the the cost of pulling images on a control-plane node we requires images to be pulled in series. While on worker nodes we by default allow 5 workers to pull images in parallel.

Considerations

Because we are pulling possibly a large number of images there is a concern about how that could affect the control-plane. For this reason only a single worker is deployed on a master node. Each image is pulled serial with a 1s cool down period. But this still results in noticeable I/O. This is a basic idle AWS cluster. While this latency on its own is not an issue under load it should be a consideration. Current proposed mitigations include exposing knobs around concurrency and the throttle duration.

examples

• mcd logs showing reconciliation.

I0416 20:11:40.798505   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 3
I0416 20:11:40.798706   43478 pinned_image_set.go:764] Completed scheduling 24% of images
I0416 20:11:40.798724   43478 pinned_image_set.go:764] Completed scheduling 49% of images
I0416 20:11:40.798734   43478 pinned_image_set.go:764] Completed scheduling 74% of images
I0416 20:11:40.798745   43478 pinned_image_set.go:764] Completed scheduling 98% of images
I0416 20:11:40.882583   43478 file_writers.go:233] Writing file "/etc/crio/crio.conf.d/50-pinned-images"
I0416 20:11:40.916349   43478 update.go:2253] Running: systemctl reload crio

• mcd logs showing an update applied to a pinned image set while reconciling.

I0416 19:52:17.312344   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 1
I0416 19:52:50.517202   43478 pinned_image_set.go:310] Cancelling workload: PinnedImageSet update
I0416 19:52:50.561012   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@hexfusion: This pull request references MCO-1017 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This PR adds the PinnedImageSet feature to MCO behind TechPreview. The feature includes the following components.

This PR adds.

• PinnedImageSet Controller
• PinnedImageSet Manager
• Per node status reporting via MachineConfigNode
• CRI gRPC client implementation

details

• CRI gRPC client: This PR implements the CRI client which allows for MCD to speak directly to CRI-O.
• PinnedImageSet controller: This controller located in the MCC watches on PinnedImageSets manages the MachineConfigPool spec pinnedImageSets based on observations of the PinnedImageSet CR's. If a PinnedImageSet includes labels matching a pool for example machineconfiguration.openshift.io/role: "worker" the pinnedImageSet will be applied to the worker pool.

spec:
 pinnedImageSets:
 - name: worker-pinned-image-set

PinnedImageSet manager: This controller located in the MCD performs the following actions given a list of PinnedImageSets

• the size of all the images in all the sets to conclude is there enough local storage.
• Speaking with CRI-O over the CRI client ask if the image exists locally if it does not add schedule the image to be pulled.
• After all the images are pulled we perform a systemctl reload crio which pinns the images. When an image is pinned it is not able to be deleted by the kubelet due to GC.
• To mitigate the the cost of pulling images on a control-plane node we requires images to be pulled in series. While on worker nodes we by default allow 5 workers to pull images in parallel.

Considerations

Because we are pulling possibly a large number of images there is a concern about how that could affect the control-plane. For this reason only a single worker is deployed on a master node. Each image is pulled serial with a 1s cool down period. But this still results in noticeable I/O. This is a basic idle AWS cluster. While this latency on its own is not an issue under load it should be a consideration. Current proposed mitigations include exposing knobs around concurrency and the throttle duration.

examples

• mcd logs showing reconciliation.

I0416 20:11:40.798505   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 3
I0416 20:11:40.798706   43478 pinned_image_set.go:764] Completed scheduling 24% of images
I0416 20:11:40.798724   43478 pinned_image_set.go:764] Completed scheduling 49% of images
I0416 20:11:40.798734   43478 pinned_image_set.go:764] Completed scheduling 74% of images
I0416 20:11:40.798745   43478 pinned_image_set.go:764] Completed scheduling 98% of images
I0416 20:11:40.882583   43478 file_writers.go:233] Writing file "/etc/crio/crio.conf.d/50-pinned-images"
I0416 20:11:40.916349   43478 update.go:2253] Running: systemctl reload crio

• mcd logs showing an update applied to a pinned image set while reconciling.

I0416 19:52:17.312344   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 1
I0416 19:52:50.517202   43478 pinned_image_set.go:310] Cancelling workload: PinnedImageSet update
I0416 19:52:50.561012   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 2

• example CR

apiVersion: machineconfiguration.openshift.io/v1alpha1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "master"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86d26e7ebcccd6f07a75db5b1e56283b25c2ee1c6a755d6ffc5a4d59beb9c504
  - name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e43b2ef4fbc42dbcbea5d67f57f3feed38f6b45fb712c99acb06490103e277a9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sergiordlr]

The PR has been pre-merge tested:
https://issues.redhat.com/browse/ECODEPQE-465
https://issues.redhat.com/browse/MCO-1054

Two issues have been found that will be fixed post-merge:
https://issues.redhat.com/browse/ECODEPQE-466
https://issues.redhat.com/browse/OCPBUGS-33046

We can add the qe-approved label

/label qe-approved

[openshift-ci-robot]

@hexfusion: This pull request references MCO-1017 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This PR adds the PinnedImageSet feature to MCO behind TechPreview.

This PR adds.

• PinnedImageSet Controller
• PinnedImageSet Manager
• Per node status reporting via MachineConfigNode
• CRI gRPC client implementation

Implements: openshift/enhancements#1481

details

• CRI gRPC client: This PR implements the CRI client which allows for MCD to speak directly to CRI-O.
• PinnedImageSet controller: This controller located in the MCC watches on PinnedImageSets manages the MachineConfigPool spec pinnedImageSets based on observations of the PinnedImageSet CR's. If a PinnedImageSet includes labels matching a pool for example machineconfiguration.openshift.io/role: "worker" the pinnedImageSet will be applied to the worker pool and any custom pools. This works in the exact same way as MachineConfig is applied today.

A custom pool would allow declaring a subset of nodes to pull and pin a targets image set. See docs on custom pools

spec:
 pinnedImageSets:
 - name: worker-pinned-image-set

PinnedImageSet manager: This controller located in the MCD performs the following actions given a list of PinnedImageSets

• the size of all the images in all the sets to conclude is there enough local storage.
• Speaking with CRI-O over the CRI client ask if the image exists locally if it does not add schedule the image to be pulled.
• After all the images are pulled we perform a systemctl reload crio which pinns the images. When an image is pinned it is not able to be deleted by the kubelet due to GC.
• To mitigate the the cost of pulling images on a control-plane node we requires images to be pulled in series. While on worker nodes we by default allow 5 workers to pull images in parallel.

Considerations

Because we are pulling possibly a large number of images there is a concern about how that could affect the control-plane. For this reason only a single worker is deployed on a master node. Each image is pulled serial with a 1s cool down period. But this still results in noticeable I/O. This is a basic idle AWS cluster. While this latency on its own is not an issue under load it should be a consideration. Current proposed mitigations include exposing knobs around concurrency and the throttle duration.

examples

• mcd logs showing reconciliation.

I0416 20:11:40.798505   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 3
I0416 20:11:40.798706   43478 pinned_image_set.go:764] Completed scheduling 24% of images
I0416 20:11:40.798724   43478 pinned_image_set.go:764] Completed scheduling 49% of images
I0416 20:11:40.798734   43478 pinned_image_set.go:764] Completed scheduling 74% of images
I0416 20:11:40.798745   43478 pinned_image_set.go:764] Completed scheduling 98% of images
I0416 20:11:40.882583   43478 file_writers.go:233] Writing file "/etc/crio/crio.conf.d/50-pinned-images"
I0416 20:11:40.916349   43478 update.go:2253] Running: systemctl reload crio

• mcd logs showing an update applied to a pinned image set while reconciling.

I0416 19:52:17.312344   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 1
I0416 19:52:50.517202   43478 pinned_image_set.go:310] Cancelling workload: PinnedImageSet update
I0416 19:52:50.561012   43478 pinned_image_set.go:329] Reconciling pinned image set: test-pinned: generation: 2

• example CR

apiVersion: machineconfiguration.openshift.io/v1alpha1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "master"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86d26e7ebcccd6f07a75db5b1e56283b25c2ee1c6a755d6ffc5a4d59beb9c504
  - name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e43b2ef4fbc42dbcbea5d67f57f3feed38f6b45fb712c99acb06490103e277a9

• node level status via MachineConfigNode

oc get machineconfignodes ip-10-0-9-106.us-west-1.compute.internal -o json | jq .status.pinnedImageSets
[
 {
   "currentGeneration": 3,
   "desiredGeneration": 3,
   "name": "test-pinned"
 }
]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[djoshy]

/retest-required

[hexfusion]

https://issues.redhat.com/browse/ECODEPQE-466 is fixed via fd2c36e

[djoshy]

/test e2e-hypershift

Greened on one of my other runs, so let's try it again

[openshift-ci]

@hexfusion: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-openstack	8693696	link	false	/test e2e-openstack
ci/prow/okd-scos-e2e-aws-ovn	6ce48d6	link	false	/test okd-scos-e2e-aws-ovn
ci/prow/e2e-aws-ovn-upgrade-out-of-change	fd2c36e	link	false	/test e2e-aws-ovn-upgrade-out-of-change

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[yuqi-zhang]

/lgtm

CI looks good, thanks for working through this!

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hexfusion, sinnykumari, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sinnykumari,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[yuqi-zhang]

/hold cancel