Ideation for Milestone 4 (0.2.0)

[joelanford]

Hi! Now that we've made lots of progress on Milestone 3's major pushes (.spec.version, .spec.channel, and catalogd integration), it is time again that we begin establishing our next milestone.

Progress Recap

To begin with, let's recap the progress we've made thus far. As of Milestone 3, we will have:

1. Added the spec.version field in the Operator API, which enables cluster admins to optionally contrain resolution by pinning the operator to a specific version.
2. Added the spec.channel field in the Operator API, which enables cluster admins to optionally constrain resolution to select versions in a specific channel.
3. Added an integration with catalogd, which is planned as OLMv1's new project for making FBC catalogs available on a cluster.
4. As part of (3), revert our temporary support for (and also dependency on) OLMv0 and it's CatalogSource API.

Thus, we have the stage set for operator-controller 0.2.0 (Milestone 4).

Milestone 4 Ideas

This list is much too large to include everything. But I think the following ideas are somewhat unblocked and ready to be worked on, so its worth discussing which of these things we think would be most impactful in three key areas:

1. What will make people interested in trying out OLMv1 and giving us feedback?
2. What will help us test a hypothesis to give us information about a future milestone?
3. What will improve upon the foundation we have and make future milestones easier to implement and maintain?

Operator Controller

• Improve the contents of the Operator status struct. Possibilities include:
  • Adding status.installedVersion and status.resolvedVersion.
  • Changing our condition types to include Installed and Resolved.
  • Removing or re-defining the meaning of the existing Ready condition.
• Trigger re-reconciliation of all Operator objects when any of the available catalogs' data has changed.
• Add support for a catalog source label selector in the Operator API. If not specified, the bundle resolved for this operator can come from any catalog. If it is specified, the bundle resolved for this operator must come from one of the matching Catalog objects.
• Add support for honoring upgrade edges defined in the catalog.
• Think about ways to define upgrade policy. Today the implicit policy is "as soon as possible" and "automatically".

Catalogd

• Add a Rukpak-like spec.source union type to the Catalog API, which would allow catalog filesystems to be sourced in a variety of ways. It would also give catalogd developers an extensible API surface to add new source mechanisms in the future.
• Continue iterating on the catalogd API surface, For example: remove the Package and BundleMetadata CRDs and replace with an aggregated-apiserver that serves a more generic CatalogMetadata API.
• Iterate on the Catalog status, and make it minimally usable for clients to understand the current state of the catalog. Examples include:
  • Were there issues unpacking or parsing the catalog content defined in the spec?
  • What is the hash of the catalog content currently being served?
  • Is the currently served catalog metadata representative of the current Catalog spec (or is it stale/out-of-date?).
    • This could happen if we successfully unpacked and started serving at time 0, and then the spec changed and we fail to unpack/parse the new catalog data. Do we keep serving the old data? Or do we serve nothing?

Rukpak

• Add a kustomize provisioner and/or improve the helm provisioner. These two types may be the sweet spot because they give flexibility to both cluster admins and operator authors (plain manifest bundles and registry+v1 bundles are not very flexible).
• Implement a new bundle source for pulling and unpacking images in-process, as a sibling to the CRI-based image source Rukpak currently supports. This source would be necessary to consume potential OLM OCI bundles and Helm OCI bundles.
• Investigate ideas related to introspecting bundle health and upgrade readiness. This will eventually be necessary information as input into the operator-controller resolver so that it can understand which operators can be upgraded.

Tenancy

• Begin brainstorming various possibilities and tradeoffs tenancy features and approaches for OLMv1 operators, which are installed no more than once per cluster.

Potential demo scripts and coast-to-coast story arcs

Detecting catalog changes and re-reconciling Operator objects

1. Create a Catalog object referencing quay.io/joelanford/example-operator-index:0.1.0
2. Show that the catalog status reflects that it was successfully unpacked.
3. Show that the catalogd metadata api reports objects for all packages, channels, and bundles in the catalog.
4. Create an Operator object referencing package example-operator
5. Show that a BundleDeployment gets created and references the bundle image quay.io/joelanford/example-operator-bundle:0.1.0
6. Update the Catalog object to reference a new image: quay.io/joelanford/example-operator-index:0.2.0
7. Show that the existing BundleDeployment is updated to reflect a new bundle image reference with quay.io/joelanford/example-operator-bundle:0.2.0
8. Delete the Operator and Catalog objects.
9. Show that no catalog metadata or bundle deployment objects remain.

Sourcing catalog content from a ConfigMap

1. Create this ConfigMap:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: example-operator-index
     namespace: catalogd-system
   data:
     catalog.json: |
       {"defaultChannel":"stable","name":"example-operator","schema":"olm.package"}
       {"entries":[{"name":"example-operator.v0.1.0"}],"name":"stable","package":"example-operator","schema":"olm.channel"}
       {"image":"quay.io/joelanford/example-operator-bundle:0.1.0","name":"example-operator.v0.1.0","package":"example-operator","properties":[{"type":"olm.gvk","value":{"group":"example.com","kind":"App","version":"v1"}},{"type":"olm.package","value":{"packageName":"example-operator","version":"0.1.0"}}],"relatedImages":[{"image":"quay.io/joelanford/example-operator-bundle:0.1.0","name":""},{"image":"quay.io/joelanford/example-operator@sha256:a5f8fce740945fcdabf7f8ac6fcd6f0f95b3a828fbf17e43de251de9d5544118","name":""}],"schema":"olm.bundle"}

2. Create this Catalog:

   ---
   apiVersion: operatorframework.io/v1alpha1
   kind: Catalog
   metadata:
     name: example-operator-index
   spec:
     source:
       type: configMaps
       configMaps:
       - configMap:
           name: example-operator-index
         path: configs

3. Show that the catalog status reflects that it was successfully unpacked.
4. Show that the catalogd metadata api reports objects for all packages, channels, and bundles in the catalog.
5. Create an Operator object referencing package example-operator
6. Show that a BundleDeployment gets created and references the bundle image quay.io/joelanford/example-operator-bundle:0.1.0
7. Delete the Operator and Catalog objects.
8. Show that no catalog metadata or bundle deployment objects remain.

Reporting failures in the Catalog object.

1. Create a Catalog object referencing image quay.io/foobar/foobar:latest
2. Show that the catalog object's status reports a failure sourcing the catalog content.

Reporting the hash of the Catalog content in the catalog's status

1. Create a Catalog object referencing quay.io/joelanford/example-operator-index:0.1.0
2. Show that the catalog status contains a hash that represents contents of the catalog.
3. Update the catalog object to reference quay.io/joelanford/example-operator-index:0.2.0
4. Show that the catalog status contains a different hash for the new contents of the catalog.
5. Update the catalog object to reference quay.io/joelanford/example-operator-index:0.1.0
6. Show that the catalog status contains the same hash that was seen in step 2.

[tmshort]

+1 to more Condition types

[everettraven]

Add a Rukpak-like spec.source union type to the Catalog API, which would allow catalog filesystems to be sourced in a variety of ways. It would also give catalogd developers an extensible API surface to add new source mechanisms in the future.

I think building on top of this, we should individually call out the sources we think would be useful sooner rather than later. I think this could look something like:

• Add an image source to catalogd so that catalog content can be sourced from OCI images
• Add a configmap source to catalogd so that catalog content can be source from a ConfigMap
• Add a http source to catalogd so that catalog content can be sourced from a web server
• Add a git source to catalogd so that catalog content can be sourced from a git repository

I think the image source should be coupled with the implementation of the source union type so we don't have any loss of functionality. I also don't think we need to have all of these other types implemented in milestone 4 but I do think it would be nice to have the configmap source implemented to help with making testing of the catalogd integration a bit easier and not rely on having access to an image registry to get catalog contents.

---

Overall for catalogd I think the following is reasonable for milestone 4:

• Add a Rukpak-like spec.source union type to the Catalog API, which would allow catalog filesystems to be sourced in a variety of ways. It would also give catalogd developers an extensible API surface to add new source mechanisms in the future.
  • Add an image source to catalogd so that catalog content can be sourced from OCI images
  • Add a configmap source to catalogd so that catalog content can be sourced from OCI images
• Continue iterating on the catalogd API surface, For example: remove the Package and BundleMetadata CRDs and replace with an aggregated-apiserver that serves a more generic CatalogMetadata API.
• [stretch] Iterate on the Catalog status, and make it minimally usable for clients to understand the current state of the catalog. Examples include:
  • Were there issues unpacking or parsing the catalog content defined in the spec?
  • What is the hash of the catalog content currently being served?
  • Is the currently served catalog metadata representative of the current Catalog spec (or is it stale/out-of-date?). This could happen if we successfully unpacked and started serving at time 0, and then the spec changed and we fail to unpack/parse the new catalog data. Do we keep serving the old data? Or do we serve nothing?

IMO one of the coast to coasts should be a repeat of #161 - This flow should still work even if we change the underlying sourcing information. The other above coast to coast of "Sourcing Catalog contents from a ConfigMap" is another good one.

An additional coast to coast to show the catalogd api changes could be:

1. Install operator-controller from the main branch
2. Create a Catalog object referencing quay.io/operatorhubio/catalog:latest
3. Show that the catalog status reflects that it was successfully unpacked.
4. Show available packages
5. Show channels available for the strimzi-kafka-operator
6. Show bundles available for the strimzi-kafka-operator
7. Create an Operator object referencing the strimzi-kafka-operator package
8. Show that a BundleDeployment is created to reflect a bundle image reference with the highest semver bundle defined in the package (as of 4/5/23, that version is 0.34.0): quay.io/operatorhubio/strimzi-kafka-operator@sha256:42a3dce711492fae22f2d919299a0594f152396f6c2fef65ffc47226ba9c786e
9. Delete the Operator and Catalog objects.
10. Show that no catalog metadata or bundle deployment objects remain.

The distinct difference here is explicit steps showing how we get the same package information with the new catalogd api in comparison to how we got it with the previous apis

[tmshort]

Never too late to think about security. Although I don't believe there are any non-k8s API interfaces (i.e. GRPC/HTTPS) now. We should consider compatibility with OpenShift (e.g. Service CA certificates) and cert-manager at a minimum.

[tmshort]

Metrics and prometheus integration. Even if the only metric we support is the number of operators installed/not-installed.

[awgreene]

Deppy

• A cli implementation of Deppy. The library could read the inputs from a YAML file and produce the expected outputs.
  -- A stretch goal could include using a KUBECONFIG to automatically generate the inputs from a running cluster.
• POC Per's idea to turn Deppy into a configurable pipeline.

Operator Controller

• Expand operator controller's test framework to include a in-directory FBC that is used for testing purposes.

RukPak

• It would be great to understand the performance costs of watching all resources created by RukPak. I would like to create a discussion on the github.com/operator-framework/rukpak channel concerning its performance. The test could focus on the memory and cpu footprint of RukPak when managing 1, 10, and 100 bundleDeployments. The data from this test could be used to justify the current implementation or the expose the need to consider a new design, possibly one in which bundleDeployments are only reconciled at some configurable time-interval.
• Introduce prometheus metrics to RukPak.

[varshaprasad96]

For Deppy, I would like to propose a few action items which may eventually affect how we do resolution in Operator Controller. That would (though not directly) influence the CLI design:

1. Including arbitrary constraints for resolution. This would enable users to provide a constraint through Operator controller (similar to this). This also addresses one of the user stories (related to extensions) in PRD.

2. Including runtime constraints. Currently, deppy does not consider the dependencies which are already installed in the cluster during resolution. This means, for example:

• Operator A needs to be installed.
• Operator A depends on Operator B v1.2.0
• Operator B depends on Operator C v1.3.0.
  If Operator C v1.4.0 is available on cluster and is not compatible with Operator B, then we need to error out.

This will influence our CLI design in a way where if we pass the config of a running cluster, it should be able to query the APIs installed and consider it while providing the resolution. IMO this feature needs to be done before moving to CLI.

3. Refactor Deppy codebase:

• Make deppy easier to consume by removing Entity and EntitySources.
• Introduce new variable sources that contain constraints specific for OLM. (need more info on this)

4. Deppy Pipeline (Introduced by @perdasilva).

Having all of this in a single milestone would be difficult. (1) and (2) should be important. (3) depends on whether we have bandwidth. Also (3) is required for (4)

[joelanford]

We had a discussion and have settled on the following milestone goals. As an outcome of this, we'll write up:

1. high level issues that capture the main goals
2. break those high level issues down into smaller chunks that can be assigned out
3. pull items from (1) and (2) into the 0.2.0 release as they are assigned.

As always, these release plans are fungible. If we finish ahead of schedule we may pull more in. If we get to our planned release date and haven't finished everything, we may punt unfinished items to a follow-on release or mark an item as a blocker and hold the release.

If anyone from the community sees an unassigned issue that they would like to work on, please comment on the issue and start a discussion with the maintainers in the kubernetes slack's #olm-dev channel, and we'll help you get started!

Catalogd

1. Continue iterating on the catalogd API surface, For example: remove the Package and BundleMetadata CRDs and replace with an aggregated-apiserver that serves a more generic CatalogMetadata API. Bootstrap unit/integration/e2e tests as necessary.

2. Add a Rukpak-like spec.source union type to the Catalog API, which would allow catalog filesystems to be sourced in a variety of ways. It would also give catalogd developers an extensible API surface to add new source mechanisms in the future.

   • Add an image source to catalogd so that catalog content can be sourced from OCI images
   • Add a configmap source to catalogd so that catalog content can be sourced from OCI images

   Bootstrap unit/integration/e2e tests as necessary.

Deppy

1. A cli implementation of Deppy. The library could read the inputs from a data structure parsed from a YAML file and produce the expected outputs.
   • [stretch] include using a KUBECONFIG to automatically generate the inputs from a running cluster.
2. POC Per's idea to turn Deppy into a configurable pipeline.
3. Remove EntitySource API, add VariableSource API.
   • Also refactor the catalogd integration to handle this.

Operator Controller

1. Improve the contents of the Operator status struct. Possibilities include:

   • Adding status.installedVersion and status.resolvedVersion.
   • Changing our condition types to include Installed and Resolved.
   • Removing or re-defining the meaning of the existing Ready condition.

   See Improve Operator API Status #183 (I think this should be split up)

2. [spike] Confirm that prometheus server is exposed, standard controller-runtime metrics are present.

3. Trigger re-reconciliation of all Operator objects when any of the available catalogs' data has changed.

4. [stretch] Add support for honoring upgrade edges defined in the catalog.

5. [stretch] Think about ways to define upgrade policy. Today the implicit policy is "as soon as possible" and "automatically".