Ideation for OLMv1 Milestone 3 (operator-controller 0.1.0)

[joelanford]

Hi! Now that we've made lots of progress on Milestone 2's major pushes (release automation and catalog APIs), I think it is time that we begin establishing our next milestone, and perhaps even think about broader areas after that.

Progress Recap

To begin with, let's recap the progress we've made thus far. As of Milestone 2, we will have:

1. The strong foundation of Rukpak that manages the mechanics of actually unpacking, installing, and pivoting bundle contents on a cluster, primarily focused on plain manifest bundles and OLMv0's registry+v1 bundles.
2. An initial MVP of catalogd, which can make file-based catalog data available on a cluster via a Catalog API. (Yes, OLMv1 is saying goodbye to support for sqlite-based catalogs. 👋)
3. A release of operator-controller that defines the simplest possible Operator API with just a spec.package. This initial operator-controller release will integrate with catalogd to pull catalog metadata from cluster-admin defined catalogs, reconcile Operator objects to lookup the highest available semver-versioned bundle to install, and then create and manage a rukpak BundleDeployment for that package.

Thus, we have the stage set for Milestone 3 (likely to also be known as operator-controller 0.1.0).

Milestone 3 Ideas

This list is much too large to include everything. But I think the following ideas are somewhat unblocked and ready to be worked on, so its worth discussing which of these things we think would be most impactful in three key areas:

1. What will make people interested in trying out OLMv1 and giving us feedback?
2. What will help us test a hypothesis to give us information about a future milestone?
3. What will improve upon the foundation we have and make future milestones easier to implement and maintain?

Operator Controller

• Add support for a version constraint in the Operator API, which will allow a cluster admin to limit the possibilities of potential bundle candidates, based on their version.
• Add support for a channel constraint in the Operator API, which will allow a cluster admin to limit the possibilities of potential bundle candidates, based on their channel membership.
• Add support for a catalog source label selector in the Operator API. If not specified, the bundle resolved for this operator can come from any catalog. If it is specified, the bundle resolved for this operator must come from one of the matching Catalog objects.
• If not done already, trigger re-reconciliation of all Operator objects when any of the available catalogs' data has changed.
• Add support for honoring upgrade edges defined in the catalog.

Catalogd

• Add a Rukpak-like spec.source union type to the Catalog API, which would allow catalog filesystems to be sourced in a variety of ways. It would also give catalogd developers an extensible API surface to add new source mechanisms in the future.
• Continue iterating on the catalogd API surface, For example: remove the Package and BundleMetadata CRDs and replace with an aggregated-apiserver that serves a more generic CatalogMetadata API.
• Iterate on the Catalog status, and make it minimally usable for clients to understand the current state of the catalog. Examples include:
  • Were there issues unpacking or parsing the catalog content defined in the spec?
  • What is the hash of the catalog content currently being served?
  • Is the currently served catalog metadata representative of the current Catalog spec (or is it stale/out-of-date?).
    • This could happen if we successfully unpacked and started serving at time 0, and then the spec changed and we fail to unpack/parse the new catalog data. Do we keep serving the old data? Or do we serve nothing?

Rukpak

• Add a kustomize provisioner and/or improve the helm provisioner. These two types may be the sweet spot because they give flexibility to both cluster admins and operator authors (plain manifest bundles and registry+v1 bundles are not very flexible).
• Implement a new bundle source for pulling and unpacking bundles from configMaps in the provisioner's namespace. This would be useful in developer and CI scenarios where building and pushing images is prohibitive. (see improve/rename local source to support multiple configmaps rukpak#611)
• Implement a new bundle source for pulling and unpacking images in-process, as a sibling to the CRI-based image source Rukpak currently supports. This source would be necessary to consume potential OLM OCI bundles and Helm OCI bundles.
• Investigate ideas related to introspecting bundle health and upgrade readiness. This will eventually be necessary information as input into the operator-controller resolver so that it can understand which operators can be upgraded.

Potential demo scripts and coast-to-coast story arcs

Honoring an optional spec.versionRange field in the Operator API

1. Create a Catalog object referencing quay.io/operatorhubio/catalog:latest
2. Show that the catalog status reflects that it was successfully unpacked.
3. Show that the catalogd metadata api reports objects for all packages, channels, and bundles in the catalog.
4. Create an Operator object referencing package prometheus with version <=0.32.0
5. Show that a BundleDeployment gets created and references the bundle image quay.io/operatorhubio/prometheus@sha256:14f75077f01feab351f7a046ccfcff6ad357bed2393048d0f6a41f6e64c63278
6. Edit the Operator object and change the version to <=0.37.0
7. Show that the existing BundleDeployment is updated to reflect a new bundle image reference with quay.io/operatorhubio/prometheus@sha256:3e281e587de3d03011440685fc4fb782672beab044c1ebadc42788ce05a21c35
8. Edit the Operator object and remove the version specification.
9. Show that the existing BundleDeployment is updated to reflect a new bundle image reference with the highest semver bundle defined in the package (as of 4/5/23, that version is 0.47.0): quay.io/operatorhubio/prometheus@sha256:5b04c49d8d3eff6a338b56ec90bdf491d501fe301c9cdfb740e5bff6769a21ed
10. Edit the Operator object and set the version to >=3.0.0.
11. Show the the Operator object has its status updated to reflect that no version is available in the specified range. Also show that the BundleDeployment is still present and remains at version 0.47.0.
12. Delete the Operator and Catalog objects.
13. Show that no catalog metadata or bundle deployment objects remain.

Honoring an optional spec.channel field in the Operator API.

1. Create a Catalog object referencing quay.io/operatorhubio/catalog:latest
2. Show that the catalog status reflects that it was successfully unpacked.
3. Show that the catalogd metadata api reports objects for all packages, channels, and bundles in the catalog.
4. Create an Operator object referencing package strimzi-kafka-operator with channel strimzi-0.32.x
5. Show that a BundleDeployment gets created and references the bundle image quay.io/operatorhubio/strimzi-kafka-operator@sha256:0d4ac0ed10d815b30dca6521d3ceab5dfc150cddda22e9ca83e97462dbcfeff4
6. Edit the Operator object and change the channel to strimzi-0.33.x
7. Show that the existing BundleDeployment is updated to reflect a new bundle image reference with quay.io/operatorhubio/strimzi-kafka-operator@sha256:4d953d5fd719ea61a7009daaa7e37c948073132f231c2d2d1b07bf3d0d5d6e3b
8. Edit the Operator object and remove the channel specification.
9. Show that the existing BundleDeployment is updated to reflect a new bundle image reference with the highest semver bundle defined in the package (as of 4/5/23, that version is 0.34.0): quay.io/operatorhubio/strimzi-kafka-operator@sha256:42a3dce711492fae22f2d919299a0594f152396f6c2fef65ffc47226ba9c786e
10. Edit the Operator object and set the channel to foobar.
11. Show the the Operator object has its status updated to reflect that no version is available in the specified channel. Also show that the BundleDeployment is still present and remains at version 0.34.0.
12. Delete the Operator and Catalog objects.
13. Show that no catalog metadata or bundle deployment objects remain.

Detecting catalog changes and re-reconciling Operator objects

1. Create a Catalog object referencing quay.io/joelanford/example-operator-index:0.1.0
2. Show that the catalog status reflects that it was successfully unpacked.
3. Show that the catalogd metadata api reports objects for all packages, channels, and bundles in the catalog.
4. Create an Operator object referencing package example-operator
5. Show that a BundleDeployment gets created and references the bundle image quay.io/joelanford/example-operator-bundle:0.1.0
6. Update the Catalog object to reference a new image: quay.io/joelanford/example-operator-index:0.2.0
7. Show that the existing BundleDeployment is updated to reflect a new bundle image reference with quay.io/joelanford/example-operator-bundle:0.2.0
8. Delete the Operator and Catalog objects.
9. Show that no catalog metadata or bundle deployment objects remain.

Sourcing catalog content from a ConfigMap

1. Create this ConfigMap:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: example-operator-index
     namespace: catalogd-system
   data:
     catalog.json: |
       {"defaultChannel":"stable","name":"example-operator","schema":"olm.package"}
       {"entries":[{"name":"example-operator.v0.1.0"}],"name":"stable","package":"example-operator","schema":"olm.channel"}
       {"image":"quay.io/joelanford/example-operator-bundle:0.1.0","name":"example-operator.v0.1.0","package":"example-operator","properties":[{"type":"olm.gvk","value":{"group":"example.com","kind":"App","version":"v1"}}, 
       {"type":"olm.package","value":{"packageName":"example-operator","version":"0.1.0"}}],"relatedImages":[{"image":"quay.io/joelanford/example-operator-bundle:0.1.0","name":""},{"image":"quay.io/joelanford/example-operator@sha256:a5f8fce740945fcdabf7f8ac6fcd6f0f95b3a828fbf17e43de251de9d5544118","name":""}],"schema":"olm.bundle"}

2. Create this Catalog:

   ---
   apiVersion: operatorframework.io/v1alpha1
   kind: Catalog
   metadata:
     name: example-operator-index
   spec:
     source:
       type: configMaps
       configMaps:
       - configMap:
           name: example-operator-index
         path: configs

3. Show that the catalog status reflects that it was successfully unpacked.
4. Show that the catalogd metadata api reports objects for all packages, channels, and bundles in the catalog.
5. Create an Operator object referencing package example-operator
6. Show that a BundleDeployment gets created and references the bundle image quay.io/joelanford/example-operator-bundle:0.1.0
7. Delete the Operator and Catalog objects.
8. Show that no catalog metadata or bundle deployment objects remain.

Reporting failures in the Catalog object.

1. Create a Catalog object referencing image quay.io/foobar/foobar:latest
2. Show that the catalog object's status reports a failure sourcing the catalog content.

Reporting the hash of the Catalog content in the catalog's status

1. Create a Catalog object referencing quay.io/joelanford/example-operator-index:0.1.0
2. Show that the catalog status contains a hash that represents contents of the catalog.
3. Update the catalog object to reference quay.io/joelanford/example-operator-index:0.2.0
4. Show that the catalog status contains a different hash for the new contents of the catalog.
5. Update the catalog object to reference quay.io/joelanford/example-operator-index:0.1.0
6. Show that the catalog status contains the same hash that was seen in step 2.

Use a catalogd-defined catalog to install an Operator.

0. Install operator-controller and its dependencies (but nothing else).
1. Create a Catalog object referencing quay.io/operatorhubio/catalog:latest
2. Show that the catalog status reflects that it was successfully unpacked.
3. Show that the catalogd metadata api reports objects for all packages, channels, and bundles in the catalog.
4. Create an Operator object referencing package strimzi-kafka-operator.
5. Show that aBundleDeployment is created to reflect a bundle image reference with the highest semver bundle defined in the package (as of 4/5/23, that version is 0.34.0): quay.io/operatorhubio/strimzi-kafka-operator@sha256:42a3dce711492fae22f2d919299a0594f152396f6c2fef65ffc47226ba9c786e
6. Delete the Operator and Catalog objects.
7. Show that no catalog metadata or bundle deployment objects remain.

[joelanford]

Another idea in the operator-controller area:

1. Add support for a catalog source label selector in the Operator API. If not specified, the bundle resolved for this operator can come from any catalog. If it is specified, the bundle resolved for this operator must come from one of the matching Catalog objects.

EDIT: I added this to the initial idea description.

[everettraven]

Taking a look at this list I am +1 for everything on there.

Since it was mentioned that it's likely not everything on this list would get done I propose that we limit the catalogd work to:

Continue iterating on the catalogd API surface, For example: remove the Package and BundleMetadata CRDs and replace with an aggregated-apiserver that serves a more generic CatalogMetadata API.

Iterate on the Catalog status, and make it minimally usable for clients to understand the current state of the catalog. Examples include:

• Were there issues unpacking or parsing the catalog content defined in the spec?
• What is the hash of the catalog content currently being served?
• Is the currently served catalog metadata representative of the current Catalog spec (or is it stale/out-of-date?).
  • This could happen if we successfully unpacked and started serving at time 0, and then the spec changed and we fail to unpack/parse the new catalog data. Do we keep serving the old data? Or do we serve nothing?

I think these items would still allow all the things in the operator-controller section to get done and we can make the catalogd variable source functionality a stretch goal.

As a bit of an aside, is there a particular set of functional/behavioral requirements from the OLMv1 PRD that we are targeting/progressing towards with this milestone? I don't think we need to base milestones strictly on these requirements but it could be helpful to visualize the overall progress we are making towards the OLMv1 requirements.

[tmshort]

When reconciling operators after catalog changes, one thing the UX document had was to list available versions on the Operator CR status. That might be an additional step before doing an upgrade (i.e. detecting the changes in the catalog correctly based on the currently installed version/spec'd version).

[bparees]

Update the Catalog object to reference a new image: quay.io/joelanford/example-operator-index:0.2.0
Show that the existing BundleDeployment is updated to reflect a new bundle image reference with quay.io/joelanford/example-operator-bundle:0.2.0

i'd like to think through this scenario a bit more.... this seems to imply an automatic upgrade, but i think elsewhere we've said we don't want to default users into automatic upgrades.

So is this just a point in time behavior until we flesh out the upgrade policy stories?

(I guess this question also applies to your other scenarios where you specify, and then modify, a version or channel on the operator resource)

[joelanford]

I think this is actually a behavior that we have today already. If new catalog data shows up and contains a higher semver bundle in an existing installed package and you bounce the operator-controller pod or touch an Operator CR, I'm fairly sure the bundle deployment will be updated with the new bundle image.

This particular scenario is mainly about triggering a reconciliation when inputs change, not about changing auto-upgrade behavior.

In general, the constraint scenarios are meant to keep the existing resolver/operator-controller behavior except that the set of bundles that are potential selections are filtered based on the new Operator spec fields.

[joelanford]

Lots of slack conversation here on this subject: https://kubernetes.slack.com/archives/C0181L6JYQ2/p1680723779244959

TL;DR: we're worried about users unknowningly creating operator CRs that will auto-upgrade. A few ideas were tossed around related to annotations, admission webhooks, and/or using status fields to create a UX where auto-upgrade is not the default.

If a user creates this Operator, the desire is to install an initial version, but then to pin to that version to avoid auto-upgrades.

apiVersion: operatorframework.io/v1alpha1
kind: Operator
metadata:
  name: foo
spec:
  package: foo

[joelanford]

And I should say, on this topic, I think we should NOT include this discussion in the scope of Milestone 3/v0.1.0. For this immediate milestone, I think the focus should be on:

• using the current resolver/controller implementation but feeding more constraints into the resolver to limit the possibilities of what ends up being a satisfiable resolution result.
• making sure that the controller is triggered and resolution re-runs when inputs change.

Separately, and in a later milestone, we should focus on the UX around opt-in or opt-out of auto-upgrades and what the right API would be to make the concept and behavior intuitive.

[anik120]

+1 to the last comment from catalogd's perspective too. Content update on poll of a newer image is going to be a very stretch goal for milestone 3

[anik120]

Thus, we have the stage set for Milestone 3 (likely to also be known as operator-controller 0.1.0).

Why 0.1.0 and not 0.0.1?

Continue iterating on the catalogd API surface, For example: remove the Package and BundleMetadata CRDs and replace with an aggregated-apiserver that serves a more generic CatalogMetadata API.

From what I understand so far, the CatalogMetadata API being talked about here is a different flavor of Package/BundleMetadata APIs. But, this issue is probably not the right place to get into the details of that. So worth calling out that more details TBD?

For catalogd, +1 on setting up the variable sourcing feature and targeting configmaps for the first variable for milestone 3. We should probably call out unit/e2e tests too for milestone 3.

But looking at the overall picture, is it just me, or this feels like a LOT to get done in milestone 3? Especially with the stuff about rukpak added, and assuming that length of milestone 3 will be similar to milestone. Stuff for operator-controller and catalogd seems doable, but feels like rukpak should probably be thinned down a bit, or agree that this milestone will be for a longer duration.

[anik120]

Last comment is also with the assumption that we'll have the same amount of resources dedicated to milestone 3 that we had for milestone 2

[joelanford]

Why 0.1.0 and not 0.0.1?

I think we already have a v0.0.1. I don't think it matters too much, but my thinking was that a regular rhythm of 0.y minor releases is what we need to get used to in the long run. At some point in the future (don't know when), we'll decide we want to do a patch release, and we'll just be able to. But for now, I would say every release is a new minor and we don't backport any bug fixes, so we'd start out with 0.1.0, 0.2.0, 0.3.0, etc.

For catalogd, +1 on setting up the variable sourcing feature and targeting configmaps for the first variable for milestone 3.

There's way more this than we can likely do in a single sprint (APIs, validation logic, webhooks, etc.), and all of that could happen before even introducing a second source implementation. It think to keep scope small, I'd say we just introduce the API structure, a single image source (we already have catalog images in the wild we can use), and then have some minimal validation, but only in the context of reconciliation (i.e. no webhooks or CRD validations). Then in later iterations, we can introduce admission-time validation and other source types.

We should probably call out unit/e2e tests too for milestone 3.

I don't think the milestone is the place to call that level of stuff out. But by all means, let's capture some of that as tasks based on what we decide to do.

But looking at the overall picture, is it just me, or this feels like a LOT to get done in milestone 3

Easy to miss, but I touch on this in the description:

This list is much too large to include everything. But I think the following ideas are somewhat unblocked and ready to be worked on, so its worth discussing which of these things we think would be most impactful in three key areas:

• What will make people interested in trying out OLMv1 and giving us feedback?
• What will help us test a hypothesis to give us information about a future milestone?
• What will improve upon the foundation we have and make future milestones easier to implement and maintain?

[tmshort]

Why 0.1.0 and not 0.0.1?

Per semantic versioning (ignoring that the first component is 0), bumping the middle component indicates new functionality, the last component indicates a bugfix. We aren't really bugfixing, but are adding new functionality. So, 0.1.0, 0.2.0, etc, is appropriate.

[anik120]

There's way more this than we can likely do in a single sprint (APIs, validation logic, webhooks, etc.), and all of that could happen before even introducing a second source implementation. It think to keep scope small, I'd say we just introduce the API structure, a single image source (we already have catalog images in the wild we can use), and then have some minimal validation, but only in the context of reconciliation (i.e. no webhooks or CRD validations). Then in later iterations, we can introduce admission-time validation and other source types.

Looks like the original write up is missing the mention of how long the milestone is being scoped out for, but with that information (one sprint), this makes sense to me.

[varshaprasad96]

+1 to the list of items mentioned here. Would just want to point out a few other tasks which may probably need some attention.

1. [Operator Controller] Uninstall scenario:
   How do the dependent APIs behave when Operator object is uninstalled from the cluster. For example, Operator X depends on Operator Y. And Operator Z depends on Operator Y. When X and Z are uninstalled, how does cleanup happen for Y.

2. [Deppy] Allowing operator resolver to accept multiple Entity Sources:
   This would require us to tweak the current implementation, to have the resolver accept multiple entitySources, in turn modifying deppySolver to accept multiple variable sources for resolution. This will enable us to handle multiple constraints separately, but make deppy consider all of them during resolution - which means we can pass in bundle and package level constraints independently. (I'm not sure if this has already been worked on, probably needs more discussion).

3. [Deppy] Operator Resolution pipeline. This is for designing the resolution pipeline.

Imo:

• (3) is not an immediate necessity for M3.
• (2) There are workarounds for this. But is definitely needed.
• (1) Seems equally important. Not sure if having it in M3 would be necessary.

[joelanford]

My two cents:

1. [Operator Controller] Uninstall scenario:

There is no concept yet of dependencies, as far as I know. So in your scenario, a user would have created X, Y, and Z. Then, if X and Z are uninstalled, nothing happens to Y. It just keeps running.

Also, to drive the point home, the user could also install X and Z without installing Y. X and Z may crash loop, but operator controller would (currently) not care or do anything about it.

2. [Deppy] Allowing operator resolver to accept multiple Entity Sources:

Seems like we may need to solve this as part of the last few steps of milestone 2's catalog integration work? Would another viable approach be to implement a composite entity source type that could present multiple entity sources to deppy as if it were one? Same for the variable source?

3. [Deppy] Operator Resolution pipeline.

I think I see that as a means to an end. Correct me if I'm wrong, but this is a design improvement that could make it easier for us to maintain resolver inputs and manipulations over time. But it isn't a directly observable value add to the overall OLMv1, right?

So IMO, this could totally be deemed a prerequisite to accomplish one of the Milestone 3 goals, but I don't think I'd call this out as a separate M3 goal on its own. WDYT?

[varshaprasad96]

Seems like we may need to solve this as part of the last few steps of milestone 2's catalog integration work?

Yes, this is kind of an implementation work, not a user facing goal at least from Operator controller's perspective. I could see it more useful for independent deppy users, but that's not to be considered here. We probably just need an issue in M3 (considering M2 is closed) to investigate this and work on a viable solution.

Would another viable approach be to implement a composite entity source type that could present multiple entity sources to deppy as if it were one? Same for the variable source?

A similar hack we are doing right now. Getting package level information independently, constructing an entity out of it and adding to an existing variable source.

I think I see that as a means to an end. Correct me if I'm wrong, but this is a design improvement that could make it easier for us to maintain resolver inputs and manipulations over time. But it isn't a directly observable value add to the overall OLMv1, right?

Well, yeah. It's again under the hood, not a user facing goal. I'm not sure if this would exactly be considered as a pre-requisite for M3, because we would still have things working without it. But IMO this would probably take ample amount of time - definitely more than 3 weeks (@perdasilva could add more on this) to brainstorm the implementation, test it out and verify if that's actually what we want. Which is why I think of this as a separate component which may need some more attention over the sprints. But since it's not urgent, we could skip it for M3 and come back to it later?

[joelanford]

Another interesting proposal is to add a way in the operator API to specify a bundle directly. We haven't discussed this in much detail yet, so we don't think we're ready to pull in any implementation work, but perhaps we work on a spike to help us define this better as part of the scope of M3.

[tmshort]

We could get that now with rukpak, no? So, I'd rather we take the time to think about it before committing to something at the higher level API.

[joelanford]

Yesterday, the community met and agreed on the scope of Milestone 3 (v0.1.0). It is captured in the following issues:

• Use a catalogd-defined catalog to install an Operator #161
• Honoring an optional spec.version field in the Operator API #162

Both of these define a demo script that we will execute and post a video of. Once these are posted, we will close these issues, and cut a new release for operator-controller v0.1.0, thus marking the completion of the milestone.

Ideas discussed here that did not make it into the scope of the milestone will be once again open for discussion for the next operator-controller release.