CRDS stack and workspace.auto are too long

[robinlioret]

What happened?

When creating the CRDs, I have the following messages:

Error from server (Invalid): error when creating "deploy/yaml/install.yaml": CustomResourceDefinition.apiextensions.k8s.io "stacks.pulumi.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "deploy/yaml/install.yaml": CustomResourceDefinition.apiextensions.k8s.io "workspaces.auto.pulumi.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Example

I'll be using a kind cluster

1. cd
2. git clone https://github.com/pulumi/pulumi-kubernetes-operator.git && cd pulumi-kubernetes-operator
3. git checkout v2.0.0-beta.2
4. kind create cluster --name pulumi-operator-test
5. kubectl config set-context kind-pulumi-operator-test
6. Apply the CRDs kubectl apply -f deploy/crds
   Output:

customresourcedefinition.apiextensions.k8s.io/updates.auto.pulumi.com created
customresourcedefinition.apiextensions.k8s.io/programs.pulumi.com created
Error from server (Invalid): error when creating "deploy/crds/auto.pulumi.com_workspaces.yaml": CustomResourceDefinition.apiextensions.k8s.io "workspaces.auto.pulumi.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
Error from server (Invalid): error when creating "deploy/crds/pulumi.com_stacks.yaml": CustomResourceDefinition.apiextensions.k8s.io "stacks.pulumi.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Clean up

• kind delete cluster --name pulumi-operator-test
• cd && rm -rf ./pulumi-kubernetes-operator

Output of pulumi about

Not relevant

Additional context

Notes

• Same issue when applying the helm chart
• Same issue when applying the deploy/yaml/install.yaml file
• I tried to go around with Argocd and a custom helm chart containing the install.yaml file. Not working either

Workaround

The better way I found to install the operator was to use kubectl create -f https://raw.githubusercontent.com/pulumi/pulumi-kubernetes-operator/refs/heads/master/deploy/yaml/install.yaml

This is dangerous : if the chart is updated during a job, the job can potentially be deleted as well.

With ArgoCD. The only way I found is to enable the Replace sync option. But this has the same drawback as the create / replace kubectl command.

Solution "ideas"

Separate the problematic CRDs into multiple resources. I don't know there is a clean way to extend the limitation of the metadata (maybe with OPA ?) or if there is a way to skip completely the last-configuration-applied when applying.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[blampe]

@robinlioret did you previously install the v1 operator on this cluster? And if so did you follow the instructions to remove the v1 installation?

[robinlioret]

No, I installed the crds from the tag v2.0.0-beta.2 in the repo. I followed the exact steps I described in the initial message (with the additional git chekout missing step.)

I edited the initial message to add this missing step.

You should be able to reproduce the bug with the following script:

cd
git clone https://github.com/pulumi/pulumi-kubernetes-operator.git
cd pulumi-kubernetes-operator
git checkout v2.0.0-beta.2
kind create cluster --name pulumi-operator-test
kubectl config set-context kind-pulumi-operator-test
kubectl apply -f deploy/crds

But it works if you replace the last apply by create. The issue is with the Kubernetes limitation of the size of one metada.annotation field.

EDIT:
I have been able to automatically deploy CRDs with ArgoCD by adding the annotation argocd.argoproj.io/sync-options: Replace=true to the CustomResourcesDefinitions. Confirming the issue is the annotation kubectl.kubernetes.io/last-applied-configuration.

However, this remains a critical issue since it prevent any update (destroy then create) of the CRDs: when a CRD is destroyed, all the belonging resources are destroyed with it (see links below). So updating the CRDs will automatically cause deletion of the resources of the Pulumi Kubernetes Operator : program, stacks, provider config, ...

With Argocd or Flux, the blast radius may be reduces because of the self-healing capabilities. But, the issue remains and may triggers deletion of the resources managed by Pulumi.

I'm looking for a solution to extend the length limit of the annotations

Links:

• CRDs destruction: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#delete-a-customresourcedefinition
• Argocd sync option: https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#replace-resource-instead-of-applying-changes

[EronWright]

Another workaround to consider is to use server-side apply, e.g. kubectl apply --server-side -f deploy/crds. This approach avoids the use of the kubectl.kubernetes.io/last-applied-configuration annotation that is causing the "too long" problem.

To address this ticket, we'll add some clarification to the installation instructions.

[robinlioret]

Good point, it worked properly. Thanks !