1.8 upgrades: Juju Error on refresh of Podspec charm with deployment in metadata to Sidecar

[NohaIhab]

Bug Description

When doing an upgrade of Charmed Kubeflow 1.7 to 1.8, some charms cannot be refreshed with Juju due to the error:

ERROR Juju on containers does not support updating deployment info for services.
The new charm's metadata contains updated deployment info.
You'll need to deploy a new charm rather than upgrading if you need this change.
 not supported (not supported)

This applies to charms that were Podspec in 1.7 and had a deployment field specified in the metadata.yaml, then were rewritten to Sidecar in 1.8.

Currently, the only way to upgrade those charms is to remove and redeploy them, this is problematic because it'll cause loss of user data (CRDs are removed, hence CRs created by users are removed).

Affected charms

• jupyter-controller
• argo-controller
• kfp-persistence
• kfp-schedwf
• kfp-viewer

Proposed solution:

Modify the affected charms in their corresponding 1.7 branches to never remove CRDs on remove, and in the upgrade guide do:

1. refresh the charm to the new 1.7 revision by simply doing juju refresh
2. refresh to 1.8 channel (currently we will test with latest/edge since we still haven't released to 1.8 channels)

To Reproduce

juju deploy jupyter-controller --channel=1.7/stable
juju refresh jupyter-controller --channel=latest/edge --trust

Environment

juju 3.1/stable
microk8s 1.25-strict/stable

[misohu]

Did some digging about the technical solution. There is a problem. These charms are implemented in PodSpec and neither of them implements the on remove hook. The thing is that the crds are removed automatically even without us implementing it. Steps to reproduce:

juju deploy kfp-schedwf --channel=2.0/stable
# check crds scheduledworkflows.kubeflow.org is there 
juju remove-application kfp-schedwf
# check crds scheduledworkflows.kubeflow.org is gone

I am not sure if there is a way for us to override this default pod spec behavior.

I was also thinking about to use lightkube to recreate the crd on remove hook for the charms. But this is not possible as the lightkube needs --trust which cannot be used in podspec charms.

[kimwnasptd]

@misohu the goal is to not let the CRD get a DELETE request at all. Even if we recreate the CRD, all user CRs will be lost since the CRD was deleted initially.

Is there no way we can control which resources get deleted when the PodSpec charm gets deleted?

[NohaIhab]

Final Workaround

since juju uses annotations and labels to cleanup when removing a charm, we can remove these annotations and labels from CRDs to prevent the deletion of user resources.

The upgrade would go as follows:

1. make sure all your workflows and notebooks are completed before running the upgrade
2. remove labels and annotations of CRDs, for example for notebooks:

kubectl annotate crd notebooks.kubeflow.org controller.juju.is/id-
kubectl annotate crd notebooks.kubeflow.org model.juju.is/id-
kubectl label crd notebooks.kubeflow.org app.juju.is/created-by-
kubectl label crd notebooks.kubeflow.org app.kubernetes.io/managed-by-
kubectl label crd notebooks.kubeflow.org app.kubernetes.io/name-
kubectl label crd notebooks.kubeflow.org model.juju.is/name-

this step will be done for the following CRDs:

• notebooks.kubeflow.org
• workflows.argoproj.io
• scheduledworkflows.kubeflow.org

3. remove the old charm
4. deploy the new charm

this way the CRDs will not get deleted, therefore the CRs will not get deleted when the charm is removed.

[kimwnasptd]

Closing this as it's a juju bug, and we had a workaround

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5152.

This message was autogenerated