training-operator cannot be upgraded from 1.7/stable to recent version

[DnPlas]

Bug Description

Due to #161, the latest training-operator charm will only have one container in the scheduled Pod, which corresponds to the container running the charm, as the workload will be deployed and scheduled separately.
Because of https://bugs.launchpad.net/juju/+bug/1991955, upgrading the charm from 1.7/stable to the most recent version, the Pod will still have two containers instead of one after running juju refresh training-operator --channel latest/edge.

To Reproduce

1. Deploy juju training-operator --channel 1.7/stable --trust
2. Refresh juju refresh training-operator --channel latest/edge/pr-162 (this works right now) or juju refresh training-operator --channel latest/edge (this will only work once refactor, chore: refactor charm to use Deployment for workload, also bumps training-operator 1.7->1.8 #167 is merged)
3. Observe

Environment

microk8s 1.29/stable
juju 3.5.0

Relevant Log Output

# Getting the pods after a refresh shows the  training-operator-0 Pod has 2/2
# which has the charm and the workload container
# Then because of the refactor, we have another Pod training-operator-7f97689fcf-2zshp
# with the workload as well

$ kubectl get pods -A
testing                               training-operator-0                      2/2     Running   0          10m
testing                               training-operator-7f97689fcf-2zshp       1/1     Running   0          9m59s

Additional Context

This issue is currently affecting the test_upgrade integration test case and the upgrade path.

Workarounds

So far, the only workaround that has worked is to remove the operator and re-deploy it.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5901.

This message was autogenerated