[Cadence] Use of helm install --wait causes the jobs to not be created

[drewhemm]

Describe the bug
When installing this chart with the --wait flag, because of the post-install hook on the resources in cadence/templates/server-job.yaml, they do not get created until Helm considers the app installed, but because the pods will not come online until the jobs have run, there is a circular dependency.

Steps to reproduce the issue:
Attempt to install the chart using helm install --wait or helm upgrade --install --wait

Expected behavior
The chart should install correctly, waiting for all resources to be created

[drewhemm]

Here is some output from running helm with --debug that shows the circular dependency very clearly:

client.go:108: [debug] creating 190 resource(s)
wait.go:53: [debug] beginning wait for 190 resources with timeout of 30m0s
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready
wait.go:225: [debug] Deployment is not ready: default/test-cadence-frontend. 0 out of 1 expected pods are ready

Helm is going to wait for these pods to be ready, and they never will be because the jobs have not run. Here is how the debug output looks when the --wait flag is absent:

client.go:108: [debug] creating 197 resource(s)
client.go:108: [debug] creating 1 resource(s)
client.go:463: [debug] Watching for changes to Job test-cadence-schema-setup with timeout of 10m0s
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: ADDED
client.go:530: [debug] test-cadence-schema-setup: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: MODIFIED
client.go:530: [debug] test-cadence-schema-setup: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-setup: MODIFIED
client.go:108: [debug] creating 1 resource(s)
client.go:463: [debug] Watching for changes to Job test-cadence-schema-update with timeout of 10m0s
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: ADDED
client.go:530: [debug] test-cadence-schema-update: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: MODIFIED
client.go:530: [debug] test-cadence-schema-update: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:491: [debug] Add/Modify event for test-cadence-schema-update: MODIFIED
client.go:254: [debug] Starting delete for "test-cadence-schema-setup" Job
client.go:254: [debug] Starting delete for "test-cadence-schema-update" Job

[sagikazarmark]

Thanks @drewhemm !

We'll look into it.