pytorchjob didn't create worker pod ,seems hang

[Twilighter9527]

What happened?

I follow this create a pytorchjob https://github.com/kubeflow/training-operator/blob/master/examples/pytorch/simple.yaml

pytorchjob yaml

pod status

kubeclt describe pytorchjob -n kubeflow

training operator log

What did you expect to happen?

this should create pytorch-simple-master-0 and pytorch-simple-worker-0,but the pytorch-simple-worker-0 seems hang,the log show that some wrong with the yaml to json, i didn't think that is the reason. first, kubeclt create -f simple.yaml, nothing wrong happened. second, i use python to read the yaml to json, it is ok. I had a similar issue a few weeks ago, and I set sidecar.istio=false to solve it。 but now is not in default namespaces and I label the node sidecar.istio=false

Environment

Kubernetes version:

$ kubectl version

Training Operator version:

$ kubectl get pods -n kubeflow -l control-plane=kubeflow-training-operator -o jsonpath="{.items[*].spec.containers[*].image}"

Training Operator Python SDK version:

$ pip show kubeflow-training

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[andreyvelich]

Thanks for creating this @Twilighter9527!
How did you add the istio annotations to your PyTorch Job ? Did you use quotes ?

sidecar.istio.io/inject: "false"

[Twilighter9527]

Thanks for creating this @Twilighter9527! How did you add the istio annotations to your PyTorch Job ? Did you use quotes ?

sidecar.istio.io/inject: "false"

yes，just like the demo,event set the level of node to sidecar.istio.io/inject: "false",nothing happen

[Twilighter9527]

[kuizhiqing]

The error log in training operator log is quite sure there is a mistake in the configure yaml.

[Twilighter9527]

The error log in training operator log is quite sure there is a mistake in the configure yaml.

but i can create the pod success(py-master-0), and i only user the master part yaml or woker part yaml can create success.

[kuizhiqing]

Have you ever try with the origin yaml file with touching nothing ?

[Twilighter9527]

Have you ever try with the origin yaml file with touching nothing ?
i user this simple.yaml, just change the image field

[kuizhiqing]

Try with CHANGE NOTHING, even image not found error will happen after pod has been created.

[Twilighter9527]

it will failed on create master pod,the log just as before, so weird.

[andreyvelich]

Hi @Twilighter9527, it looks like your cluster doesn't have access to the public DockerHub registry.
Do you need to configure proxies to pull the public images ?