Reset Dask worker to use TCP even if it was configured to use TLS in yaml file

[weiwang217]

Describe the issue:
The DASK operator reset to use TCP even if it was configured to use TLS

- name: DASK_SCHEDULER_ADDRESS
value: tls://scheduler.join.svc.cluster.local:8786
- name: DASK_TEMPORARY_DIRECTORY
value: /tmp
- name: DASK_WORKER_NAME
value: default-worker-6a9c9e4f94
- name: DASK_SCHEDULER_ADDRESS
value: tcp://scheduler.join.svc.cluster.local:8786

The code to append the config is here:

dask-kubernetes/dask_kubernetes/operator/controller/controller.py

Line 156 in fa7255b

"name": "DASK_SCHEDULER_ADDRESS",

Minimal Complete Verifiable Example:

# Put your MCVE code here

Anything else we need to know?:

Environment:

• Dask version:
• Python version:
• Operating System:
• Install method (conda, pip, source):

[jacobtomlinson]

Thanks for raising this @weiwang217. I've opened #837 to resolve this. Would you mind testing that PR out and letting me know if it solves your problem?

[weiwang217]

Thanks! How can I build and install the dask operator? Jacob Tomlinson ***@***.***> 于2023年10月19日周四 03:48写道：

…

Thanks for raising this @weiwang217 <https://github.com/weiwang217>. I've opened #837 <#837> to resolve this. Would you mind testing that PR out and letting me know if it solves your problem? — Reply to this email directly, view it on GitHub <#836 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZF2RVJUPVM3WL7I4EDBN3YAEAPRAVCNFSM6AAAAAA6GFV5BWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZQGU2TKNBTHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Wang, Wei MAIL: ***@***.***

[jacobtomlinson]

We have documentation on how to do this here https://kubernetes.dask.org/en/latest/testing.html#testing-operator-controller-prs

[weiwang217]

It worked. When is the code going to be merged into the main? Thanks! Thanks, Wei Jacob Tomlinson ***@***.***> 于2023年10月20日周五 02:39写道：

…

We have documentation on how to do this here https://kubernetes.dask.org/en/latest/testing.html#testing-operator-controller-prs — Reply to this email directly, view it on GitHub <#836 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABZF2RUJWBMGVBJMVI6T5BLYAJBFJAVCNFSM6AAAAAA6GFV5BWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZSGQYDONZYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Wang, Wei MAIL: ***@***.***

[kjleftin]

Hi Jacob,

I have a suspicion that change may have caused a regression when working with replicas > 1. When I start a new DaskJob, all but one replica fails to connect to the scheduler because of duplicate names. Indeed when I run
kubectl describe pod <worker_pod>

I see:
Worker 1:

    Environment:
      DASK_WORKER_NAME:                                  simple-job-default-worker-a10a25ac26
      DASK_SCHEDULER_ADDRESS:                      tcp://simple-job-scheduler.join.svc.cluster.local:8786
      ...

Worker 2:

    Environment:
      DASK_WORKER_NAME:                                  simple-job-default-worker-00add84cde
      DASK_SCHEDULER_ADDRESS:                      tcp://simple-job-scheduler.join.svc.cluster.local:8786
      DASK_WORKER_NAME:                                   simple-job-default-worker-a10a25ac26
      DASK_SCHEDULER_ADDRESS:                      tcp://simple-job-scheduler.join.svc.cluster.local:8786

Because the last defined environment variable is the first replica, all replicas share the same name.

Do you mind taking a look?

(Context: I'm on the same team as weiwang217 and we just noticed this change recently)

[jacobtomlinson]

Thanks for reporting this @kjleftin. Why are you setting the DASK_WORKER_NAME in your config?

[kjleftin]

Hi Jacob,

I'm following the example code in https://kubernetes.dask.org/en/latest/operator_resources.html#daskjob

Specifically, passing the DASK_WORKER_NAME env. variable to the dask worker CLI:

            - name: worker
              image: "ghcr.io/dask/dask:latest"
              imagePullPolicy: "IfNotPresent"
              args:
                - dask-worker
                - --name
                - $(DASK_WORKER_NAME)
                - --dashboard
                - --dashboard-address
                - "8788"

Note that I'm not setting DASK_WORKER_NAME explicitly. That is handled by the Dask Operator. (Before this change, each worker would have a different value for DASK_WORKER_NAME, but after this change, each worker has the same value).

[jacobtomlinson]

@kjleftin ok thanks for the clarification. I expect we may need to use copy to avoid this. I'll take a look at the PR and update it.