Allow DNS to resolve workers

[max-sixty]

Is there a way of allowing the webserver to resolve the workers through their names?

Currently, this stack puts workers into StatefulSets. I think the barrier to using a Deployment is that before log files are put in GCS, they fail to resolve:

*** Fetching from: http://airflow-worker-6dd9f5bc74-h97vf:8793/log/[***]
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-6dd9f5bc74-h97vf', port=8793): Max retries exceeded with url: /log/[***] (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f142598b210>: Failed to establish a new connection: [Errno -2] Name or service not known',))

If we could surmount that, we could use a Deployment, which means we can run on preemptive machines, and generally allow moving pods around

[max-sixty]

On further reading, my understanding of a StatefulSet was poor. Is there actually any benefit to a Deployment over a Stateful set in this case?

[amith1893]

@max-sixty Were you able to use a Deployment over a StatefulSet ? I am curious to know as even I am looking into it

[max-sixty]

I didn't pursue this much further, unfortunately