Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python dependency management #17

Open
gsemet opened this issue Oct 28, 2017 · 0 comments
Open

Python dependency management #17

gsemet opened this issue Oct 28, 2017 · 0 comments

Comments

@gsemet
Copy link
Contributor

gsemet commented Oct 28, 2017

Let's imagine we have DAGs that does some math or connect to a database. This dags will probably use a python dependency, that should be fectched and installed on the python environment of the Dags.
If the scheduler + workers + web ui does not have these dependencies, they will crash (I think the scheduler can handle that).

My PR #16 uses git-sync to automatically synchronize the Dags with a git repository. But it will work until a new python dependency is needed

So there is two solutions:

  • force users to build there own docker image, and hope they do rolling update properly to deploy on their workers.
  • allow installation from the workers (from a requirements.txt defined in the Dag folders) in entrypoint.sh.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant