Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecating run_kubernetes (and maybe run_docker). #92

Open
nbren12 opened this issue Aug 6, 2020 · 0 comments
Open

Deprecating run_kubernetes (and maybe run_docker). #92

nbren12 opened this issue Aug 6, 2020 · 0 comments

Comments

@nbren12
Copy link
Contributor

nbren12 commented Aug 6, 2020

These functions served us well, but haven't been used in fv3net for a while, mainly because it was difficult to extend run_kubernetes to support I/O patterns involving shared volumes. We have largely replaced use of run_kuberernetes with argo workflows as well as these composable functions for working with containers: https://github.com/VulcanClimateModeling/fv3net/blob/master/external/fv3kube/fv3kube/containers.py. In both cases, we have decomposed the process of running fv3 into a few steps

  1. prepare the full fv3config.yml object from a shorter diff configuration
  2. write the run directory
  3. run the model
  4. post process

Since each of these steps have very different requirements (e.g. 2 and 4 needs auth, but 3 does not) we now run them in separate containers.

IMO, for containerized workflows the role of fv3config is really isolated to step 2, so I think it would make sense to deprecate/remove run_kubernetes. Thoughts?

It is important to support run_native as a legacy option for testing purposes, and I'm not sure about run_docker for local development. I don't really use it, but others might.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant