Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce some helpers on top of skein service/applicationspec abstraction #219

Open
fhoering opened this issue Jul 28, 2020 · 0 comments
Open

Comments

@fhoering
Copy link
Contributor

fhoering commented Jul 28, 2020

A common use case we have is a fire a forget mode where one launches only one container that executes the real workload (allowing for example to do hyper parameter tuning with many of those containers in parallel)

In this case it is useful to have an additional layer on top of service, applicationspec to only have one entry point. Also in interactive/jupyter mode it is sometimes useful to directly use a function and use cloudpickle to ship this function.

We have added this code to one of our projects but I think the best place would be to host it somewhere in this repo.
https://github.com/criteo/cluster-pack/blob/master/cluster_pack/skein/skein_launcher.py

There is no real risk on maintenance I think. In the worst case it is unused. Not much complexity.

WDYT ? I can do that PR and add some tests if you think it makes sens to add it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant