Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create persistent per-user folder on HPC and mount into singularity containers #58

Open
zhiyuli opened this issue Aug 17, 2022 · 1 comment

Comments

@zhiyuli
Copy link
Contributor

zhiyuli commented Aug 17, 2022

For each user, we create a persistent folder at a location other than scratch (configurable in json), folder name could be <jupyter_username>@<jupyter_instance_url>.
This per-user folder will be mounted into each container at the same local path, such as /compute_persistent.

The use case is, user submits a job A, which dumps some data into the persistent folder; and later user could submit job B to read data from the persistent folder.

Background:
current core support specifying additional folders to be mounted to containers on HPC, this is configurable in json. but this is not a per-user mount. it is a shared mount. all users see the same folder in their singularity containers (and could change them as current mount type is RW). this ticket is to make the folder on a per-user basis

This would overlap with some of the problems being addressed by the "reusable data" work. But I think the above might be a quick "hack" to enable "reusable data" in a short term.

@zhiyuli
Copy link
Contributor Author

zhiyuli commented Aug 17, 2022

We will need to have a discussion on this before implementation. Please leave comments/suggestions/question in this ticket

@alexandermichels alexandermichels added this to the Future milestone Jan 24, 2023
@alexandermichels alexandermichels removed this from the Future milestone Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants