Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust Batch provisioning for managed compute environment #858

Closed
KlaasH opened this issue Dec 10, 2021 · 2 comments
Closed

Adjust Batch provisioning for managed compute environment #858

KlaasH opened this issue Dec 10, 2021 · 2 comments

Comments

@KlaasH
Copy link
Contributor

KlaasH commented Dec 10, 2021

Issue #857 calls for manually creating a managed Batch compute environment to run the analysis. After we've done that successfully, the next step is to adjust the provisioning/deployment infrastructure to create and use an environment like that rather than the current unmanaged one.

@BryanQuigley BryanQuigley self-assigned this Jun 28, 2022
@BryanQuigley BryanQuigley removed their assignment Jul 6, 2022
@BryanQuigley BryanQuigley reopened this Jul 6, 2022
@KlaasH
Copy link
Contributor Author

KlaasH commented Mar 23, 2023

AWS has added the option to configure much more ephemeral storage on Fargate tasks (up to 200GB; the old limit was 20GB), so we should be able to skip the managed environment and convert the analysis to Fargate. https://aws.amazon.com/about-aws/whats-new/2023/03/aws-batch-configurable-ephemeral-storage-fargate/

Per #839 (comment) (linked from the comment that's linked from the comment above), ~150GB would suffice to provide enough space and a bit of buffer for our biggest jobs.

@KlaasH
Copy link
Contributor Author

KlaasH commented Dec 12, 2023

This issue is from an earlier iteration of the plan, where we were talking about moving to a managed EC2-based compute environment. Now we're talking about using Fargate, so this is superseded by issue #941.

@KlaasH KlaasH closed this as completed Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants