Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Add --cores all to rebuild-mpxv.yaml #149

Merged
merged 3 commits into from
Apr 13, 2023

Conversation

corneliusroemer
Copy link
Member

@corneliusroemer corneliusroemer commented Apr 12, 2023

This is required for the action to be compatible with snakemake version >5.11.0
which started to require the --cores option

For more context see:

I added --cores all when no --cpus was passed

This is required for the action to be compatible with
snakemake version >5.11.0 which started to require the
`--cores` option
For more context see:
- nextstrain/cli#272
-https://bedfordlab.slack.com/archives/C01LCTT7JNN/p1681324279363569?thread_ts=1681293140.124269&cid=C01LCTT7JNN
@corneliusroemer corneliusroemer requested a review from tsibley April 12, 2023 20:11
@@ -35,6 +35,7 @@ jobs:
. \
envdir env.d snakemake notify_on_deploy \
--configfiles config/config_mpxv.yaml config/nextstrain_automation.yaml \
--cores all \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I don't think we should use --cores all because of nextstrain/cli#175. (I mentioned this earlier, but I might not have explained it well then.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't use all whenever there was an explicit number given. I didn't know what to pass otherwise when --cpus wasn't passed.

What does nextstrain build use as --cpus by default? CLI --help-all said None? So if there's no machine up it won't run? Or will it be like 0.00001?

What would you put instead in case of no --cpus?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It uses the default as configured in AWS Batch. That will vary for different organizations, but for us, it's currently 4.

For this case above, I would explicitly pass --cpus 4 to nextstrain build to make the AWS Batch default explicit (until nextstrain/cli#175 is fixed) and then also --cores 4 to snakemake (until nextstrain/cli#272 is fixed, probably via first-class env var management support).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To get things moving quickly I will merge with all as I cannot see this ever causing real issues with the way we run workflows - snakemake may schedule too many jobs overall but that doesn't actually harm us.

We can easily replace all --cores all instances later

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, ok, I guess, but I don't see how --cores all here is in any way better or quicker than --cpus 4 + --cores 4.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make what I'd said earlier very clear: if we run concurrent workflows with small numbers of requested CPUs, they are likely to get packed onto the same machine, where overscheduling jobs by Snakemake is likely to make them all slower. Slow enough to notice? Maybe, maybe not. I don't know, as I rarely pay attention to our workflow execution times.

@corneliusroemer corneliusroemer merged commit c249ac2 into master Apr 13, 2023
@corneliusroemer corneliusroemer deleted the snakemake-cores-patch branch April 13, 2023 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants