Support Snakemake version 8 across core pathogen workflows

[victorlin]

Currently, our managed runtimes come with Snakemake version 7 so our core pathogen workflows have been written to work with that.

Snakemake version 8 was released in December 2023.

It's known to not work in ncov, reported by an external user. Not sure about other repos as we haven't tried yet.

Tasks

• create docker-base image to simplify testing of compatibility: docker.io/nextstrain/base:branch-victorlin-install-snakemake-8
• Update ncov to be compatible with version 8
• Remove --stats when we upgrade to Snakemake v8 ncov-ingest#452
• add other pathogen repos
• Bump Snakemake version in managed runtimes

[corneliusroemer]

Thanks @victorlin!

How about we create a snakemake 8 image and do test runs with it in repos? Then we'll know what needs to change.

I've added it as a task

[victorlin]

Sure! That seems pretty low effort. I'll draft a docker-base PR now.

[victorlin]

Test image is available as nextstrain/base:branch-victorlin-install-snakemake-8. It's not possible to use the proposed simple name nextstrain/base:snakemake8 without manually pushing another tag which could get out of sync with PR. The next simplest would be nextstrain/base:branch-snakemake8. Mine is lengthier due to branch naming habits, but copy/pasting this value is probably a good idea anyways.

[victorlin]

I'm wondering how we would follow through with a big change like this. Is it possible to update to be Snakemake 8 compatible and still work on Snakemake 7? Or are the two mutually exclusive (this is my impression)? Currently all our pathogen repos pull the latest Docker image, so this would require either pinning specific runtime versions or a well-coordinated update.

[genehack]

I'm wondering how we would follow through with a big change like this. Is it possible to update to be Snakemake 8 compatible and still work on Snakemake 7? Or are the two mutually exclusive (this is my impression)?

Seems like confirming whether or not 8 is backcompat with 7 would be a first step? Has anybody built a list of changes? (That would also help with the inevitable "why are we doing this" question I expect.)

Currently all our pathogen repos pull the latest Docker image, so this would require either pinning specific runtime versions or a well-coordinated update.

This feels like a small piece of a larger conversation about long-term pathogen repo maintenance practices.

[tsibley]

This feels like a small piece of a larger conversation about long-term pathogen repo maintenance practices.

This, for sure. While the current practices worked well for a smaller number of production pathogens, they won't necessarily work as well for the larger number we're growing to.