Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ingest to pathogen-repo-template #234

Merged
merged 7 commits into from
Feb 6, 2024

Conversation

joverlee521
Copy link
Contributor

@joverlee521 joverlee521 commented Jan 30, 2024

Description of proposed changes

Update ingest directory to match pathogen-repo-template layout.

This PR does not change any output data formats to not break any downstream uses of the data, i.e. the fields for the NDJSON and metadata.tsv were not changed.

Checklist

Part of work to update this repo to match the pathogen-repo-template.

We are removing the `source-data` directory and consolidating everything
into the config directory since there's really no functional difference
between them.
Part of work to update this repo to match the pathogen-repo-template.

Simplify directory structure by putting the core Snakemake rules in
a "rules" directory.
Part of work to update this repo to match the pathogen-repo-template.

Always provide default config values that can then easily be overridden
with --configfiles/--config options. This makes it simpler to change a
subset of config values or extend the default configs.

Renames the config/ to defaults/ to reflect this change.
Match changes in pathogen-repo-template in nextstrain/pathogen-repo-guide#27
Part of work to update this repo to match the pathogen-repo-template.

Simplify the core workflow by moving the Nextstrain automation specific
rules to the new build config.
@joverlee521
Copy link
Contributor Author

Not sure what's going on with the pre-commit.ci failure.

All tests pass when I run pre-commit run --all-files locally:

Screenshot 2024-01-30 at 2 57 01 PM

@joverlee521
Copy link
Contributor Author

Test run completed successfully and the metadata diff output to the test #scratch channel included the 10 new sequences that have become available since this morning's automated ingest run. No other data changes were reported.

@joverlee521
Copy link
Contributor Author

pre-commit.ci failure should be unrelated to changes in this PR. It is likely to be the same failure that is on the master branch. I see the same failure locally after rebasing this branch onto master.

I'm going to leave the pre-commit.ci failure in this PR alone since it will be fixed separately in #233 (or we can exclude .github/dependabot.yml in the .yamlfmt config)

@joverlee521 joverlee521 requested a review from a team January 31, 2024 19:31
@joverlee521
Copy link
Contributor Author

Merging since test run was successful. Will monitor automated runs the next couple of days.

@joverlee521 joverlee521 merged commit d4eb28b into master Feb 6, 2024
7 of 8 checks passed
@joverlee521 joverlee521 deleted the update-ingest-to-template branch February 6, 2024 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant