Skip to content

Commit

Permalink
Remove set-branch-ingest-config
Browse files Browse the repository at this point in the history
Recently learned that Snakemake no longer overwrites the entire config
when providing nested config values via the `--config` option.¹ This
allows us to remove the set-branch-ingest-config script and directly set
branch config values via the command line option in our GitHub Action
workflows.

I've opted to use `jq` to interpolate the branch name into the S3_DST
because I didn't want to fiddle with escaping quotes in the JSON string.

¹ nextstrain/dengue#13 (comment)
  • Loading branch information
joverlee521 committed Oct 30, 2023
1 parent c310037 commit 0d0154f
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 35 deletions.
11 changes: 5 additions & 6 deletions .github/workflows/fetch-and-ingest-branch.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,10 @@ jobs:

- name: run_pipeline
run: |
# Create JSON string for the nested upload config
GITHUB_BRANCH=${GITHUB_REF#refs/heads/}
./phylogenetic/bin/set-branch-ingest-config \
--config-yaml ingest/config/optional.yaml \
--s3-dst s3://nextstrain-data/files/workflows/monkeypox/branch/"${GITHUB_BRANCH}" \
> ingest/config/optional-branch.yaml
S3_DST="s3://nextstrain-data/files/workflows/monkeypox/branch/${GITHUB_BRANCH}"
UPLOAD_CONFIG=$(jq -n --arg S3_DST "$S3_DST" '{"s3": {"dst": $S3_DST }}')
nextstrain build \
--aws-batch \
Expand All @@ -43,8 +42,8 @@ jobs:
--env AWS_SECRET_ACCESS_KEY \
--env GITHUB_RUN_ID \
ingest \
--configfiles config/config.yaml config/optional-branch.yaml \
--config trigger_rebuild=False
--configfiles config/config.yaml config/optional.yaml \
--config trigger_rebuild=False send_slack_notifications=False upload="$UPLOAD_CONFIG"
env:
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
Expand Down
29 changes: 0 additions & 29 deletions phylogenetic/bin/set-branch-ingest-config

This file was deleted.

0 comments on commit 0d0154f

Please sign in to comment.