[DRAFT] TSPS-183 Beagle imputation hg38 wdl and associated support wdls #1333

mmorgantaylor · 2024-07-15T19:26:48Z

Description

Give your PR a concise yet descriptive title.
Please explain the changes you made here.
Explain the motivation for making this change. What existing problem does the pull request solve?
Mention any issues fixed, addressed, or otherwise related to this pull request, including issue numbers or hard links for issues in other repos.
You can delete these instructions once you have written your PR description.

Checklist

If you can answer "yes" to the following items, please add a checkmark next to the appropriate checklist item(s) and notify our WARP documentation team by tagging either @ekiernan or @kayleemathews in a comment on this PR.

Did you add inputs, outputs, or tasks to a workflow?
Did you modify, delete or move: file paths, file names, input names, output names, or task names?
If you made a changelog update, did you update the pipeline version number?

Staging -> master

dev -> staging

Staging -> master

dev -> staging

Staging -> master

This reverts commit 3757137.

* try auto specifying chr at end of basename * both tasks * add liftovervcfs to dockstore * allow specifying max mem

dsde-jenkins · 2024-07-15T19:26:51Z

Can one of the admins verify this patch?

github-actions · 2024-07-15T19:27:01Z

Remember to squash merge!

github-actions · 2024-07-15T19:27:13Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
WholeGenomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
ExomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
VariantCalling.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeCramOnly.changelog.md has not been changed and needs to be updated
BroadInternalUltimaGenomics.changelog.md has not been changed and needs to be updated
BroadInternalRNAWithUMIs.changelog.md has not been changed and needs to be updated
BroadInternalImputation.changelog.md has not been changed and needs to be updated
BroadInternalArrays.changelog.md has not been changed and needs to be updated
ExternalWholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExternalExomeReprocessing.changelog.md has not been changed and needs to be updated
WholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExomeReprocessing.changelog.md has not been changed and needs to be updated
CheckFingerprint.changelog.md has not been changed and needs to be updated
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeagle.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeaglePreChunk.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/CreateImputationRefPanelBeagle.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/LiftoverVcfs.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Imputation.changelog.md has not been changed and needs to be updated
Arrays.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

github-actions · 2024-07-15T19:27:14Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
WholeGenomeGermlineSingleSample.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
ExomeGermlineSingleSample.wdl has not been changed and needs updating
VariantCalling.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeCramOnly.wdl has not been changed and needs updating
BroadInternalUltimaGenomics.wdl has not been changed and needs updating
BroadInternalRNAWithUMIs.wdl has not been changed and needs updating
BroadInternalImputation.wdl has not been changed and needs updating
BroadInternalArrays.wdl has not been changed and needs updating
ExternalWholeGenomeReprocessing.wdl has not been changed and needs updating
ExternalExomeReprocessing.wdl has not been changed and needs updating
WholeGenomeReprocessing.wdl has not been changed and needs updating
ExomeReprocessing.wdl has not been changed and needs updating
CheckFingerprint.wdl has not been changed and needs updating
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeagle.changelog.md: No such file or directory
ImputationBeagle.wdl and ImputationBeagle.changelog.md do not have matching versions
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeaglePreChunk.changelog.md: No such file or directory
ImputationBeaglePreChunk.wdl and ImputationBeaglePreChunk.changelog.md do not have matching versions
CreateImputationRefPanelBeagle.wdl has not had its version updated
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/CreateImputationRefPanelBeagle.changelog.md: No such file or directory
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/LiftoverVcfs.changelog.md: No such file or directory
LiftoverVcfs.wdl and LiftoverVcfs.changelog.md do not have matching versions
Imputation.wdl has not had its version updated
Arrays.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

* try creating bed files * try again * try again again * a different thing * use bedtools and bed ref panel files * oops update the correct task * fix * use the right freaking file name * remove comment

github-actions · 2024-07-19T16:51:20Z

Remember to squash merge!

github-actions · 2024-07-19T16:51:31Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
snm3C.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
Optimus.changelog.md has not been changed and needs to be updated
Multiome.changelog.md has not been changed and needs to be updated
atac.changelog.md has not been changed and needs to be updated
WholeGenomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
ExomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
VariantCalling.changelog.md has not been changed and needs to be updated
ReblockGVCF.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeCramOnly.changelog.md has not been changed and needs to be updated
BroadInternalUltimaGenomics.changelog.md has not been changed and needs to be updated
BroadInternalRNAWithUMIs.changelog.md has not been changed and needs to be updated
BroadInternalImputation.changelog.md has not been changed and needs to be updated
BroadInternalArrays.changelog.md has not been changed and needs to be updated
ExternalWholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExternalExomeReprocessing.changelog.md has not been changed and needs to be updated
WholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExomeReprocessing.changelog.md has not been changed and needs to be updated
CheckFingerprint.changelog.md has not been changed and needs to be updated
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeagle.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeaglePreChunk.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/CreateImputationRefPanelBeagle.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/LiftoverVcfs.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Imputation.changelog.md has not been changed and needs to be updated
Arrays.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

github-actions · 2024-07-19T16:51:31Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
snm3C.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
Optimus.wdl has not been changed and needs updating
Multiome.wdl has not been changed and needs updating
atac.wdl has not been changed and needs updating
WholeGenomeGermlineSingleSample.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
ExomeGermlineSingleSample.wdl has not been changed and needs updating
VariantCalling.wdl has not been changed and needs updating
ReblockGVCF.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeCramOnly.wdl has not been changed and needs updating
BroadInternalUltimaGenomics.wdl has not been changed and needs updating
BroadInternalRNAWithUMIs.wdl has not been changed and needs updating
BroadInternalImputation.wdl has not been changed and needs updating
BroadInternalArrays.wdl has not been changed and needs updating
ExternalWholeGenomeReprocessing.wdl has not been changed and needs updating
ExternalExomeReprocessing.wdl has not been changed and needs updating
WholeGenomeReprocessing.wdl has not been changed and needs updating
ExomeReprocessing.wdl has not been changed and needs updating
CheckFingerprint.wdl has not been changed and needs updating
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeagle.changelog.md: No such file or directory
ImputationBeagle.wdl and ImputationBeagle.changelog.md do not have matching versions
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeaglePreChunk.changelog.md: No such file or directory
ImputationBeaglePreChunk.wdl and ImputationBeaglePreChunk.changelog.md do not have matching versions
CreateImputationRefPanelBeagle.wdl has not had its version updated
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/CreateImputationRefPanelBeagle.changelog.md: No such file or directory
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/LiftoverVcfs.changelog.md: No such file or directory
LiftoverVcfs.wdl and LiftoverVcfs.changelog.md do not have matching versions
Imputation.wdl has not had its version updated
Arrays.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

github-actions · 2024-07-26T15:14:55Z

Remember to squash merge!

github-actions · 2024-07-26T15:15:05Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
Multiome.changelog.md has not been changed and needs to be updated
PairedTag.changelog.md has not been changed and needs to be updated
snm3C.changelog.md has not been changed and needs to be updated
Optimus.changelog.md has not been changed and needs to be updated
SlideSeq.changelog.md has not been changed and needs to be updated
atac.changelog.md has not been changed and needs to be updated
MultiSampleSmartSeq2SingleNucleus.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeCramOnly.changelog.md has not been changed and needs to be updated
VariantCalling.changelog.md has not been changed and needs to be updated
ReblockGVCF.changelog.md has not been changed and needs to be updated
WholeGenomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
ExomeGermlineSingleSample.changelog.md has not been changed and needs to be updated
UltimaGenomicsWholeGenomeGermline.changelog.md has not been changed and needs to be updated
ExternalWholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExternalExomeReprocessing.changelog.md has not been changed and needs to be updated
WholeGenomeReprocessing.changelog.md has not been changed and needs to be updated
ExomeReprocessing.changelog.md has not been changed and needs to be updated
BroadInternalRNAWithUMIs.changelog.md has not been changed and needs to be updated
BroadInternalUltimaGenomics.changelog.md has not been changed and needs to be updated
BroadInternalImputation.changelog.md has not been changed and needs to be updated
BroadInternalArrays.changelog.md has not been changed and needs to be updated
CheckFingerprint.changelog.md has not been changed and needs to be updated
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/CreateImputationRefPanelBeagle.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/LiftoverVcfs.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeagle.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeaglePreChunk.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Imputation.changelog.md has not been changed and needs to be updated
Arrays.changelog.md has not been changed and needs to be updated
Some changelog files need updating. See output for details.
validation_failed

github-actions · 2024-07-26T15:15:06Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
Multiome.wdl has not been changed and needs updating
PairedTag.wdl has not been changed and needs updating
snm3C.wdl has not been changed and needs updating
Optimus.wdl has not been changed and needs updating
SlideSeq.wdl has not been changed and needs updating
atac.wdl has not been changed and needs updating
MultiSampleSmartSeq2SingleNucleus.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeCramOnly.wdl has not been changed and needs updating
VariantCalling.wdl has not been changed and needs updating
ReblockGVCF.wdl has not been changed and needs updating
WholeGenomeGermlineSingleSample.wdl has not been changed and needs updating
ExomeGermlineSingleSample.wdl has not been changed and needs updating
UltimaGenomicsWholeGenomeGermline.wdl has not been changed and needs updating
ExternalWholeGenomeReprocessing.wdl has not been changed and needs updating
ExternalExomeReprocessing.wdl has not been changed and needs updating
WholeGenomeReprocessing.wdl has not been changed and needs updating
ExomeReprocessing.wdl has not been changed and needs updating
BroadInternalRNAWithUMIs.wdl has not been changed and needs updating
BroadInternalUltimaGenomics.wdl has not been changed and needs updating
BroadInternalImputation.wdl has not been changed and needs updating
BroadInternalArrays.wdl has not been changed and needs updating
CheckFingerprint.wdl has not been changed and needs updating
CreateImputationRefPanelBeagle.wdl has not had its version updated
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/CreateImputationRefPanelBeagle.changelog.md: No such file or directory
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/LiftoverVcfs.changelog.md: No such file or directory
LiftoverVcfs.wdl and LiftoverVcfs.changelog.md do not have matching versions
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeagle.changelog.md: No such file or directory
ImputationBeagle.wdl and ImputationBeagle.changelog.md do not have matching versions
grep: /home/runner/work/warp/warp/pipelines/broad/arrays/imputation_beagle/ImputationBeaglePreChunk.changelog.md: No such file or directory
ImputationBeaglePreChunk.wdl and ImputationBeaglePreChunk.changelog.md do not have matching versions
Imputation.wdl has not had its version updated
Arrays.wdl has not been changed and needs updating
Some WDLs or changelog files need updating. See output for details.
validation_failed

update ImputationBeagle

jessicaway · 2024-11-13T16:18:29Z

@mmorgantaylor is this still relevant?

mmorgantaylor · 2024-11-13T17:10:53Z

@jessicaway yes this is our working branch for beagle imputation for the imputation service. cc @jsotobroad. Jess, would it be helpful for us to squash and rebase off of develop?

* add array imputation quota consumed wdl * add changelogs for imputation array related workflows --------- Co-authored-by: Jose Soto <[email protected]>

jessicaway · 2024-11-13T21:20:41Z

OK, no problem! Whatever works best is fine. We were just cleaning up PRs and thought we would check in. I don't think we have touched imputation much recently, so I don't anticipate any conflicts

* changes to help beagle imputation wdl run on a 400k sample reference panel --------- Co-authored-by: Jose Soto <[email protected]> * remove create imputation ref panel beagle wdl and changelog * PR feedback --------- Co-authored-by: Jose Soto <[email protected]> Co-authored-by: M. Morgan Aster <[email protected]>

Co-authored-by: Jose Soto <[email protected]>

nikellepetrillo and others added 30 commits February 7, 2024 09:48

Merge pull request #1198 from broadinstitute/staging

9f137a7

Staging -> master

Merge pull request #1200 from broadinstitute/develop

92a887e

dev -> staging

Merge pull request #1202 from broadinstitute/staging

045b998

Staging -> master

Merge pull request #1208 from broadinstitute/develop

664f959

dev -> staging

Merge pull request #1209 from broadinstitute/staging

b8a32c7

Staging -> master

wip add beagle imputation stuff

c65dc53

add 2 wdls to dockstore.yml

2306187

fix docker gar url

85e19de

use the right path for jars

7e2be90

wip on imputation wdl

9c64eeb

oops use correct jar

39482c4

missing equals

f175179

fix java call again

648f938

fix java call

14357bf

oops match file names

457339b

update beagle jar to 01Mar24.d36

3fc22a8

debug GatherVcfs

435ba19

debug GatherVcfs 2

b72281a

try to resolve missing file issue

fd04907

don't impute over padding

bce4c9d

make the index again

64d307f

supply vcf_index input to SelectVariantsByIds

54b05d6

update Imputation wdl too

f2efe2c

newlines

0d3c862

update for hg38

3757137

Revert "update for hg38"

30c634a

This reverts commit 3757137.

update for hg38

e8bf6ce

liftover wdl

fa3bbd4

remove GCP-specific vm commands

7af23b4

use gatk

b5bf5f4

mmorgantaylor added 6 commits July 8, 2024 12:49

add prechunk wdl to dockstore

6201343

use acr for default ubuntu image

f4ca86e

add preemptible 3

cec272e

use acr gatk docker as default

4d7ec88

don't use preemptibles on GatherVcfs

8cb21b1

basename fix for imputation beagle ref panel generation (#1332)

c57210c

* try auto specifying chr at end of basename * both tasks * add liftovervcfs to dockstore * allow specifying max mem

pullapprove bot requested review from fleharty, gbggrant, kachulis and samuelklee July 15, 2024 19:26

TSPS-269 Speed up CountVariantsInChunksBeagle by using bedtools (#1335)

95c3941

* try creating bed files * try again * try again again * a different thing * use bedtools and bed ref panel files * oops update the correct task * fix * use the right freaking file name * remove comment

update pipeline version to 0.0.2

700fdec

Updated pipeline_versions.txt with all pipeline version information

fd0ffed

TSPS-293: Fix up streaming imputation beagle (#1347)

c0754cc

update ImputationBeagle

add array imputation quota consumed wdl (#1425)

f4f6e47

* add array imputation quota consumed wdl * add changelogs for imputation array related workflows --------- Co-authored-by: Jose Soto <[email protected]>

jsotobroad and others added 2 commits November 21, 2024 10:20

add set -e -o pipefail to all relevant imputation tasks (#1434)

e51628a

Co-authored-by: Jose Soto <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DRAFT] TSPS-183 Beagle imputation hg38 wdl and associated support wdls #1333

[DRAFT] TSPS-183 Beagle imputation hg38 wdl and associated support wdls #1333

mmorgantaylor commented Jul 15, 2024

dsde-jenkins commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 19, 2024

github-actions bot commented Jul 19, 2024

github-actions bot commented Jul 19, 2024

github-actions bot commented Jul 26, 2024

github-actions bot commented Jul 26, 2024

github-actions bot commented Jul 26, 2024

jessicaway commented Nov 13, 2024

mmorgantaylor commented Nov 13, 2024

jessicaway commented Nov 13, 2024

[DRAFT] TSPS-183 Beagle imputation hg38 wdl and associated support wdls #1333

Are you sure you want to change the base?

[DRAFT] TSPS-183 Beagle imputation hg38 wdl and associated support wdls #1333

Conversation

mmorgantaylor commented Jul 15, 2024

Description

Checklist

dsde-jenkins commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

github-actions bot commented Jul 15, 2024

🔍Changelog Validation Results:

github-actions bot commented Jul 15, 2024

🔍Version Validation Results:

github-actions bot commented Jul 19, 2024

github-actions bot commented Jul 19, 2024

🔍Changelog Validation Results:

github-actions bot commented Jul 19, 2024

🔍Version Validation Results:

github-actions bot commented Jul 26, 2024

github-actions bot commented Jul 26, 2024

🔍Changelog Validation Results:

github-actions bot commented Jul 26, 2024

🔍Version Validation Results:

jessicaway commented Nov 13, 2024

mmorgantaylor commented Nov 13, 2024

jessicaway commented Nov 13, 2024