v2.0.0-alpha.5 (#244)

* Update base.config Match error-handling from nf-core (solves problem with status 140 that I had) * Use groovy to parse samplesheets (#243) * implement groovy functions for samplesheet parsing * set up utility class * add build to map * add json support * rename json parser * delete samplesheet module * remove mount docs (deprecated) * remove out.json from expected test output * fix parsing vcf samplesheet * Update report (renv and include licenses) (#203) * fix report link and setup renv lockfile * update dockerfile to use renv * add license terms * clarify custom scores + licenses * keep empty file in assets folder for optional input * Drop aggregation during report generation (#249) * drop small sampleset warning when integrating reference data * drop aggregation * update lockfile: drop data.table * load scores the same way * bump utils version * drop effect_type (shouldn't actually be in columns array) * use dev image * update score test * fix test * bump pgscatalog_utils version * boring but big: re-sync nf-core * fix cache error in report * nf-core resync * set deno dir * update schema * set up auto-generated environment * set workDir consistently * improve caching * fix help * set outdir default * use binary packages to build report image * fix missing dependencies * fix fraposa_pca * add storeDir to relabel_ids * fix score metadata in report * store results permanently * write to genotypes_cache, if set * update conda * add custom example scorefile * Update schema to say what the cache is for * Add information on cache and pipeline speed * Not only for -resume * fix storeDir * fix tempdir on HPC/singularity * gcp compatbility updates * remove semantic structure from storeDirs (bad) * Update cache.rst * don't copy twice * bump report singularity version * fix quarto * fix output * fix fraposa * fix capturing intersect counts --------- Co-authored-by: smlmbrt <[email protected]> * Fix cache names (#260) * Update plink2_makebed.nf Remove chromosome so that it doesn't cause cache collision (should only run once per sampleset anyways) * Fix prefix and output names * Remove chromosome ID from orient * set pytest version * fix node deprecation warnings * bump pytest-workflow * downgrade pytest * fix node deprecation warning * bump python action again * fix tmpdir errors in report * fix --genotypes_cache * make temporary dir in working directory * fix missing cachedir * cache relabelled afreq and variants separately * oops * add changelog to docs * bump report from dev to production * fix report distribution plots for non-ancestry runs * Update arm.rst with new profile * add deeep cache * stop using wildcards * swap cross operator -> combine * set pgscatlaog_utils version to v0.5.2 * fix log output * fix match_combine cache * oops * fix score aggregation cache * error if scoring files don't get a calculation result * update error strategy * fix missing quarto package * Fix race condition in cache (#262) * stop using global variables * use local variables * Add information about versions * add version to report (#265) * add version to report * update author --------- Co-authored-by: Sam Lambert <[email protected]>
PGScatalog · Mar 19, 2024 · 538a01d · 538a01d
1 parent 83326a1
commit 538a01d
Show file tree

Hide file tree

Showing 81 changed files with 1,744 additions and 1,680 deletions.
diff --git a/.github/workflows/ancestry-conda.yml b/.github/workflows/ancestry-conda.yml
@@ -16,15 +16,15 @@ jobs:
 
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Set environment variables
         run: |
           echo "ANCESTRY_REF_DIR=$RUNNER_TEMP" >> $GITHUB_ENV
           echo "ANCESTRY_TARGET_DIR=$RUNNER_TEMP" >> $GITHUB_ENV
 
       - name: Restore reference data
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: |
             ${{ env.ANCESTRY_TARGET_DIR }}/GRCh38_HAPNEST_TARGET_ALL.pgen
@@ -52,7 +52,7 @@ jobs:
           sudo mv nextflow /usr/local/bin/
 
       - name: Set up test requirements
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: '3.10'
           cache: 'pip'

diff --git a/.github/workflows/ancestry-vcf.yml b/.github/workflows/ancestry-vcf.yml
@@ -30,13 +30,13 @@ jobs:
           echo "ANCESTRY_TARGET_DIR=$RUNNER_TEMP" >> $GITHUB_ENV
       
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
-      - uses: nf-core/setup-nextflow@v1
+      - uses: nf-core/setup-nextflow@v2
 
       - name: Restore docker images
         id: restore-docker
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: ${{ runner.temp }}/docker
           key: ${{ inputs.container-cache-key }}
@@ -48,7 +48,7 @@ jobs:
           find ${{ runner.temp }}/docker/ -name '*.tar' -exec sh -c 'docker load < {}' \;
 
       - name: Restore reference data
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: |
             ${{ env.ANCESTRY_TARGET_DIR }}/GRCh38_HAPNEST_TARGET_ALL.pgen
@@ -65,7 +65,7 @@ jobs:
         run: plink2 --pfile ${ANCESTRY_TARGET_DIR}/GRCh38_HAPNEST_TARGET_ALL vzs --export vcf bgz --out ${ANCESTRY_TARGET_DIR}/GRCh38_HAPNEST_TARGET_ALL
 
       - name: Set up test requirements
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: '3.10'
           cache: 'pip'
@@ -97,13 +97,13 @@ jobs:
           echo "ANCESTRY_TARGET_DIR=$RUNNER_TEMP" >> $GITHUB_ENV
       
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
-      - uses: nf-core/setup-nextflow@v1
+      - uses: nf-core/setup-nextflow@v2
 
       - name: Restore singularity setup
         id: restore-singularity-setup
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: /opt/hostedtoolcache/singularity/${{ env.SINGULARITY_VERSION }}/x64
           key: ${{ runner.os }}-singularity-${{ env.SINGULARITY_VERSION }}
@@ -115,13 +115,13 @@ jobs:
 
       - name: Restore singularity container images
         id: restore-singularity
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: ${{ env.NXF_SINGULARITY_CACHEDIR }}
           key: ${{ inputs.container-cache-key }}
 
       - name: Restore reference data
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: |
             ${{ env.ANCESTRY_TARGET_DIR }}/GRCh38_HAPNEST_TARGET_ALL.pgen
@@ -138,7 +138,7 @@ jobs:
         run: plink2 --pfile ${ANCESTRY_TARGET_DIR}/GRCh38_HAPNEST_TARGET_ALL vzs --export vcf bgz --out ${ANCESTRY_TARGET_DIR}/GRCh38_HAPNEST_TARGET_ALL
 
       - name: Set up test requirements
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: '3.10'
           cache: 'pip'

diff --git a/.github/workflows/ancestry.yml b/.github/workflows/ancestry.yml
@@ -30,13 +30,13 @@ jobs:
           echo "ANCESTRY_TARGET_DIR=$RUNNER_TEMP" >> $GITHUB_ENV
       
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
-      - uses: nf-core/setup-nextflow@v1
+      - uses: nf-core/setup-nextflow@v2
 
       - name: Restore docker images
         id: restore-docker
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: ${{ runner.temp }}/docker
           key: ${{ inputs.container-cache-key }}
@@ -48,7 +48,7 @@ jobs:
           find ${{ runner.temp }}/docker/ -name '*.tar' -exec sh -c 'docker load < {}' \;
 
       - name: Restore reference data
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: |
             ${{ env.ANCESTRY_TARGET_DIR }}/GRCh38_HAPNEST_TARGET_ALL.pgen
@@ -59,7 +59,7 @@ jobs:
           fail-on-cache-miss: true
 
       - name: Set up test requirements
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: '3.10'
           cache: 'pip'
@@ -91,13 +91,13 @@ jobs:
           echo "ANCESTRY_TARGET_DIR=$RUNNER_TEMP" >> $GITHUB_ENV
       
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
-      - uses: nf-core/setup-nextflow@v1
+      - uses: nf-core/setup-nextflow@v2
 
       - name: Restore singularity setup
         id: restore-singularity-setup
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: /opt/hostedtoolcache/singularity/${{ env.SINGULARITY_VERSION }}/x64
           key: ${{ runner.os }}-singularity-${{ env.SINGULARITY_VERSION }}
@@ -109,13 +109,13 @@ jobs:
 
       - name: Restore singularity container images
         id: restore-singularity
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: ${{ env.NXF_SINGULARITY_CACHEDIR }}
           key: ${{ inputs.container-cache-key }}
 
       - name: Restore reference data
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: |
             ${{ env.ANCESTRY_TARGET_DIR }}/GRCh38_HAPNEST_TARGET_ALL.pgen
@@ -126,7 +126,7 @@ jobs:
           fail-on-cache-miss: true
 
       - name: Set up test requirements
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: '3.10'
           cache: 'pip'

diff --git a/.github/workflows/conda.yml b/.github/workflows/conda.yml
@@ -32,7 +32,7 @@ jobs:
 
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - uses: conda-incubator/setup-miniconda@v2
         with:  

diff --git a/.github/workflows/module.yml b/.github/workflows/module.yml
@@ -27,13 +27,13 @@ jobs:
 
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
-      - uses: nf-core/setup-nextflow@v1
+      - uses: nf-core/setup-nextflow@v2
 
       - name: Restore docker images
         id: restore-docker
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: ${{ runner.temp }}/docker
           key: ${{ inputs.cache-key }}
@@ -45,7 +45,7 @@ jobs:
           find ${{ runner.temp }}/docker/ -name '*.tar' -exec sh -c 'docker load < {}' \;
 
       - name: Set up test requirements
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: '3.10'
           cache: 'pip'
@@ -75,13 +75,13 @@ jobs:
 
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
-      - uses: nf-core/setup-nextflow@v1
+      - uses: nf-core/setup-nextflow@v2
 
       - name: Restore singularity setup
         id: restore-singularity-setup
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: /opt/hostedtoolcache/singularity/${{ env.SINGULARITY_VERSION }}/x64
           key: ${{ runner.os }}-singularity-${{ env.SINGULARITY_VERSION }}
@@ -93,13 +93,13 @@ jobs:
 
       - name: Restore singularity container images
         id: restore-singularity
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: ${{ env.NXF_SINGULARITY_CACHEDIR }}
           key: ${{ inputs.cache-key }}
 
       - name: Set up test requirements
-        uses: actions/setup-python@v3
+        uses: actions/setup-python@v5
         with:
           python-version: '3.10'
           cache: 'pip'

diff --git a/.github/workflows/preload-docker.yml b/.github/workflows/preload-docker.yml
@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Pull and save docker
         run: |
@@ -22,7 +22,7 @@ jobs:
 
       - name: Save docker
         id: cache-docker
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: ${{ runner.temp }}/docker
           key: docker-${{ github.sha }}
diff --git a/.github/workflows/preload-reference.yml b/.github/workflows/preload-reference.yml
@@ -17,7 +17,7 @@ jobs:
 
       - name: Cache reference data
         id: cache-ref
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: |
             ${{ env.ANCESTRY_TARGET_DIR }}/GRCh38_HAPNEST_TARGET_ALL.pgen

diff --git a/.github/workflows/preload-singularity.yml b/.github/workflows/preload-singularity.yml
@@ -15,11 +15,11 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Cache singularity setup
         id: cache-singularity-setup
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: /opt/hostedtoolcache/singularity/${{ env.SINGULARITY_VERSION }}/x64
           key: ${{ runner.os }}-singularity-${{ env.SINGULARITY_VERSION }}
@@ -44,7 +44,7 @@ jobs:
 
       - name: Cache singularity images
         id: cache-singularity-pull
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: ${{ env.NXF_SINGULARITY_CACHEDIR }}
           key: singularity-${{ github.sha }}
diff --git a/.github/workflows/standard-test.yml b/.github/workflows/standard-test.yml
@@ -33,13 +33,13 @@ jobs:
         run: echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> $GITHUB_OUTPUT
         id: extract_branch
 
-      - uses: nf-core/setup-nextflow@v1
+      - uses: nf-core/setup-nextflow@v2
         with:
           version: ${{ matrix.nxf_ver }}
 
       - name: Restore docker images
         id: restore-docker
-        uses: actions/cache/restore@v3
+        uses: actions/cache/restore@v4
         with:
           path: ${{ runner.temp }}/docker
           key: ${{ inputs.cache-key }}
@@ -70,13 +70,13 @@ jobs:
         run: echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> $GITHUB_OUTPUT
         id: extract_branch
 
-      - uses: nf-core/setup-nextflow@v1
+      - uses: nf-core/setup-nextflow@v2
         with:
           version: ${{ matrix.nxf_ver }}
 
       - name: Restore singularity setup
         id: restore-singularity-setup
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: /opt/hostedtoolcache/singularity/${{ env.SINGULARITY_VERSION }}/x64
           key: ${{ runner.os }}-singularity-${{ env.SINGULARITY_VERSION }}
@@ -88,7 +88,7 @@ jobs:
 
       - name: Restore singularity container images
         id: restore-singularity
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: ${{ env.NXF_SINGULARITY_CACHEDIR }}
           key: ${{ inputs.cache-key }}

diff --git a/.gitignore b/.gitignore
@@ -6,3 +6,7 @@ results/
 testing/
 testing*
 *.pyc
+.Rproj.user
+assets/report/renv/
+assets/report/report.Rproj
+.Rprofile
diff --git a/assets/NO_FILE b/assets/NO_FILE
diff --git a/assets/NO_FILE_INTERSECT_COUNT b/assets/NO_FILE_INTERSECT_COUNT
diff --git a/assets/examples/samplesheet.json b/assets/examples/samplesheet.json
@@ -1 +1,11 @@
-[{"sampleset":"cineca_synthetic_subset","vcf_path":null,"chrom":22,"bed":"https:\/\/gitlab.ebi.ac.uk\/nebfield\/test-datasets\/-\/raw\/master\/pgsc_calc\/cineca_synthetic_subset.bim","bim":"https:\/\/gitlab.ebi.ac.uk\/nebfield\/test-datasets\/-\/raw\/master\/pgsc_calc\/cineca_synthetic_subset.bed","fam":"https:\/\/gitlab.ebi.ac.uk\/nebfield\/test-datasets\/-\/raw\/master\/pgsc_calc\/cineca_synthetic_subset.fam"}]
+[
+    {
+        "sampleset": "hgdp",
+        "chrom": null,
+        "vcf_import_dosage": false,
+        "geno": "https:\/\/gitlab.ebi.ac.uk\/nebfield\/test-datasets\/-\/raw\/master\/pgsc_calc\/cineca_synthetic_subset.bed",
+        "pheno": "https:\/\/gitlab.ebi.ac.uk\/nebfield\/test-datasets\/-\/raw\/master\/pgsc_calc\/cineca_synthetic_subset.fam",
+        "variants": "https:\/\/gitlab.ebi.ac.uk\/nebfield\/test-datasets\/-\/raw\/master\/pgsc_calc\/cineca_synthetic_subset.bim",
+        "format": "bfile"
+    }
+]
diff --git a/assets/examples/scorefiles/customgrch37.txt b/assets/examples/scorefiles/customgrch37.txt
@@ -0,0 +1,6 @@
+#pgs_name=testlift
+#trait_reported=test
+#genome_build=GRCh37
+chr_name	chr_position	effect_allele	other_allele	effect_weight
+1	4859488	A	G	-0.0504139458332669
+1	5297845	G	A	0.040436175688854
diff --git a/assets/examples/scorefiles/customgrch38.txt b/assets/examples/scorefiles/customgrch38.txt
@@ -0,0 +1,6 @@
+#pgs_name=test
+#trait_reported=test
+#genome_build=GRCh38
+chr_name	chr_position	effect_allele	other_allele	effect_weight
+1	4799428	A	G	-0.0504139458332669
+1	5237785	G	A	0.040436175688854