Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GPU-based tests #6953

Merged
merged 24 commits into from
Nov 16, 2024
Merged

Add GPU-based tests #6953

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
d946f4d
split nf-test section into its own action.yml and add gpu test cases
mashehu Nov 7, 2024
473857e
add gpu profile
mashehu Nov 7, 2024
09bc2d3
Update test.yml
mashehu Nov 8, 2024
3514f77
add checkout action before running nf-test action
mashehu Nov 8, 2024
88fbfe3
Merge branch 'add-gpu-tests' of github.com:mashehu/modules into add-g…
mashehu Nov 8, 2024
51f66e1
add miniconda-setup step
mashehu Nov 9, 2024
616e0ef
Update tests/config/nextflow.config
mashehu Nov 11, 2024
7cb14f4
try to fix composite action
mashehu Nov 15, 2024
3a164c9
Merge branch 'add-gpu-tests' of github.com:mashehu/modules into add-g…
mashehu Nov 15, 2024
36897d8
Merge branch 'master' into add-gpu-tests
mashehu Nov 15, 2024
0bdcbb2
add profile info to include step, to avoid empty profile values
mashehu Nov 15, 2024
18de958
Merge branch 'add-gpu-tests' of github.com:mashehu/modules into add-g…
mashehu Nov 15, 2024
e9d237b
touch fastqc to kick of some tests
mashehu Nov 15, 2024
004b9a6
add gpu profile to nf-test.config
mashehu Nov 15, 2024
834614a
Merge branch 'master' of github.com:nf-core/modules into add-gpu-tests
mashehu Nov 15, 2024
660d68f
fix miniconda setup step
mashehu Nov 15, 2024
f90f4c2
remove miniconda version pin
mashehu Nov 15, 2024
12115c4
try to exclude all non-gpu tests
mashehu Nov 15, 2024
236276d
remove changed paths
mashehu Nov 15, 2024
88494a6
add check for changes in parabricks
mashehu Nov 15, 2024
d50b3cb
fix pip error
mashehu Nov 15, 2024
6fa8d59
add setup-python to main script
mashehu Nov 15, 2024
b2de5f7
remove conda defaults channels
mashehu Nov 15, 2024
4609f2c
Merge branch 'master' into add-gpu-tests
sateeshperi Nov 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions .github/actions/nf-test-action/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
name: "NF Test Action"
description: "Runs nf-test with common setup steps"
inputs:
path:
description: "Path to test"
required: true
profile:
description: "Profile to use"
required: true
runners:
description: "Runners to use"
required: false
default: "self-hosted"

runs:
using: "composite"
steps:
- uses: actions/checkout@v4

- uses: actions/setup-java@v4
with:
distribution: "temurin"
java-version: "17"

- name: Setup Nextflow
uses: nf-core/setup-nextflow@v2

- name: Install nf-test
uses: nf-core/setup-nf-test@v1
with:
version: "0.9.2"
install-pdiff: true

- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5
with:
python-version: "3.11"

- name: Setup apptainer
if: ${{ inputs.profile == 'singularity' }}
uses: eWaterCycle/setup-apptainer@main

- name: Set up Singularity
if: ${{ inputs.profile == 'singularity' }}
shell: bash
run: |
mkdir -p $NXF_SINGULARITY_CACHEDIR
mkdir -p $NXF_SINGULARITY_LIBRARYDIR

- name: Conda setup
if: ${{inputs.profile == 'conda'}}
shell: bash
with:
miniconda-version: "23.5.1"
auto-update-conda: true
conda-solvers: libmamba
run: |
echo $(realpath $CONDA)/condabin >> $GITHUB_PATH
echo $(realpath python) >> $GITHUB_PATH

# Set up secrets
- name: Set up nextflow secrets
# TODO Only run if the tag includes `sentieon`
if: ${{ secrets.SENTIEON_ENCRYPTION_KEY }} != null && ${{ secrets.SENTIEON_LICENSE_MESSAGE }} != null
shell: bash
run: |
nextflow secrets set SENTIEON_AUTH_DATA $(python3 tests/modules/nf-core/sentieon/license_message.py encrypt --key "${{ secrets.SENTIEON_ENCRYPTION_KEY }}" --message "${{ secrets.SENTIEON_LICENSE_MESSAGE }}")

- name: Run nf-test
shell: bash
env:
SENTIEON_LICSRVR_IP: ${{ secrets.SENTIEON_LICSRVR_IP }}
SENTIEON_AUTH_MECH: "GitHub Actions - token"
run: |
if [ "${{ inputs.profile }}" == "docker" ]; then
PROFILE="docker_self_hosted"
else
PROFILE=${{ inputs.profile }}
fi

NFT_WORKDIR=~ \
nf-test test \
--profile=${{ inputs.profile }} \
--tap=test.tap \
--verbose \
${{ inputs.path }}

- uses: pcolby/tap-summary@0959cbe1d4422e62afc65778cdaea6716c41d936 # v1
if: ${{ inputs.path != '' }}
with:
path: >-
test.tap

- name: Clean up
if: always()
shell: bash
run: |
sudo rm -rf /home/ubuntu/tests/
135 changes: 36 additions & 99 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,7 @@ jobs:
with:
fetch-depth: 2 # To retrieve the preceding commit.

# TODO: change back to using dorny/paths-filter when https://github.com/dorny/paths-filter/pull/133 is implemented
- uses: mirpedrol/paths-filter@main
- uses: dorny/paths-filter@v3
id: filter
with:
filters: "tests/config/pytest_modules.yml"
Expand Down Expand Up @@ -485,7 +484,7 @@ jobs:
!${{ github.workspace }}/.singularity

nf-test:
runs-on: ${{ github.event.inputs.runners || 'self-hosted' }}
runs-on: ${{ github.event.inputs.runners || 'self-hosted' }}
name: nf-test
needs: [nf-test-changes]
if: ( needs.nf-test-changes.outputs.paths != '[]' )
Expand Down Expand Up @@ -666,108 +665,45 @@ jobs:
path: subworkflows/nf-core/fasta_newick_epang_gappa
env:
NXF_ANSI_LOG: false
NFTEST_VER: "0.9.0"
SENTIEON_LICENSE_MESSAGE: ${{ secrets.SENTIEON_LICENSE_MESSAGE }}
SENTIEON_ENCRYPTION_KEY: ${{ secrets.SENTIEON_ENCRYPTION_KEY }}

steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- uses: actions/setup-java@8df1039502a15bceb9433410b1a100fbe190c53b # v4
with:
distribution: "temurin"
java-version: "17"
- name: Setup Nextflow
uses: nf-core/setup-nextflow@v2

- name: Install nf-test
uses: nf-core/setup-nf-test@v1
- name: Run nf-test Action
uses: ./.github/actions/nf-test-action
with:
version: ${{ env.NFTEST_VER }}

- name: Setup apptainer
if: matrix.profile == 'singularity'
uses: eWaterCycle/setup-apptainer@main
path: ${{ matrix.path }}
profile: ${{ matrix.profile }}

- name: Set up Singularity
if: matrix.profile == 'singularity'
run: |
mkdir -p $NXF_SINGULARITY_CACHEDIR
mkdir -p $NXF_SINGULARITY_LIBRARYDIR

- name: Set up Python
uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5
with:
python-version: "3.11"

- uses: actions/cache@6849a6489940f00c2f30c0fb92c6274307ccb58a # v4
id: cache-pip-pdiff
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-pdiff
restore-keys: |
${{ runner.os }}-pip-pdiff

- name: Install Python dependencies
run: python -m pip install --upgrade pip pdiff cryptography

- name: Set up miniconda
if: matrix.profile == 'conda'
uses: conda-incubator/setup-miniconda@d2e6a045a86077fb6cad6f5adf368e9076ddaa8d # v3
with:
miniconda-version: "latest"
auto-update-conda: true
channels: conda-forge,bioconda

- name: Conda setup
if: matrix.profile == 'conda'
run: |
conda clean -a
conda install -n base conda-libmamba-solver
conda config --set solver libmamba
echo $(realpath $CONDA)/condabin >> $GITHUB_PATH
echo $(realpath python) >> $GITHUB_PATH

# Set up secrets
- name: Set up nextflow secrets
# TODO Only run if the tag includes `sentieon`
if: env.SENTIEON_ENCRYPTION_KEY != null && env.SENTIEON_LICENSE_MESSAGE != null
run: |
nextflow secrets set SENTIEON_AUTH_DATA $(python3 tests/modules/nf-core/sentieon/license_message.py encrypt --key "${{ secrets.SENTIEON_ENCRYPTION_KEY }}" --message "${{ secrets.SENTIEON_LICENSE_MESSAGE }}")
nf-test-gpu:
runs-on: "gpu"
name: nf-test-gpu
needs: [nf-test-changes]
if: ( needs.nf-test-changes.outputs.paths != '[]' )
strategy:
fail-fast: false
matrix:
path: ["${{ fromJson(needs.nf-test-changes.outputs.paths) }}"]
profile: [conda, docker_self_hosted, singularity]
include:
- path: modules/nf-core/parabricks/applybqsr
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this trigger tests for these three modules whenever if: ( needs.nf-test-changes.outputs.paths != '[]' ) is true.

- path: modules/nf-core/parabricks/fq2bam
- path: modules/nf-core/parabricks/fq2bammeth
exclude:
- path: modules/nf-core/nf-test
- profile: conda
path: modules/nf-core/parabricks/fq2bam
- profile: conda
path: modules/nf-core/parabricks/fq2bammeth
- profile: conda
path: modules/nf-core/parabricks/applybqsr
env:
NXF_ANSI_LOG: false

# Test the module
- name: Run nf-test
if: ${{ matrix.path != '' }}
env:
NFT_DIFF: "pdiff"
NFT_DIFF_ARGS: "--line-numbers --width 120 --expand-tabs=2"
SENTIEON_LICSRVR_IP: ${{ secrets.SENTIEON_LICSRVR_IP }}
SENTIEON_AUTH_MECH: "GitHub Actions - token"
run: |
# use "docker_self_hosted" if it runs on self-hosted runner and matrix.profile=docker
if [ "${{ matrix.profile }}" == "docker" ]; then
PROFILE="docker_self_hosted"
else
PROFILE=${{ matrix.profile }}
fi

NFT_WORKDIR=~ \
nf-test test \
--profile=${{ matrix.profile }} \
--tap=test.tap \
--verbose \
${{ matrix.path }}

- uses: pcolby/tap-summary@0959cbe1d4422e62afc65778cdaea6716c41d936 # v1
if: ${{ matrix.path != '' }}
steps:
- name: Run nf-test Action
uses: ./.github/actions/nf-test-action
with:
path: >-
test.tap

- name: Clean up
if: always()
run: |
sudo rm -rf /home/ubuntu/tests/
path: ${{ matrix.path }}
profile: ${{ matrix.profile }},gpu

confirm-pass:
runs-on: ubuntu-latest
Expand All @@ -781,6 +717,7 @@ jobs:
pytest,
nf-test-changes,
nf-test,
nf-test-gpu,
]
if: always()
steps:
Expand Down
4 changes: 4 additions & 0 deletions tests/config/nextflow.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,10 @@ if ("$PROFILE" == "singularity") {
podman.enabled = true
podman.userEmulation = true
podman.runOptions = "--runtime crun --platform linux/x86_64 --systemd=always"
} else if ("$PROFILE" == "gpu") {
mashehu marked this conversation as resolved.
Show resolved Hide resolved
docker.runOptions = '-u $(id -u):$(id -g) --gpus all'
mashehu marked this conversation as resolved.
Show resolved Hide resolved
apptainer.runOptions = '--nv'
singularity.runOptions = '--nv'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer this stuff to go in the configuration of the individual tools, so it can function as a reference implementation for users. Adding it here feels too "secret sauce" that will be hard for other people to follow.

Plus we should use process.containerOptions for process specific stuff.

Or we could add label 'gpu' to achieve a similar thing.

process {
    withLabel "gpu" {
        process.containerOptions = "--gpus all"
    }
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for label

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we handle the singularity/apptainer option --nv then?

} else {
docker.enabled = true
docker.userEmulation = false
Expand Down
Loading