Error finding Python in HPC environment

[jreimertz]

Bug report

Nextflow does not appear to be able to find the correct Python3 in my HPC environment. I first discovered this when trying to run a Python script in my workflow using a shebang in the script I get the error:

Command error:
/usr/bin/env: python3: No such file or directory

Upon looking further into this I was able to determine that Nextflow only seems to be able to find Python version 2.7 (which is not even installed in my HPC environment). This doesn't appear to be a PATH issue as searching for python3 with which python3 returns all of the locations on the PATH two of which include python3

Steps to reproduce this error

I've made a simple nf script to test this issue:

#!/usr/bin/env nextflow

params.test_py = "$projectDir/test_python.py"

println "test python script: $params.test_py"

process testPython {
    input:
    file testScript

    output:
    path 'py_out.txt'

    shell:
    """
    ./$testScript >> 'py_out.txt'
    """
}

workflow {
    testPython(file(params.test_py))
    }

Where test_python.py is:

#!/usr/bin/env python3

import sys

print('This is a Python script')
print(sys.version)

This outputs the following error:

ERROR ~ Error executing process > 'testPython'

Caused by:
Process testPython terminated with an error exit status (126)

Command executed:

./test_python.py >> 'py_out.txt'

Command exit status:
126

Command output:
(empty)

Command error:
.command.sh: ./test_python.py: /usr/bin/env: python3: No such file or directory

Alternatively, if I change the shebang in test_python.py to /usr/bin/env python the script runs and the output looks like:

This is a Python script
2.7.5 (default, Jun 28 2022, 15:30:04)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]

When testing to find python3 I get the following error:

ERROR ~ Error executing process > 'testPython'

Caused by:
Process testPython terminated with an error exit status (1)

Command executed:

which python3

Command exit status:
1

Command output:
(empty)

Command error:
which: no python3 in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/projectDir/bin)

However if I check each of these locations on the PATH I can find python3 in both /usr/bin and /projectDir/bin

Compared to if I use which python which returns:

/usr/bin/python

Further confusing to me is that if within the nf script I use whereis python3 Nextflow can find it in /projectDir/bin

Environment

• Nextflow version: 24.04.3
• Java version: OpenJDK 21.0.3 - Zulu21
• Operating system: Rocky-Linux-8
• Bash version: GNU bash, version 4.4.20(1)-release (x86_64-redhat-linux-gnu)

Additional context

I am using a HPC environment so I don't have full sudo permissions, and for that reason Nextflow is installed in a folder I have permissions to work with opposed to the HPC core bin directory. Also worth noting that I can run this script fine from the commandline with python3, so I believe this is a Nextflow error

[bentsherman]

Which executor are you using? If the jobs are running on a different node than where you launch Nextflow, they might have a different environment and not have access to all of the commands you can access from the submitter node. I would try to run which python3 in the same environment where the jobs are executed.

[jreimertz]

I'm using slurm. Currently I'm using a srun job to test the workflow, but for the full pipeline I would use sbatch to submit a shell script with the nextflow command. From the commandline if I run which python3 in the srun job I get:

/programs/x86_64-linux/system/biogrids_bin/python3

[bentsherman]

I would look at the .command.run script that is generated for the process in the work directory. It is the job script that is submitted by Nextflow, so you can try to submit it manually to see if it still fails, and inspect the script to see if it is doing something that conflicts with the job environment

[jreimertz]

Submitting .command.run also fails with the output in .command.log:

which: no python3 in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/projectDir/bin)

Looking through .command.run the only thing that seems like it could be affecting the environment is that I have an apptainer definition in my config file, but since I've been running Nextflow without the -with-apptainer flag my understanding was that this shouldn't be affecting the environment.

For reference this is the chunk I'm referring to:

nxf_launch() {
    set +u; env - PATH="$PATH" ${TMP:+APPTAINERENV_TMP="$TMP"} ${TMPDIR:+APPTAINERENV_TMPDIR="$TMPDIR"} ${NXF_TASK_WORKDIR:+APPTAINERENV_NXF_TASK_WORKDIR="$NXF_TASK_WORKDIR"} apptainer exec --no-home --pid -B /projectDir -B /temp_work/work/ef/dcd7a7bc04724cdaebeabacbe7ac1b /projectDir/cellranger_bcl2fastq_singularity_container_latest.sif /bin/bash -c "cd $NXF_TASK_WORKDIR; eval $(nxf_container_env); /bin/bash -ue /temp_work/work/ef/dcd7a7bc04724cdaebeabacbe7ac1b/.command.sh"
}

[stevekm]

are you sure this is not a version of python that exists inside of your container /projectDir/cellranger_bcl2fastq_singularity_container_latest.sif ?

[jreimertz]

This ended up being the issue. After commenting out the lines in my config file related to the container Nextflow was able to find Python3 with no issues

[pditommaso]

Bare in mind that the environment in the batch job can be different from the one you can access with the interactive shell.

[jreimertz]

Thank you for your help with sorting this out, I thought I had exhausted all troubleshooting options but clearly not

[stevekm]

Bare in mind that the environment in the batch job can be different from the one you can access with the interactive shell.

this is excellent advice. Additionally, some more things I find that I need to always remember;

• verify that the container is being used ; that its the container you expected to be used ; that the software inside the container is what you expected
  • Singularity sometimes pulls in environment variables and sometimes mounts user $HOME, or sometimes not, when it runs, which can affect the binaries and libraries (especially bad with R which looks in some lib dirs under ~ automatically
• environment on different nodes can be inconsistent
• interactive vs. non-interactive logins / command executions can be different (both for scheduler jobs, and for normal shell sessions)
  • some ~/.bashrc these days ship with sections that immediately quit if they detect they are not running an interactive session, which can cause your env to not get configured as you expected

basically, you gotta do sanity-checking of your entire environment from top to bottom, and especially inside the context of which your job is running. Most all errors tend to get identified this way, eventually.