Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error with CPU Part 1, execution of Full Multiome Tutorial #69

Closed
4 tasks done
emix-c opened this issue Oct 16, 2024 · 4 comments
Closed
4 tasks done

Error with CPU Part 1, execution of Full Multiome Tutorial #69

emix-c opened this issue Oct 16, 2024 · 4 comments

Comments

@emix-c
Copy link

emix-c commented Oct 16, 2024

Checks before submitting the issue

  • I installed the latest Dictys version successfully
  • I downloaded the whole folder for the tutorial
  • I ran the tutorial cells and notebooks (if more than one) in the correct order and only once
  • The preceeding cells and notebooks finished successfully (i.e. the reported error is the first encountered)

Describe the error

  • How did you install Dictys (which option)?
    I installed using conda.
  • Which tutorial?
    The full multiome tutorial.
  • What (full message) and where (cell and notebook being executed) was the error?
    I am encountering an error when trying to run the CPU Part1, execution block.
    I am specifically running into an error with Subset5, I am not seeing this error with any else.
    I adjusted the number of threads so that my system could handle it. Before, when I had it set to -j 36, some of the make commands to make expression0.tsv.gz were getting killed.
    Whenever I made adjustments, I deleted the entire tmp_static folder and reran the below block.

%%bash
set -eo pipefail
cd ..
#Run CPU part of inference
make -f makefiles/static.mk -j 8 -k cpu || true

I ran into this error:
Traceback (most recent call last):
File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in
docstringrunner(package)
File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner
run_args(pkgname,funcs,args)
File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args
return func(*a,**ka)
File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 67, in macs2
d2 = shell.cmdfile(cmd,[],infiles={'cellnames.txt': namestxt},quiet=False,cd=True)
File "/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/utils/shell.py", line 181, in cmdfile
raise RuntimeError('Command failed, possibly due to program error: ' + cmda)
RuntimeError: Command failed, possibly due to program error: /home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/scripts/chromatin_macs2.sh cellnames.txt /home/groups/CEDAR/chaoe/projects/Dictys/data/bams /home/groups/CEDAR/chaoe/projects/Dictys/tmp_static/Subset5/reads.bam /home/groups/CEDAR/chaoe/projects/Dictys/tmp_static/Subset5/reads.bai /home/groups/CEDAR/chaoe/projects/Dictys/tmp_static/Subset5/peaks.bed hs 0.05 8
[bam_sort_core] merging from 2 files and 8 in-memory blocks...
/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/scripts/chromatin_macs2.sh: line 38: 1597134 Killed samtools sort -o "$output_bam" $nodes2 -l 1 02-filtered.bam

make: *** [makefiles/common.mk:138: tmp_static/Subset5/reads.bam] Error 1
makefiles/common.mk:138: warning: pattern recipe did not update peer target 'tmp_static/Subset5/peaks.bed'.
makefiles/common.mk:138: warning: pattern recipe did not update peer target 'tmp_static/Subset5/reads.bai'.

  • Did you deviate from any of the tutorial instructions?
    No I didn't.

Optional steps (may accelerate troubleshooting)
I did not do any optional steps.

  • Attach the error notebook and all preceeding notebooks with run output
@lingfeiwang
Copy link
Collaborator

Hi emix-c,

Probably your computer ran out of memory. Could you try again with -j 4 instead?

Lingfei

@emix-c
Copy link
Author

emix-c commented Oct 22, 2024

Hi Lingfei,
Yes I got it working by reducing the number of jobs!
I do have a quick question, since I'm running pytorch with CPU only, I only run 1.1 CPU part execution and then 4.1 Aggregating Network part execution, so I skip 2.1 GPU execution and 3.1 CPU part 2, execution, is that correct?

I ran 1.1 CPU part execution and tried to run 4.1 but got these errors.

WARNING:2724116:2024-10-22 09:36:31,129:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,156:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,158:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,159:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,602:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset11 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,603:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,604:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,682:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset14 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,683:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,683:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,683:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,684:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,684:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,684:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,684:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,685:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,685:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,685:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,686:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,686:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,686:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,687:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,687:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:36:31,687:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory')
WARNING:2724116:2024-10-22 09:37:00,168:/home/exacloud/gscratch/CEDAR/chaoe/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:510:Removing 11/14 failed subsets.

@lingfeiwang
Copy link
Collaborator

lingfeiwang commented Oct 22, 2024

These errors are not anticipated in the tutorial notebook. Did you specify DEVICE to cpu in 2-makefile.ipynb first? Are these the first error you encountered? I suggest to rerun the whole tutorial in a new folder to find and report the first error you encounter.

Copy link

Stale issue message

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants