Run the workflow step by step

[thibaulttabarin]

We need documentation on how to run the workflow step by step or on specific rule, mainly for development purpose.

[johnbradley]

@thibaulttabarin I put together a rough attempt at explaining how to run each step one at a time here:
https://github.com/hdr-bgnn/BGNN_Snakemake/blob/38-step-by-step/DEVELOPMENT.md. Am I on the right track, or is there some nuance I am missing? Thanks.

[thibaulttabarin]

@johnbradley, Sounds good, but how you would process the whole dataset for a single rule... Let say I want to reprocess all the images (from Images/) with an updated version of generate_metadata and check the result before I produce the whole set of results (Cropped/, Segmented/, Morphology/). How would you do that? In your documentation, there is only the option of reprocessing a single input/output.

[johnbradley]

@thibaulttabarin The most direct way to run the workflow through a step is using the --until <stepname> snakemake command line argument. For example:

snakemake -c1 --use-singularity --config list=List/list_test.csv --until generate_metadata

See --until in https://snakemake.readthedocs.io/en/stable/executing/cli.html.

---

Alternatively if you just want to process some of the files until this step you can specify multiple targets as well:

snakemake -c1 --use-singularity --config list=List/list_test.csv -- Metadata/INHS_FISH_14841.json Metadata/INHS_FISH_38136.json

---

I would note that the files being generated are based on the contents of --config list=List/list_test.csv and not whatever files happen to reside in Images/.

[johnbradley]

I updated the DEVELOPMENT documentation with the --until argument: https://github.com/hdr-bgnn/BGNN_Snakemake/blob/38-step-by-step/DEVELOPMENT.md#stop-early

[thibaulttabarin]

it doesn't work as expected... only run one file and stop... and after that it throws this error
Using shell: /usr/bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Traceback (most recent call last):
File "/users/PAS2136/thibaulttabarin/.conda/envs/snakemake/lib/python3.10/site-packages/snakemake/init.py", line 726, in snakemake
success = workflow.execute(
File "/users/PAS2136/thibaulttabarin/.conda/envs/snakemake/lib/python3.10/site-packages/snakemake/workflow.py", line 1083, in execute
logger.run_info("\n".join(dag.stats()))
File "/users/PAS2136/thibaulttabarin/.conda/envs/snakemake/lib/python3.10/site-packages/snakemake/dag.py", line 2377, in stats
"min threads": min(min_threads.values()),
ValueError: min() arg is an empty sequence

[thibaulttabarin]

ok it worked after remove the last folder Morphology/

[johnbradley]

@thibaulttabarin I think you hit this bug: snakemake/snakemake#1815. It sounds like there is a problem if you run with --until when there is nothing to be done.

[thibaulttabarin]

yep, I agree, sounds like it. For this step by step part I feel it doesn't work the way I_would_like/it_is_conveniant_to_develop_with... Let talk about tomorrow at the meeting