Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log recommendations to Snakemake style guide #173

Merged
merged 3 commits into from
Nov 16, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions src/reference/snakemake-style-guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -193,3 +193,37 @@ Example:
--output-sequences {output.sequences:q} \
--output-metadata {output.metadata:q}
"""

Log standard out and error output to log files and the terminal
===============================================================

Use `the Snakemake log directive <https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#log-files>`_ for each rule that writes output to either standard out or error and direct output to the corresponding log file.
Use the ``tee`` command to ensure that output gets written to both the log file and the terminal, so users can track their workflow progress interactively and use the log file later for debugging.

Example:

.. code-block:: python

rule filter:
input:
metadata="results/metadata.tsv",
output:
metadata="results/filtered_metadata.tsv",
log:
"logs/filter.txt"
shell:
"""
augur filter \
--metadata {input.metadata} \
--output-metadata {output.metadata} 2>&1 | tee {log}
"""

Before using ``tee``, ensure that your workflow uses `bash's pipefail option <http://redsymbol.net/articles/unofficial-bash-strict-mode/>`_, so successful ``tee`` execution does not mask errors from earlier commands in the pipe.
Snakemake uses bash's strict mode by default, so the pipefail option should be enabled by default.
However, some workflows may override the defaults `locally at specific rules <https://snakemake.readthedocs.io/en/stable/project_info/faq.html#my-shell-command-fails-with-exit-code-0-from-within-a-pipe-what-s-wrong>`_ or globally as with `a custom shell prefix <https://github.com/nextstrain/ncov/pull/751>`_.

Run workflows with ``--show-failed-logs``
=========================================

Run workflows with the ``--show-failed-logs`` which will print the logs for failed jobs to the terminal when the workflow exits.
This pattern helps users identify error messages without first finding the corresponding log file.