Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fq lint module update: exit on failed validation #7000

Closed
wants to merge 4 commits into from
Closed

fq lint module update: exit on failed validation #7000

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b3c6ecb
Have fq lint module exit on failed validation
oligomyeggo Nov 15, 2024
4e8ef40
Update subworkflow
oligomyeggo Nov 15, 2024
cc79afc
Add module output to subworkflow
oligomyeggo Nov 15, 2024
98a2105
Remove custom changes from fq lint main.nf
oligomyeggo Nov 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions modules/nf-core/fq/lint/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
process FQ_LINT {
tag "$meta.id"
label 'process_low'
errorStrategy 'terminate'

conda "${moduleDir}/environment.yml"
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
Expand Down Expand Up @@ -29,5 +30,10 @@ process FQ_LINT {
"${task.process}":
fq: \$(echo \$(fq lint --version | sed 's/fq-lint //g'))
END_VERSIONS

if ! tail -n 1 ${prefix}.fq_lint.txt | grep -q 'fq-lint end'; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't change the module, generally we try to let the tools work as they do, without overlaying additional logic.

We can add some logic to the subworkflow + workflow that would filter out any libraries failing linting. You may already see logic in rnaseq that we use for trimming, strand failures etc, and we can use the same mechanism.

But my understanding is that fq_lint will return a non-zero error code on a failure? That should trigger a stop without this. Am I mistaken?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pinin4fjords Yeah, kind of. fq lint has different validators that are all assigned a different code, and if a FastQ file fails the linting, you will see in the log what code is failed on. You can't grep for the codes themselves in the log, as the log will say upfront what validators are enabled by code. That's why I had to grep for "fq-lint end" and check if it doesn't exist. That's the list line you'll see in the linting log if the linting was successful, so failed FastQ files won't have that in their log.

So, fq lint doesn't actually put out an error code itself. If we left the current module alone as is, then that process will successfully complete on any FastQ file, even corrupt ones, which defeats the purpose of adding the linting into the pipeline.

I am definitely open to trying out some different logic to filter out samples that failed linting. Are you thinking those samples would just filtered out, and the pipeline would keep going with the good samples, or that the pipeline would exit if it finds a sample that failed linting? We find that our users often want the pipeline to stop so that they can try reuploading (or redownloading from source and then reuploading) the offending FastQ file, which has worked in a lot of cases. Maybe another conditional that could be set as to if a user wants the pipeline to exit if a sample fails linting or if they want the pipeline to continue with successful samples?

Copy link
Member

@pinin4fjords pinin4fjords Nov 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is the following. Assume a bad fastq file, foo.fq, with content like:

@SEQ_ID_1
GATTTGGGGTTTTCCCAGTT
THIS SHOULD NOT BE HERE
IIIIIIIIIIIIIIIIIIII
@SEQ_ID_2
AGCTAGCTAGGCTAGCTAAG
+
JJJJJJJJJJJJJJJJJJJJ

The output of fq lint foo.fq is:

2024-11-17T17:03:29.770623Z  INFO fq::commands::lint: fq-lint start
2024-11-17T17:03:29.770645Z  INFO fq::commands::lint: validating single end read
2024-11-17T17:03:29.770651Z  INFO fq::validators: disabled validators: []
2024-11-17T17:03:29.770658Z  INFO fq::validators: enabled single read validators: ["[S003] NameValidator", "[S004] CompleteValidator", "[S002] AlphabetValidator", "[S001] PlusLineValidator", "[S005] ConsistentSeqQualValidator", "[S006] QualityStringValidator"]
2024-11-17T17:03:29.770668Z  INFO fq::validators: enabled paired read validators: []
2024-11-17T17:03:29.770671Z  INFO fq::commands::lint: starting validation
/data/foo.fq:3:1: [S001] PlusLineValidator: missing + prefix

This produces a non-zero error code:

$ echo $?
1

Because Nextflow runs processes with set -e turned on, the task will fail when it encounters that issue. Your addition will have no effect.

I see that fq has its own codes, but that wasn't what I meant.

But the other point I mentioned is important. Modules in nf-core should reflect the native behaviour of the tool as much as possible, otherwise users don't know what to expect. We shouldn't layer in additional errors etc as you were doing here (even if it worked). If this WAS a problem (and again, I don't think it is, because of set -e), we would need to introduce any custom logic at the pipeline level, perhaps as a 'local' module that parsed the error logs.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, ok, I follow now! Let me up take out the custom code from the module.

I think I got mixed up and noticed the pipeline was trying to re-run failed linting jobs, which I didn't want - unless the linting job was failing due to a pipeline resource error. How are those two different scenarios handled in nextflow, i.e., if the linting completes but the FastQ file was bad and it produces a non-zero error code, I would want the pipeline to fail. However, if the linting doesn't complete because of a resource issue or something, then I would want the pipeline to retry that task.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the specific rnaseq example, we'd just make sure to set the appropriate label so the process gets enough resource to start.

But to the general point, you can use dynamic retry strategies to catch exit codes reflecting e.g. OOM:

errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually you could PR against this, if you feel the current value isn't appropriate.

echo "ERROR: Linting failure detected for ${meta.id}. See ${prefix}.fq_lint.txt for details."
exit 1
fi
"""
}
12 changes: 12 additions & 0 deletions subworkflows/nf-core/fastq_qc_trim_filter_setstrandedness/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ include { BBMAP_BBSPLIT } from '../../../modules/nf-core/bbmap
include { CAT_FASTQ } from '../../../modules/nf-core/cat/fastq/main'
include { SORTMERNA } from '../../../modules/nf-core/sortmerna/main'
include { SORTMERNA as SORTMERNA_INDEX } from '../../../modules/nf-core/sortmerna/main'
include { FQ_LINT } from '../../../modules/nf-core/fq/lint/main'

include { FASTQ_SUBSAMPLE_FQ_SALMON } from '../fastq_subsample_fq_salmon'
include { FASTQ_FASTQC_UMITOOLS_TRIMGALORE } from '../fastq_fastqc_umitools_trimgalore'
Expand Down Expand Up @@ -106,6 +107,7 @@ workflow FASTQ_QC_TRIM_FILTER_SETSTRANDEDNESS {
umi_discard_read // integer: 0, 1 or 2
stranded_threshold // float: The fraction of stranded reads that must be assigned to a strandedness for confident assignment. Must be at least 0.5
unstranded_threshold // float: The difference in fraction of stranded reads assigned to 'forward' and 'reverse' below which a sample is classified as 'unstranded'
skip_linting // boolean: true/false

main:

Expand All @@ -114,6 +116,16 @@ workflow FASTQ_QC_TRIM_FILTER_SETSTRANDEDNESS {
ch_trim_read_count = Channel.empty()
ch_multiqc_files = Channel.empty()

//
// MODULE: Lint FastQ files
//
if(!skip_linting) {
FQ_LINT (
oligomyeggo marked this conversation as resolved.
Show resolved Hide resolved
ch_fastq_lint.map{ meta, fastqs -> [meta, fastqs.flatten()] }
)
ch_versions = ch_versions.mix(FQ_LINT.out.versions.first())
}

ch_reads
.branch {
meta, fastqs ->
Expand Down