Merge branch 'dev' into Conditional-tidk-flags

README.md

@@ -14,8 +14,7 @@
 
 The pipeline takes a list of genomes and annotations (from raw files or Refseq IDs), and runs commonly used tools to assess their quality.
 
-There will be three different ways you can run this pipeline. 1. Genome only, 2. Annotation only, or 3. Genome and Annotation. 
-**Only Genome plus Annotation is functional**
+There are three different ways you can run this pipeline. 1. Genome only, 2. Annotation only, or 3. Genome and Annotation. **Only Genome plus Annotation is functional**
 
 <!-- TODO nf-core:
 For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction
@@ -49,6 +48,10 @@ For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#intr
 2. Describes your annotation : `[AGAT]`: Gene, feature, length, averages, counts.
 3. Summary with MulitQC.
 
+In addition to the three different modes described above, it is also possible to run the pipeline with or without sequencing reads. When supplying sequencing reads, Merqury can also be run. [Merqury](https://github.com/marbl/merqury) is a tool for genome quality assessment that uses k-mer counts from raw sequencing data to evaluate the accuracy and completeness of a genome assembly. Meryl is the companion tool that efficiently counts and stores k-mers from sequencing reads, enabling Merqury to estimate metrics like assembly completeness and base accuracy. These tools provide a k-mer-based approach to assess assembly quality, helping to identify potential errors or gaps.​
+
+To run the pipeline with reads, you must supply a single FASTQ file for each genome in the samplesheet. It is assumed that reads used to create the assembly are from long read technology such as PacBio or ONT, and are therefore single end. If reads are in a .bam file, they must be converted to FASTQ format first. If you have paired end reads, these must be interleaved first.
+
 ## Usage
 
 > [!NOTE]
@@ -57,31 +60,37 @@ For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#intr
 First, prepare a `samplesheet.csv`, where your input data points to genomes + or annotations:
 
 ```csv
-species,refseq,fasta,gff
-Homo_sapiens,,/path/to/genome.fasta,/path/to/annotation.gff3
-Gorilla_gorilla,,/path/to/genome.fasta,
-Pan_paniscus,,/path/to/genome.fasta,/path/to/annotation.gff3
+species,refseq,fasta,gff,fastq
+Homo_sapiens,,/path/to/genome.fasta,/path/to/annotation.gff3,/path/to/reads.fq.gz
+Gorilla_gorilla,,/path/to/genome.fasta,,/path/to/reads.fq.gz
+Pan_paniscus,,/path/to/genome.fasta,/path/to/annotation.gff3,/path/to/reads.fq.gz
 ```
 
 Or to Refseq IDs of your species:
 
 ```csv
-species,refseq,fasta,gff
-Pongo_abelii,GCF_028885655.2,,
-Macaca_mulatta,GCF_003339765.1,,
+species,refseq,fasta,gff,fastq
+Pongo_abelii,GCF_028885655.2,,,/path/to/reads.fq.gz
+Macaca_mulatta,GCF_003339765.1,,,/path/to/reads.fq.gz
 ```
 
 You can mix the two input types **(in development)**.
 
 Each row represents a species, with its associated genome, gff or Refseq ID (to autodownload the genome + gff).
 
-You can run the pipeline using test profiles or example input samplesheets:
+You can run the pipeline using test profiles or example input samplesheets. To run a test set with a samplesheet containing reads:
 
 ```
 nextflow run main.nf -resume -profile docker,test --outdir results
 ```
 
-or
+If you supply sequencing reads in your samplesheet, you can still disable merqury by using the parameter `--merqury_skip true`. Alternatively, use a different test profile that does _not_ contain sequencing reads:
+
+```
+nextflow run main.nf -resume -profile docker,test_nofastq --outdir results
+```
+
+To run this pipeline on an example samplesheet included in the repo assets:
 
 ```
 nextflow run main.nf -resume -profile docker --input assets/samplesheet.csv --outdir results
@@ -106,6 +115,8 @@ ecoflow/genomeqc was originally written by Chris Wyatt, Fernando Duarte.
 
 We thank the following people for their extensive assistance in the development of this pipeline:
 
+- [Stephen Turner](https://github.com/stephenturner/) ([Colossal Biosciences](https://colossal.com/))
+
 <!-- TODO nf-core: If applicable, make list of people who have also contributed -->
 
 ## Contributions and Support

assets/samplesheet.csv

@@ -1,5 +1,5 @@
-species,refseq,fasta,gff
-Vespula_vulgaris,GCF_905475345.1,,
-Vespa_velutina,GCF_912470025.1,,
-Apis_mellifera,GCF_003254395.2,,
-Osmia_bicornis,GCF_907164935.1,,
+species,refseq,fasta,gff,fastq
+Vespula_vulgaris,GCF_905475345.1,,,
+Vespa_velutina,GCF_912470025.1,,,
+Apis_mellifera,GCF_003254395.2,,,
+Osmia_bicornis,GCF_907164935.1,,,

assets/schema_input.json

@@ -14,17 +14,22 @@
             },
             "refseq": {
                 "type": "string",
-                "errorMessage": "FastQ file for reads 1 must be provided, cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'"
+                "errorMessage": "RefSeq accession number"
             },
             "fasta": {
                 "type": "string",
                 "format": "file-path",
-                "errorMessage": "FastQ file for reads 1 must be provided, cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'"
+                "errorMessage": "FASTA file with genome assembly"
             },
             "gff": {
                 "type": "string",
                 "format": "file-path",
-                "errorMessage": "FastQ file for reads 2 cannot contain spaces and must have extension '.fq.gz' or '.fastq.gz'"
+                "errorMessage": "GFF file with genome annotation"
+            },
+            "fastq": {
+                "type": "string",
+                "format": "file-path",
+                "errorMessage": "Single compressed FASTQ file, must have extension '.fq.gz' or '.fastq.gz'"
             }
         },
         "required": ["species"]

bin/busco_2_table.py

@@ -1,3 +1,4 @@
+#!/usr/bin/python3
 import pandas as pd
 import argparse
 

bin/plot_tree_summary.R

@@ -1,3 +1,4 @@
+#!/usr/bin/Rscript
 # Load necessary libraries
 if (!requireNamespace("argparse", quietly = TRUE)) {
   install.packages("argparse")

bin/plot_tree_summary2.R

@@ -1,3 +1,4 @@
+#!/usr/bin/Rscript
 # Load necessary libraries
 if (!requireNamespace("argparse", quietly = TRUE)) {
   install.packages("argparse")

bin/quast_2_table.py

@@ -1,3 +1,4 @@
+#!/usr/bin/python3
 import pandas as pd
 import argparse
 import os

conf/modules.config

@@ -56,6 +56,28 @@ process {
         ]
     } 
 
+    withName: 'MERQURY_MERQURY' {
+        publishDir = [
+            path: { "${params.outdir}/merqury" },
+            mode: params.publish_dir_mode,
+            saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
+        ]
+    } 
+    withName: 'MERYL_COUNT' {
+        publishDir = [
+            path: { "${params.outdir}/meryl_count" },
+            mode: params.publish_dir_mode,
+            saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
+        ]
+    } 
+    withName: 'MERYL_UNIONSUM' {
+        publishDir = [
+            path: { "${params.outdir}/meryl_unionsum" },
+            mode: params.publish_dir_mode,
+            saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
+        ]
+    } 
+
     withName: 'SORT_BY_LENGTH' {
         ext.prefix = { "${meta.id}_sorted" }
     }
@@ -73,4 +95,11 @@ process {
         ]
     }
 
+    withName: 'LONGEST' {
+        publishDir = [
+            path: { "$params.outdir/output_data/longest" }, 
+            mode: params.publish_dir_mode,
+            saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
+        ]
+    }
 }

conf/test.config

@@ -24,4 +24,7 @@ params {
     // TODO nf-core: Give any required params for the test so that command line flags are not needed
     input  = params.pipelines_testdata_base_path + 'genomeqc/samplesheet/input_bacteria.csv'
 
+    // Run meryl+merqury
+    merqury_skip = false
+
 }

conf/test_full.config

@@ -12,7 +12,7 @@
 
 params {
     config_profile_name        = 'Full test profile'
-    config_profile_description = 'Full test dataset of hymenpotera genomes'
+    config_profile_description = 'Full test dataset of Hymenoptera genomes'
 
     input = 'assets/samplesheet.csv'
 

conf/test_nofastq.config

@@ -0,0 +1,30 @@
+/*
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    Nextflow config file for running minimal tests
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    Defines input files and everything required to run a fast and simple pipeline test.
+
+    Use as follows:
+        nextflow run ecoflow/genomeqc -profile test,<docker/singularity> --outdir <OUTDIR>
+
+----------------------------------------------------------------------------------------
+*/
+
+params {
+    config_profile_name        = 'Test profile'
+    config_profile_description = 'Minimal test dataset to check pipeline function'
+
+    // Limit resources so that this can run on GitHub Actions
+    max_cpus   = 2
+    max_memory = '6.GB'
+    max_time   = '6.h'
+
+    // Input data
+    // TODO nf-core: Specify the paths to your test data on nf-core/test-datasets
+    // TODO nf-core: Give any required params for the test so that command line flags are not needed
+    input  = params.pipelines_testdata_base_path + 'genomeqc/samplesheet/input_bacteria_nofastq.csv'
+
+    // Don't try to skip merqury, but since you don't have reads, it shouldn't get run anyway
+    merqury_skip = false
+
+}

modules.json

@@ -30,6 +30,21 @@
             "git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
             "installed_by": ["modules"]
           },
+          "merqury/merqury": {
+            "branch": "master",
+            "git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
+            "installed_by": ["modules"]
+          },
+          "meryl/count": {
+            "branch": "master",
+            "git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
+            "installed_by": ["modules"]
+          },
+          "meryl/unionsum": {
+            "branch": "master",
+            "git_sha": "666652151335353eef2fcd58880bcef5bc2928e1",
+            "installed_by": ["modules"]
+          },
           "multiqc": {
             "branch": "master",
             "git_sha": "b7ebe95761cd389603f9cc0e0dc384c0f663815a",

modules/local/gffread.nf

@@ -63,9 +63,9 @@ process GFFREAD {
 
     fi
 
-    ${projectDir}/bin/gff_to_genetranshash.2.pl
-    ${projectDir}/bin/prot_fasta_to_longest.pl ${prefix}.prot.fa ${prefix}_longestisoform.txt
-    ${projectDir}/bin/fasta_topIsoform.pl ${prefix}.splicedcds.fa ${prefix}_longestisoform.txt
+    gff_to_genetranshash.2.pl
+    prot_fasta_to_longest.pl ${prefix}.prot.fa ${prefix}_longestisoform.txt
+    fasta_topIsoform.pl ${prefix}.splicedcds.fa ${prefix}_longestisoform.txt
 
 
     #This part checks if longest isoform worked, if not we will continue with all proteins into Orthofinder. Warning sent to screen.

modules/local/longest.nf

@@ -4,7 +4,6 @@ process LONGEST {
     label 'process_med_memory'
 
     container = 'biocontainers/agat:1.3.0--pl5321hdfd78af_0'
-    publishDir "$params.outdir/output_data/longest" , mode: "${params.publish_dir_mode}", pattern:"*.txt"
 
     input:
     tuple val (meta),  path(gff)

modules/local/tree_summary.nf

@@ -27,18 +27,18 @@ process TREE_SUMMARY {
     cut -f 1,3,4,5,6,7 Busco_combined >> Busco_combined_cut
     sed -i \'s/\\.fasta//g\' Busco_combined_cut
 
-    python3 ${projectDir}/bin/busco_2_table.py Busco_combined_cut Busco_to_plot.tsv
+    busco_2_table.py Busco_combined_cut Busco_to_plot.tsv
 
     # Combine QUAST ouput
-    python3 ${projectDir}/bin/quast_2_table.py *quast.tsv -o Quast_to_plot.tsv -col N50,N90 -plot_types bar,bar
+    quast_2_table.py *quast.tsv -o Quast_to_plot.tsv -col N50,N90 -plot_types bar,bar
 
     #Remove unwanted extensions from Busco tables
     sed \'s/.prot.fa.largestIsoform.fa//g\' Busco_to_plot.tsv > Busco_to_plot_final.tsv
     sed \'s/.prot.fa.largestIsoform.fa//g\' Quast_to_plot.tsv > Quast_to_plot_final.tsv
 
     # Run summary plot
-    /usr/bin/Rscript ${projectDir}/bin/plot_tree_summary2.R tree.nw Busco_to_plot_final.tsv --tree_size 0.6
-    /usr/bin/Rscript ${projectDir}/bin/plot_tree_summary.R  tree.nw Quast_to_plot_final.tsv --tree_size 0.6
+    plot_tree_summary2.R tree.nw Busco_to_plot_final.tsv --tree_size 0.6
+    plot_tree_summary.R  tree.nw Quast_to_plot_final.tsv --tree_size 0.6
 
     cat <<-END_VERSIONS > versions.yml
     "${task.process}":

modules/nf-core/merqury/merqury/environment.yml

@@ -0,0 +1,5 @@
+channels:
+  - conda-forge
+  - bioconda
+dependencies:
+  - bioconda::merqury=1.3

modules/nf-core/merqury/merqury/main.nf

@@ -0,0 +1,87 @@
+process MERQURY_MERQURY {
+    tag "$meta.id"
+    label 'process_low'
+
+    // WARN: Version information not provided by tool on CLI. Please update this string when bumping container versions.
+    conda "${moduleDir}/environment.yml"
+    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
+        'https://depot.galaxyproject.org/singularity/merqury:1.3--hdfd78af_1':
+        'biocontainers/merqury:1.3--hdfd78af_1' }"
+
+    input:
+    tuple val(meta), path(meryl_db), path(assembly)
+
+    output:
+    tuple val(meta), path("*_only.bed")          , emit: assembly_only_kmers_bed
+    tuple val(meta), path("*_only.wig")          , emit: assembly_only_kmers_wig
+    tuple val(meta), path("*.completeness.stats"), emit: stats
+    tuple val(meta), path("*.dist_only.hist")    , emit: dist_hist
+    tuple val(meta), path("*.spectra-cn.fl.png") , emit: spectra_cn_fl_png
+    tuple val(meta), path("*.spectra-cn.hist")   , emit: spectra_cn_hist
+    tuple val(meta), path("*.spectra-cn.ln.png") , emit: spectra_cn_ln_png
+    tuple val(meta), path("*.spectra-cn.st.png") , emit: spectra_cn_st_png
+    tuple val(meta), path("*.spectra-asm.fl.png"), emit: spectra_asm_fl_png
+    tuple val(meta), path("*.spectra-asm.hist")  , emit: spectra_asm_hist
+    tuple val(meta), path("*.spectra-asm.ln.png"), emit: spectra_asm_ln_png
+    tuple val(meta), path("*.spectra-asm.st.png"), emit: spectra_asm_st_png
+    tuple val(meta), path("${prefix}.qv")        , emit: assembly_qv
+    tuple val(meta), path("${prefix}.*.qv")      , emit: scaffold_qv
+    tuple val(meta), path("*.hist.ploidy")       , emit: read_ploidy
+    tuple val(meta), path("*.hapmers.blob.png")  , emit: hapmers_blob_png           , optional: true
+    path "versions.yml"                          , emit: versions
+
+    when:
+    task.ext.when == null || task.ext.when
+
+    script:
+    // def args = task.ext.args ?: ''
+    prefix = task.ext.prefix ?: "${meta.id}"
+    def VERSION = 1.3 // WARN: Version information not provided by tool on CLI. Please update this string when bumping container versions.
+    """
+    # Nextflow changes the container --entrypoint to /bin/bash (container default entrypoint: /usr/local/env-execute)
+    # Check for container variable initialisation script and source it.
+    if [ -f "/usr/local/env-activate.sh" ]; then
+        set +u  # Otherwise, errors out because of various unbound variables
+        . "/usr/local/env-activate.sh"
+        set -u
+    fi
+    # limit meryl to use the assigned number of cores.
+    export OMP_NUM_THREADS=$task.cpus
+
+    merqury.sh \\
+        $meryl_db \\
+        $assembly \\
+        $prefix
+
+    cat <<-END_VERSIONS > versions.yml
+    "${task.process}":
+        merqury: $VERSION
+    END_VERSIONS
+    """
+
+    stub:
+    prefix = task.ext.prefix ?: "${meta.id}"
+    def VERSION = 1.3 // WARN: Version information not provided by tool on CLI. Please update this string when bumping container versions.
+    """
+    touch ${prefix}_only.bed
+    touch ${prefix}_only.wig
+    touch ${prefix}.completeness.stats
+    touch ${prefix}.dist_only.hist
+    touch ${prefix}.spectra-cn.fl.png
+    touch ${prefix}.spectra-cn.hist
+    touch ${prefix}.spectra-cn.ln.png
+    touch ${prefix}.spectra-cn.st.png
+    touch ${prefix}.spectra-asm.fl.png
+    touch ${prefix}.spectra-asm.hist
+    touch ${prefix}.spectra-asm.ln.png
+    touch ${prefix}.spectra-asm.st.png
+    touch ${prefix}.qv
+    touch ${prefix}.${prefix}.qv
+    touch ${prefix}.hist.ploidy
+
+    cat <<-END_VERSIONS > versions.yml
+    "${task.process}":
+        merqury: $VERSION
+    END_VERSIONS
+    """
+}