You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we generate byte ranges referencing the cram file, and stream these through the various steps (FASTQ conversion etc) involved in alignment, to remove unnecessary I/O of intermediate files. Currently this excludes a filtering steps that are currently (e.g., BLASTN for PacBio) or planned for the pipeline.
To add in further steps that would also benefit from chunked processing, we need either:
Special version of the cram_filter_<aligner>_*.nf modules for each specific step
Write the chunks to file and pass these through all required modules
Separate map-reduce for filtering steps
Option 1 could mean a lot of different version of the align modules, but saves on intermediate files. Option 2 would allow us to use a lot more 'stock' modules, which would be easier to maintain, but creates more intermediates. Option 3 is probably not a great option.
The text was updated successfully, but these errors were encountered:
Description of feature
Currently we generate byte ranges referencing the cram file, and stream these through the various steps (FASTQ conversion etc) involved in alignment, to remove unnecessary I/O of intermediate files. Currently this excludes a filtering steps that are currently (e.g., BLASTN for PacBio) or planned for the pipeline.
To add in further steps that would also benefit from chunked processing, we need either:
cram_filter_<aligner>_*.nf
modules for each specific stepOption 1 could mean a lot of different version of the align modules, but saves on intermediate files. Option 2 would allow us to use a lot more 'stock' modules, which would be easier to maintain, but creates more intermediates. Option 3 is probably not a great option.
The text was updated successfully, but these errors were encountered: