Bin QC Improvements #707

dialvarezs · 2024-10-27T21:31:08Z

This PR adds:

CheckM2 as an alternative for bin qc
Updates CheckM and GUNC modules
A new BIN_QC subworkflow, integrating CheckM, CheckM2, BUSCO and GUNC

Closes #607.

PR checklist

- Update modules - Update integration in mag and with other tools (bin_summary, gtdb-tk) - Update test - Update schema

jfy133 · 2024-10-27T21:40:34Z

@nf-core-bot fix linting

jfy133 · 2024-10-27T21:42:43Z

Before you continue (sorry this is a bit late):

I generally don't like to deprecate old version of tools for a while, but rather keep them as alternative tools.

In some cases people want to stick with the original version for compatibility with previous runs

Could you 'revert' (or reinstall) the old checkm module and wrap it in an if/else statement (but within the subworkflow :) )

@muabnezor did a similar thing when adding porechop_ABI hree: #674

dialvarezs · 2024-10-27T21:49:06Z

@jfy133 that makes sense, I will revert the CheckM removal. Bad for me for not asking before 😅.

dialvarezs · 2024-10-28T12:15:01Z

Sorry, I didn't catch your last comment about including both tools in a single workflow. With that in mind, would make sense to include BUSCO as well, and just make a "bin_qc" subworkflow?

Also, simplify bin_summary regarding bin qc

jfy133 · 2024-10-29T09:47:00Z

Sorry, I didn't catch your last comment about including both tools in a single workflow. With that in mind, would make sense to include BUSCO as well, and just make a "bin_qc" subworkflow?

Yes that would be perfect! We need to subworkflow the sh*t out of this monster 😅 thank you!!!

dialvarezs · 2024-11-28T19:12:57Z

It should be ready now. There is a last minor issue that should be solved by this PR: nf-core/modules#7119

jfy133 · 2024-12-05T08:58:30Z

Running the failed test one more time, the conoct test has failed a few times before too right?

dialvarezs · 2024-12-05T12:43:28Z

From the ci logs it seems to be lack of space in the runner.

dialvarezs · 2024-12-05T13:32:35Z

Oh, I realized something now. I totally forgot to skip Bin QC if params.skip_binqc is set. 😅

jfy133 · 2024-12-06T04:55:40Z

Glad you worked it out!

Unfortunately since yesterday now have both kids home sick so I will only have time to test next week again (I'm really really sorry for this, apparently it's being a really bad year in my city in the kindergartens and schools for viruses and stuff 😣)

jfy133

A few more last comments, but otherwise all I'm doing for the rest of the day is running the tests!

One major thing missing now though: We've removed the standalone checkm test in the ci.yml, but this has not been replaced by an alternative test, so now we have no checkM test... we should make sure all three are executed at least one across all our tests.

I suggest:

test.config: run BUSCO (default, no change)
test_adapterremoval.config: run checkm
test_bbnorm.config: run checkm2

Does that make sense?

CITATIONS.md

docs/output.md

modules/local/combine_tsv.nf

nextflow_schema.json

subworkflows/local/bin_qc.nf

jfy133 · 2024-12-11T09:57:14Z

subworkflows/local/bin_qc.nf

+        if (params.save_busco_db) {
+            // publish files downloaded by Busco
+            ch_downloads = BUSCO.out.busco_downloads
+                .groupTuple()
+                .map { _lin, downloads -> downloads[0] }
+                .toSortedList()
+                .flatten()
+            BUSCO_SAVE_DOWNLOAD(ch_downloads)
+
+            ch_versions = ch_versions.mix(BUSCO_SAVE_DOWNLOAD.out.versions.first())
+        }


Better description of second half of my old comment on this:

This should go below the BUSCO module, given it takes the output of the BUSCO module

Do you see any reason why this cannot just be replaced with an extra publishDir entry in modules.conf for BUSCO where if --save_busco_db, then save the database files.

subworkflows/local/bin_qc.nf

jfy133 · 2024-12-11T13:19:32Z

Checked on your PR vs dev:

Normal test (BUSCO) (busco_summary.tsv and binsummary.tsv comparable ✅)
Test with CheckM (checkm_summary.tsv and binsummary.tsv comparable ✅)
Test with CheckM2 (checkm_summary.tsv exists and binsummary.tsv has table merged with quast results ✅)
Test with checkm2 + GUNC (check GUNC executes basically and produces a gunc_Summary.tsv)

Sorry this has taken so long, but the tests are taking around 25-30m each time 🙄 which doesn't help

Ah I need to do one more manual check which is with GTDBTk which I'll have to do one a cluster tomorrow, as you've tweaked that slightly

Test with GTDBTk

But I'm pretty sure this is ready once my final comments above are addressed :) , mostly the missing addition of the CheckMs to the tests.

jfy133 · 2024-12-11T19:35:31Z

Crap, BUSCO is borked everywhere isn't it 😢

dialvarezs · 2024-12-11T19:40:50Z

Yeah, it's already reported here: https://gitlab.com/ezlab/busco/-/issues/776 and it seems that the only solution is to update. Today I have been looking at the code to update and migrate to the nf-core BUSCO module.

Do you mind if I integrate those changes in this PR, or do you prefer a new one?

For the meantime, can you give a review to the module update? nf-core/modules#7199
I' getting an error with the meta that I'm not sure how to solve...

jfy133 · 2024-12-11T19:43:01Z

Self note: testing GTDBtk with following commands:

CHECKM-NEW nextflow run dialvarezs/mag -r dev-checkm2 -profile test,mpcdf_viper --outdir ./results_dialvarezs_checkm_gdtbk --binqc_tool checkm2 --skip_gtdbtk false --gtdb_db /ptmp/jfellowsy/databases/gtdb/release220/ --binqc_tool checkm --checkm_db ../databases/checkm/ --krona_db /ptmp/jfellowsy/databases/krona/taxonomy/ --centrifuge_db /ptmp/jfellowsy/mag/testdbs/minigut_cf.tar.gz --kraken2_db /ptmp/jfellowsy/mag/testdbs/minigut_kraken.tgz -resume (a bit confused because I specify checkm2 and accidently gave checkm1 db, but it seems checkm1 executed -> because I have --binqc_tool twice 😅) ? ✅
CHECKM-OLD nextflow run nf-core/mag -r dev -profile test,mpcdf_viper --outdir ./results_checkm_gdtbk --skip_gtdbtk false --gtdb_db /ptmp/jfellowsy/databases/gtdb/release220/ --binqc_tool checkm --checkm_db ../databases/checkm/ --krona_db /ptmp/jfellowsy/databases/krona/taxonomy/ --centrifuge_db /ptmp/jfellowsy/mag/testdbs/minigut_cf.tar.gz --kraken2_db /ptmp/jfellowsy/mag/testdbs/minigut_kraken.tgz ✅ multiprocessing socket in use error -> two checkM jobs on same node breaks checkm, will either maxForks or just keep resuming)
CHECKM2-NEW: nextflow run dialvarezs/mag -r dev-checkm2 -profile test,mpcdf_viper --outdir ./results_dialvarezs_checkm2_gdtbk --binqc_tool checkm2 --skip_gtdbtk false --gtdb_db /ptmp/jfellowsy/databases/gtdb/release220/ --checkm_db /ptmp/jfellowsy/databases/checkm2/ --krona_db /ptmp/jfellowsy/databases/krona/taxonomy/ --centrifuge_db /ptmp/jfellowsy/mag/testdbs/minigut_cf.tar.gz --kraken2_db /ptmp/jfellowsy/mag/testdbs/minigut_kraken.tgz ✅
BUSCO-NEW: nextflow run dialvarezs/mag -r dev-checkm2 -profile test,mpcdf_viper --outdir ./results_dialvarezs_busco_gdtbk --binqc_tool busco --skip_gtdbtk false --gtdb_db /ptmp/jfellowsy/databases/gtdb/release220/ --busco_db /ptmp/jfellowsy/databases/busco/bacteria_obd10 --krona_db /ptmp/jfellowsy/databases/krona/taxonomy/ --centrifuge_db /ptmp/jfellowsy/mag/testdbs/minigut_cf.tar.gz --kraken2_db /ptmp/jfellowsy/mag/testdbs/minigut_kraken.tgz ✅

jfy133 · 2024-12-11T19:43:51Z

Yeah, it's already reported here: https://gitlab.com/ezlab/busco/-/issues/776 and it seems that the only solution is to update. Today I have been looking at the code to update and migrate to the nf-core BUSCO module.

Do you mind if I integrate those changes in this PR, or do you prefer a new one?

Let's do a new one, it's breaking small MEGAHIT fix PR, so if it's separate we can pull into that one and this one at the same time.

For the meantime, can you give a review to the module update? nf-core/modules#7199 I' getting an error with the meta that I'm not sure how to solve...

Will have a look now! EDIT: fixed!

jfy133 · 2024-12-13T13:07:42Z

Old BUSCO files should be back now!

And I'm waiting for my last GTDBTK related check but all the other runs looking good!

dialvarezs · 2024-12-13T13:14:44Z

Old BUSCO files should be back now!

That's good news. The BUSCO update PR ended up getting a bit large with the removal of all those local modules.

jfy133 · 2024-12-13T14:41:42Z

@dialvarezs My manual tests with GTDBTk work!

Just need the tests configs as in my comment above, but otherwise this is ready :D

nf-core-bot · 2024-12-13T14:43:10Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.0.2.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

dialvarezs · 2024-12-13T15:02:55Z

@jfy133 great!
And I agree with you about the tests, I will add that right now.

jfy133

If tests pass with the new configs...

jfy133 · 2024-12-13T15:14:03Z

Thank you very much @dialvarezs ! This is huge work!

dialvarezs · 2024-12-13T16:25:18Z

There was a last minor bug when CheckM was not run for certain bins (specifically eukaryotic ones). I updated the condition to check if the CheckM bins are a subset of the depth bins, rather than requiring them to be equal.

feat: Migrate from CheckM to CheckM2

53864d2

- Update modules - Update integration in mag and with other tools (bin_summary, gtdb-tk) - Update test - Update schema

dialvarezs marked this pull request as draft October 27, 2024 21:31

dialvarezs added 3 commits October 27, 2024 19:05

fix: Bring back CheckM

ca2f97b

fix: One more thing

05e3393

fix: Linting

3e49dbf

dialvarezs changed the title ~~feat: Replace from CheckM to CheckM2~~ feat: Add CheckM2 Oct 27, 2024

fix: Option and checks in bin summary script

b1b6518

dialvarezs force-pushed the dev-checkm2 branch from c63e084 to b1b6518 Compare October 27, 2024 22:40

dialvarezs added 2 commits October 27, 2024 19:43

Fix: missing import

73b5794

docs: Output, changelog and citation for CheckM2

325310b

dialvarezs marked this pull request as ready for review October 28, 2024 10:30

dialvarezs added 3 commits October 28, 2024 07:32

docs: readme

b88302a

Merge remote-tracking branch 'upstream/dev' into dev-checkm2

d62a671

docs: Update readme

3da7441

dialvarezs added 2 commits October 28, 2024 09:57

refactor: Merge checkm and checkm2 subworkflows in a single one

3d98415

Also, simplify bin_summary regarding bin qc

fix: Restore mistakenly deleted code

8b4fcc7

dialvarezs added 4 commits October 31, 2024 17:26

Bin QC workflow

76dac5c

Merge remote-tracking branch 'upstream/dev' into dev-checkm2

0ebab07

Cleanup

38ce756

Final touches

3702329

dialvarezs force-pushed the dev-checkm2 branch 2 times, most recently from a4f42ef to da52285 Compare November 1, 2024 00:27

Integrate GUNC in BIN_QC subworkflow

0eb167a

dialvarezs force-pushed the dev-checkm2 branch from 4007932 to 0eb167a Compare November 1, 2024 04:13

Code style improvements

f0a6999

Fix declaration

80641b0

Update checkm/qa module

4b084b1

Merge remote-tracking branch 'upstream/dev' into dev-checkm2

f75e9c4

Handle skip_binqc properly

d176fce

jfy133 reviewed Dec 11, 2024

View reviewed changes

dialvarezs force-pushed the dev-checkm2 branch from c111681 to 3a38a37 Compare December 11, 2024 18:55

Address review comments

3a38a37

Merge branch 'dev' into dev-checkm2

009abd1

Update tests to integrate CheckM and CheckM2

a2afa5e

jfy133 self-requested a review December 13, 2024 15:11

jfy133 approved these changes Dec 13, 2024

View reviewed changes

Fix for when checkm is not run for some bins

61c23fe

dialvarezs merged commit c096f9a into nf-core:dev Dec 13, 2024
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bin QC Improvements #707

Bin QC Improvements #707

dialvarezs commented Oct 27, 2024 •

edited

Loading

jfy133 commented Oct 27, 2024

jfy133 commented Oct 27, 2024

dialvarezs commented Oct 27, 2024

dialvarezs commented Oct 28, 2024 •

edited

Loading

jfy133 commented Oct 29, 2024

dialvarezs commented Nov 28, 2024

jfy133 commented Dec 5, 2024

dialvarezs commented Dec 5, 2024

dialvarezs commented Dec 5, 2024 •

edited

Loading

jfy133 commented Dec 6, 2024

jfy133 left a comment

jfy133 Dec 11, 2024

jfy133 commented Dec 11, 2024 •

edited

Loading

jfy133 commented Dec 11, 2024

dialvarezs commented Dec 11, 2024 •

edited

Loading

jfy133 commented Dec 11, 2024 •

edited

Loading

jfy133 commented Dec 11, 2024 •

edited

Loading

jfy133 commented Dec 13, 2024

dialvarezs commented Dec 13, 2024

jfy133 commented Dec 13, 2024

nf-core-bot commented Dec 13, 2024

dialvarezs commented Dec 13, 2024

jfy133 left a comment

jfy133 commented Dec 13, 2024

dialvarezs commented Dec 13, 2024 •

edited

Loading

Bin QC Improvements #707

Bin QC Improvements #707

Conversation

dialvarezs commented Oct 27, 2024 • edited Loading

PR checklist

jfy133 commented Oct 27, 2024

jfy133 commented Oct 27, 2024

dialvarezs commented Oct 27, 2024

dialvarezs commented Oct 28, 2024 • edited Loading

jfy133 commented Oct 29, 2024

dialvarezs commented Nov 28, 2024

jfy133 commented Dec 5, 2024

dialvarezs commented Dec 5, 2024

dialvarezs commented Dec 5, 2024 • edited Loading

jfy133 commented Dec 6, 2024

jfy133 left a comment

Choose a reason for hiding this comment

jfy133 Dec 11, 2024

Choose a reason for hiding this comment

jfy133 commented Dec 11, 2024 • edited Loading

jfy133 commented Dec 11, 2024

dialvarezs commented Dec 11, 2024 • edited Loading

jfy133 commented Dec 11, 2024 • edited Loading

jfy133 commented Dec 11, 2024 • edited Loading

jfy133 commented Dec 13, 2024

dialvarezs commented Dec 13, 2024

jfy133 commented Dec 13, 2024

nf-core-bot commented Dec 13, 2024

dialvarezs commented Dec 13, 2024

jfy133 left a comment

Choose a reason for hiding this comment

jfy133 commented Dec 13, 2024

dialvarezs commented Dec 13, 2024 • edited Loading

dialvarezs commented Oct 27, 2024 •

edited

Loading

dialvarezs commented Oct 28, 2024 •

edited

Loading

dialvarezs commented Dec 5, 2024 •

edited

Loading

jfy133 commented Dec 11, 2024 •

edited

Loading

dialvarezs commented Dec 11, 2024 •

edited

Loading

jfy133 commented Dec 11, 2024 •

edited

Loading

jfy133 commented Dec 11, 2024 •

edited

Loading

dialvarezs commented Dec 13, 2024 •

edited

Loading