Add hybpiper #5464

TomHarrop · 2023-08-30T06:45:02Z

Adding the tool hybpiper

FOR CONTRIBUTOR:

- I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
- License permits unrestricted use (educational + commercial)
- This PR adds a new tool or tool collection
- This PR updates an existing tool or tool collection
- This PR does something else (explain below)

bgruening · 2023-08-30T07:59:09Z

@TomHarrop to trigger tests you need to add a .shed.yml file.

TomHarrop · 2023-08-30T08:08:49Z

Thanks. Will add. I still have a bit of other work to do on the wrapper.

tools/hybpiper/hybpiper.xml

Catch designation in stats etc.

…ed_conditionals

TomHarrop · 2023-09-06T05:11:40Z

tools/hybpiper/hybpiper.xml

+        <conditional name="job_conditional">
+            <param name="hybpiper_job" value="assemble"/>
+            <param name="paired_input">
+                <collection type="paired" name="NZ874">


Hi @bgruening @bernt-matthias thanks for your feedback, this is almost ready but I need some help with this test.

Setting name on the collection causes a linting error (The attribute 'name' is not allowed).

If i I remove name, element_identifier is set to Unnamed Collection , hybpiper chokes on the space, and the test fails.

Is there a way to set the element_identifier in <test> -> <collection> ?

Setting the collection name seems not possible at the moment. Could be worth opening an issue for this. But the question is: do the tests depend on that? I guess they should not.

Thanks, I opened an issue: galaxyproject/galaxy#16654

It does work (element_identifier resolves to NZ874), it's just linting that fails.

For now, I've added a check for special characters and set this test to expect_failure="true" in 2eeb313

TomHarrop · 2023-09-06T23:36:30Z

Thanks again for the help, I think this is ready for review / merge now. 🎉

tools/hybpiper/hybpiper.xml

bgruening · 2023-09-19T05:34:07Z

tools/hybpiper/hybpiper.xml

+                    <option value="supercontig">Supercontig</option>
+                </param>
+                <param name="heatmap" type="boolean" checked="false" label="Produce a heatmap for each of the selected statistics" />
+                <param name="sequence_type_select" type="select" display="checkboxes" label="Choose sequences to extract" multiple="true">


is this one optional or not, is it possible to select none of them?

They are all optional. I marked them optional in 28bcb4490.

Although if the user really deselects everything there won't be any outputs in Galaxy.

We should avoid that, correct? Thats why I have ask. To make sure the users has at least one output.

Sorry, if I confused you :(

OK, but I'm not sure how to prevent it in the wrapper. I tried a few things e.g.

#unless $job_conditional.stats_type_select or $job_conditional.sequence_type_select: #raise ValueError("No outputs selected") #end unless

But the user will never see the error because there are no outputs.

Users should be able to select some stats or some sequences, but not nothing. Although I reckon deselecting all the outputs and then getting no output is quite logical :)

but then we simply make it optional=false?

I mean the output select?

OK, I made stats optional=false in 022a3b2.

stats will be generated no matter what else the user selects. I wanted to avoid it because the underlying software is so slow but it's probably an edge case.

Hmm can't use checkboxes in that case. I think this would make the tool worse than having no output if all the output is de-selected. The user has to actually deselect everything since I have selected="true" on the usual options.

Apart from making it a single param, is there a way to say the user has to select at least one item from $job_conditional.stats_type_select or $job_conditional.sequence_type_select?

tools/hybpiper/hybpiper.xml

Co-authored-by: Björn Grüning <[email protected]>

TomHarrop · 2023-09-19T07:44:15Z

Thanks for the review @bgruening

bgruening · 2023-09-19T09:58:12Z

Damn the linting was not happy with checkbox and optional=false. Are you happy with the last commit. Then we can merge I think

TomHarrop · 2023-09-19T09:59:29Z

I would personally just revert 022a3b2 but I can live with it either way.

bgruening · 2023-09-19T10:06:25Z

Ok, please revert.

tools-iuc/tools/homer/macros.xml

Line 49 in c083399

    
           <validator type="no_options" message="No pre-installed organism DBs are available. Contact your galaxy administrator."/>

This you could try. A validator with type="no_option", maybe that works?

This reverts commit 022a3b2.

TomHarrop · 2023-09-20T00:49:27Z

Sorry, I had an urgent appointment with a bibimbap last night 🍚

I think the confusion is that there are two optional params here. The user can select nothing from either of the selectors, but has to select at least one output across the two params. So the validator didn't work either.

I'm doing this because the setup for extracting stats (selector one) or extracting sequences (selector two) is expensive and it could be run on hundreds of samples at once. So it's good that the user can do both in one tool run. BUT, the procedure for extracting those things is also slow, so it's good that the user can exclude one if they don't want that output.

I've got the effect by adding an output that Galaxy expects if the user deselects everything. Then there is a check in the <command> and an error will be visible in the history. Added a test for that too. Hopefully that does it!

Thanks again @bgruening

bgruening · 2023-09-21T22:34:24Z

Ok, please revert.

tools-iuc/tools/homer/macros.xml

Line 49 in c083399

<validator type="no_options" message="No pre-installed organism DBs are available. Contact your galaxy administrator."/>

This you could try. A validator with type="no_option", maybe that works?

Have you tried this validator?

TomHarrop · 2023-09-21T22:48:56Z

Yes

TomHarrop · 2023-09-22T04:09:22Z

I can't use the validator because there are two params. This is what it's doing:

1. This works. Stats are optional.

2. This works. Sequences are optional.

3. This should fail (and does, with "ERROR: No outputs selected")

bgruening · 2023-09-23T16:33:38Z

Thanks @TomHarrop!

initial tests

e1cac68

TomHarrop marked this pull request as draft August 30, 2023 06:45

TomHarrop changed the title ~~initial tests~~ Add hybpiper Aug 30, 2023

add assemble

db3d5ca

TomHarrop added 4 commits August 31, 2023 10:44

process prefix parameter

1002cd2

add test for prefix

698a8b3

add shed.yml

1d42849

avoid spaces in converted input names

c4c73c4

bgruening reviewed Aug 31, 2023

View reviewed changes

tools/hybpiper/hybpiper.xml Outdated Show resolved Hide resolved

TomHarrop added 2 commits August 31, 2023 14:11

don't use tar.gz

a85beda

started adding post-processing, not working yet

957c48c

bernt-matthias reviewed Aug 31, 2023

View reviewed changes

tools/hybpiper/hybpiper.xml Outdated Show resolved Hide resolved

tools/hybpiper/hybpiper.xml Outdated Show resolved Hide resolved

tools/hybpiper/hybpiper.xml Outdated Show resolved Hide resolved

TomHarrop changed the title ~~Add hybpiper~~ DRAFT / WIP: hybpiper Sep 1, 2023

TomHarrop and others added 15 commits September 1, 2023 17:55

parse label from R1

b62f259

Use from_provided_metadata to set designation for hybpiper archives

0792e28

Catch designation in stats etc.

tidy spacing

7b6aaa2

add recovery heatmap

e319983

linting

4f5d3aa

add test output

175134c

initial refactor of outputs

d52eda8

revert number of outputs. CI giving different error

91afa65

initial refactor of outputs

c7a2c3f

Merge branch 'hybpiper_nested_conditionals_rebase' into hybpiper_nest…

e4d8cf2

…ed_conditionals

format list

30f8528

catch stats and sequence output

5741691

deal with differences in o9utput names

1b21649

outputs

3eb9015

filter order matters

b76c7b0

TomHarrop commented Sep 6, 2023

View reviewed changes

TomHarrop added 2 commits September 6, 2023 17:06

add column names

e24c714

Add method for checking sample identifiers

2eeb313

TomHarrop marked this pull request as ready for review September 6, 2023 23:35

TomHarrop changed the title ~~DRAFT / WIP: hybpiper~~ Add hybpiper Sep 6, 2023

bernt-matthias mentioned this pull request Sep 7, 2023

Allow setting the name attribute on test collections galaxyproject/galaxy#16654

Closed

add help, check quoting

256ceaf

bgruening reviewed Sep 19, 2023

View reviewed changes

TomHarrop and others added 5 commits September 19, 2023 17:03

Update tools/hybpiper/hybpiper.xml

5a16394

Co-authored-by: Björn Grüning <[email protected]>

Update tools/hybpiper/hybpiper.xml

92ee0a0

Co-authored-by: Björn Grüning <[email protected]>

Update tools/hybpiper/hybpiper.xml

38c448f

Co-authored-by: Björn Grüning <[email protected]>

Update tools/hybpiper/hybpiper.xml

53eb7f3

Co-authored-by: Björn Grüning <[email protected]>

mark params optional

28bcb44

TomHarrop and others added 2 commits September 19, 2023 19:47

stats are compulsory

022a3b2

Update hybpiper.xml

10065de

TomHarrop added 2 commits September 20, 2023 10:20

Revert "stats are compulsory"

6dea550

This reverts commit 022a3b2.

make a blank output if the user deselects everything

b9cdd41

bgruening approved these changes Sep 23, 2023

View reviewed changes

bgruening merged commit b439a8b into galaxyproject:main Sep 23, 2023
10 checks passed

TomHarrop deleted the hybpiper branch February 12, 2024 00:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hybpiper #5464

Add hybpiper #5464

TomHarrop commented Aug 30, 2023 •

edited

Loading

bgruening commented Aug 30, 2023

TomHarrop commented Aug 30, 2023

TomHarrop Sep 6, 2023

bernt-matthias Sep 6, 2023 •

edited

Loading

TomHarrop Sep 6, 2023

TomHarrop commented Sep 6, 2023

bgruening Sep 19, 2023

TomHarrop Sep 19, 2023

bgruening Sep 19, 2023

bgruening Sep 19, 2023

TomHarrop Sep 19, 2023

bgruening Sep 19, 2023

bgruening Sep 19, 2023

TomHarrop Sep 19, 2023

TomHarrop Sep 19, 2023

TomHarrop commented Sep 19, 2023

bgruening commented Sep 19, 2023

TomHarrop commented Sep 19, 2023

bgruening commented Sep 19, 2023

TomHarrop commented Sep 20, 2023

bgruening commented Sep 21, 2023

TomHarrop commented Sep 21, 2023

TomHarrop commented Sep 22, 2023

bgruening commented Sep 23, 2023

Add hybpiper #5464

Add hybpiper #5464

Conversation

TomHarrop commented Aug 30, 2023 • edited Loading

bgruening commented Aug 30, 2023

TomHarrop commented Aug 30, 2023

Choose a reason for hiding this comment

bernt-matthias Sep 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomHarrop commented Sep 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomHarrop commented Sep 19, 2023

bgruening commented Sep 19, 2023

TomHarrop commented Sep 19, 2023

bgruening commented Sep 19, 2023

TomHarrop commented Sep 20, 2023

bgruening commented Sep 21, 2023

TomHarrop commented Sep 21, 2023

TomHarrop commented Sep 22, 2023

1. This works. Stats are optional.

2. This works. Sequences are optional.

3. This should fail (and does, with "ERROR: No outputs selected")

bgruening commented Sep 23, 2023

TomHarrop commented Aug 30, 2023 •

edited

Loading

bernt-matthias Sep 6, 2023 •

edited

Loading