-
Notifications
You must be signed in to change notification settings - Fork 0
/
cwl.html
322 lines (277 loc) · 11 KB
/
cwl.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
<title>Rabix: CWL by example</title>
<meta name="description" content="CWL tips and tricks from Rabix Tools">
<meta name="robots" content="noodp">
<link rel="canonical" href="http://www.rabix.io">
<link rel="apple-touch-icon-precomposed" sizes="57x57" href="images/apple-touch-icon-57x57.png" />
<link rel="apple-touch-icon-precomposed" sizes="114x114" href="images/apple-touch-icon-114x114.png" />
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="images/apple-touch-icon-72x72.png" />
<link rel="apple-touch-icon-precomposed" sizes="144x144" href="images/apple-touch-icon-144x144.png" />
<link rel="apple-touch-icon-precomposed" sizes="60x60" href="images/apple-touch-icon-60x60.png" />
<link rel="apple-touch-icon-precomposed" sizes="120x120" href="images/apple-touch-icon-120x120.png" />
<link rel="apple-touch-icon-precomposed" sizes="76x76" href="images/apple-touch-icon-76x76.png" />
<link rel="apple-touch-icon-precomposed" sizes="152x152" href="images/apple-touch-icon-152x152.png" />
<link rel="icon" type="image/png" href="images/favicon-196x196.png" sizes="196x196" />
<link rel="icon" type="image/png" href="images/favicon-96x96.png" sizes="96x96" />
<link rel="icon" type="image/png" href="images/favicon-32x32.png" sizes="32x32" />
<link rel="icon" type="image/png" href="images/favicon-16x16.png" sizes="16x16" />
<link rel="icon" type="image/png" href="images/favicon-128.png" sizes="128x128" />
<meta name="application-name" content=" " />
<meta name="msapplication-TileColor" content="#FFFFFF" />
<meta name="msapplication-TileImage" content="mstile-144x144.png" />
<meta name="msapplication-square70x70logo" content="mstile-70x70.png" />
<meta name="msapplication-square150x150logo" content="mstile-150x150.png" />
<meta name="msapplication-wide310x150logo" content="mstile-310x150.png" />
<meta name="msapplication-square310x310logo" content="mstile-310x310.png" />
<meta property="og:locale" content="en_US">
<meta property="og:type" content="website">
<meta property="og:title" content="CWL Patterns">
<meta property="og:description" content="CWL tips and tricks from Rabix Tools">
<meta property="og:url" content="http://www.rabix.io">
<meta property="og:site_name" content="Rabix: Power tools for the Common Workflow Language">
<meta property="og:image" content="http://rabix.io/images/facebook-Linked_Image___rabix_social.png">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:description" content="Rabix: CWL Patterns">
<meta name="twitter:title" content="CWL tips and tricks from Rabix Tools">
<meta name="twitter:image" content="http://rabix.io/images/twitter-In-Stream_Wide___rabix_social.png">
<meta name="twitter:creator" content="@Sbgenomics">
<link href="css/turretcss.min.css" rel="stylesheet">
<link href="css/rabix.css" rel="stylesheet">
<!-- Custom Fonts -->
<link rel="stylesheet" href="css/etline-font.css">
<link rel="stylesheet" href="highlight/styles/vs.css">
<script src="highlight/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.4.2/respond.min.js"></script>
<![endif]-->
<script>
(function (i, s, o, g, r, a, m) {
i['GoogleAnalyticsObject'] = r;
i[r] = i[r] || function () {
(i[r].q = i[r].q || []).push(arguments)
}, i[r].l = 1 * new Date();
a = s.createElement(o),
m = s.getElementsByTagName(o)[0];
a.async = 1;
a.src = g;
m.parentNode.insertBefore(a, m)
})(window, document, 'script', 'https://www.google-analytics.com/analytics.js', 'ga');
ga('create', 'UA-25508146-12', 'auto');
ga('send', 'pageview');
</script>
</head>
<body>
<header>
<nav>
<a href="index.html">Rabix</a>
</nav>
<h1>CWL Patterns</h1>
Tips for using the Common Workflow Language
</header>
<div class="content">
<section>
<p>This page contains some common patterns that are found when
describing tools and workflows in CWL. It is meant for users
who are currently writing CWL and need to quickly look up a
practical pattern. If you want to add something to the list
just fork the <a href="https://github.com/rabix/rabix.github.io">website repository</a>,
make the required changes and create a pull request.
The patterns are not in any particular order, and the webpage is
meant to be searched using the browser's find function.
</p>
<p>
For a systematic and gentle introduction to CWL please see
the <a href="https://www.commonwl.org/user_guide/">
CWL user guide</a> which is a good resource for a self-paced
introduction to CWL.
</p>
<p>
Biostars is a good source of help on CWL via the
<a href="https://www.biostars.org/t/cwl/">CWL tag</a>.
</p>
</section>
<hr>
<section>
<p class="question">
Q: My program performs joint processing on a batch of multiple files.
I used to pass each file name on the command line, but when I scale
up I run into unix "Argument list too long" errors. What can I do?
</p>
A: The most resilient solution is to alter the program to accept a
manifest file. Most programs allow file names to be passed, one to a line,
in an input text file. You can then use javascript and an
<code>InitialWorkDirRequirement</code> to create this manifest file
on the fly from the a File list input as follows:
<pre><code class="yaml">
class: CommandLineTool
cwlVersion: v1.0
inputs:
input: 'File[]?'
baseCommand: cat
arguments: [manifest.txt]
requirements:
- class: DockerRequirement
dockerPull: alpine
- class: InitialWorkDirRequirement
listing:
- entryname: manifest.txt
entry: |-
${
var x = ""
for (var i = 0; i < inputs.input.length; i++)
{
x += inputs.input[i].path + "\n";
}
return x;
}
writable: false
- class: InlineJavascriptRequirement
outputs:
output: stdout
stdout: out.txt
</code></pre>
</section>
<section>
<p class="question">
Q: I have a Python script that I want to embed in my tool wrapper
(not make part of the docker image). How would I do this?
</p>
A: You can embed the script as an <code>InitialWorkDirRequirement</code>
<pre><code class="yaml">class: CommandLineTool
cwlVersion: v1.0
inputs:
message: string
baseCommand: ["python", "myscript.py"]
requirements:
InitialWorkDirRequirement:
listing:
- entryname: myscript.py
entry: |-
print("$(inputs.message)")
outputs:
example_out:
type: stdout
stdout: output.txt</code></pre>
</section>
<section>
<p class="question">
Followup Q: My embedded (Bash/Python/R) script has "$" signs in it
and this is conflicting with CWL parameter references. How do get them to
play nicely together?
</p>
A: You can escape the "$" in your script with "\$"
<p class="question">
Followup Q: I'd rather just paste the script into the CWL and not
have to go through it, escaping the "$" signs. Do I have another option?
</p>
A: You could embed your script in the "contents" field of the default
value of a file as follows:
<pre><code class="yaml">#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
label: Run an embedded Python script
hints:
DockerRequirement:
dockerPull: python:3
baseCommand: python
inputs:
script:
type: File
inputBinding:
position: 1
default:
class: File
basename: "script.py"
contents: |-
cash = 256.75
print("This costs ${}".format(cash))
outputs:
results:
type: stdout</code></pre>
To note in this solution is that a user can supply a different file
to the <code>script</code> input and override the default script.
This can be considered a bug or a feature, depending on your use case.
</section>
<section>
<p class="question">
Q: I have a large file which I want to index. My indexing tool insists
on creating the index in the same location as the file. I don't want
to copy the file into my working directory, and, of course, I don't
have write access to the original file.
</p>
<p>
A: It is best to stage the input file using
<code>InitialWorkDirRequirement</code>
<pre><code class="yaml">cwlVersion: v1.0
class: CommandLineTool
baseCommand: cat
inputs:
input_file:
type: File
inputBinding:
position: 1
valueFrom: $(self.basename)
requirements:
InitialWorkDirRequirement:
listing:
- $(inputs.in1)
outputs:
out1: stdout
hints:
DockerRequirement:
dockerPull: alpine</code></pre>
</p>
</section>
<section>
<p class="question">
Q: I have one input which can be
(a) an array files, (b) nothing at all, (c) an array of null.
How do I write my input to accommodate this?
</p>
A: The closest is probably:
<pre><code class="yaml">my_input:
type:
- type: array
items: ["null", File]
- "null"</code></pre>
Which will allow nothing at all and a list containing both Files
and/or nulls
</section>
<section>
<p class="question">
Q: I have a tool - like say <code>samtools</code> or <code>vcf-sort</code>
- that writes to <code>stdout</code>.
What is a succinct CWL idiom for collecting data written to stdout into a file?
In bash one would do <code>vcf-sort -c in.vcf > out.vcf</code>
</p>
<p>
A: Use:
<pre><code class="yaml">outputs:
out: stdout
stdout: out.vcf</code></pre>
</p>
<p>
If you don't explicitly need a file <code>out.vcf</code> in that tool
itself: you are just interested in the data, you can be even briefer:
<pre><code class="yaml">outputs:
out: stdout</code></pre>
</p>
</section>
</div>
<footer>
<p>Rabix is a Trademark of <a href="https://www.sevenbridges.com" target="_blank">Seven Bridges Genomics</a>
<p>Copyright © 2019 Seven Bridges Genomics</p>
</footer>
</body>
</html>