wip! workflows as programs prototyping

[tsibley]

See commit messages. Opening for visibility and soliciting commentary. Not for merge yet.

Related-to: nextstrain/measles#55

Checklist

• Checks pass

[huddlej]

@tsibley It was really cool to see a working demo of this idea today! A couple of thoughts I had from your lab meeting presentation:

Providing --help documentation with the workflow

You mentioned that workflow authors will need to provide documentation for the workflow interface (config parameters, files, etc.) and (implicitly?) referenced the ncov workflow reference docs as an example of this. One pattern I liked from Snk's implementation of workflows-as-programs is the auto-generation of --help outputs for each workflow based on the config file. Snk generates an argument for each config YAML option in a workflow and the --help flag prints the list of arguments and their types as a way to expose the workflow's interface to users.

Ideally, Nextstrain users could have a similar option like nextstrain run measles --help or nextstrain help measles, etc. that exposes the workflow interface in the same way any other CLI does. I don't know where all of the help details would live in each workflow, but maybe we could discuss this general pattern?

I thought maybe you did some work with this repo to allow you to document nextstrain's interface in one place that can be exposed through --help and through the RTD website. That approach could increase the availability of that documentation without increasing the number of copies we have to maintain.

Maybe avoid overloading the setup command name?

The current prototype in your slides showed nextstrain setup measles as a way to download a copy of that Nextstrain workflow locally for use by the nextstrain run command. This syntax overloads the existing meaning of setup for runtimes (nextstrain setup conda). Since we're thinking about runtimes and workflows as separate entities (for now?), would it be clearer to provide a nextstrain workflow-setup measles interface or something like that?

If we had nextstrain workflow setup measles (this is getting verbose, though), we could also have nextstrain workflow list as a way for users to list available workflows.

Re: providing the nextstrain run interface implies a need to better version workflows

Following up on @genehack's comment at lab meeting, I don't agree that providing this run interface requires us to do better job of versioning our workflows. Your description of the interfaces suggested that users can setup workflows by commit or tag. This feels like it's just removing the manual git clone step from the user's experience. They still need to run nextstrain update measles to "upgrade" it to the latest commit, so they opt-in to new versions like they would by running git pull.

I do agree that we should do a better job of versioning our workflows independent of how users run them, though!

[tsibley]

@huddlej

Snk generates an argument for each config YAML option in a workflow and the --help flag prints the list of arguments and their types as a way to expose the workflow's interface to users.

Yeah, I saw that in Snk. I'm not sure about auto-generating CLI options based on the workflow's config. It seems to me to be a more complicated way of providing config.yaml and one that's prone to all the issues that come from duplicating an interface into a different context. And our configs are substantial enough that I'm inclined to push users towards using config files as a best practice rather than lengthy sets of CLI overrides. I'd be interested in discussing it more, though. I can see how we'd use config schemas, which we want anyway, to generate the CLI options.

Maybe avoid overloading the setup command name?

Yeah, I've waffled on this a bit in my thinking. My old notes are full of nextstrain workflow {setup,update,run,…} hypotheticals, but my thinking changed recently. In my notes last week, I wrote:

Is it weird to have nextstrain setup and nextstrain update work for pathogens as well as runtimes? If so, we can introduce separation by nouns, e.g.

nextstrain workflow {setup,update}
nextstrain runtime {setup,update}

but if we're going to preserve backwards compat (and I really think we should), then we'd still have nextstrain {setup,update} aliased to nextstrain runtime {setup,update} for a while (maybe indefinitely) and I wonder if that sort of defeats the point.

My thinking of overloading setup and update to work for pathogens as well as runtimes comes out of

1. preserving the existing subcommands mostly being verbs (taking nouns as arguments) as opposed to nouns (taking verbs and more nouns as arguments), and
2. realizing it doesn't seem to add ambiguity (at least with only runtimes and pathogens as the kind of nouns): for the computer, the list of recognized runtimes is short and fixed, everything else is a pathogen; for the human, measles is obviously a pathogen, docker is obviously something else.

An exception to the verb-nouns command pattern is the remote family of commands which uses the noun-verb-nouns pattern (remote, upload, groups/foo, …). That might give some support to a workflow or pathogen subcommand with its own verbal subcommands, but it seems to me to push things too deep. Going that way, I'd also feel compelled to move setup, update, and check-setup under nextstrain runtime {setup,update,check-setup}, though that felt pointless for the reasons I wrote last week. And I'd still feel the need to have nextstrain run instead of (or in addition to) nextstrain workflow run so as to better center the core activity and match existing workflow-centric commands (view, build, etc). It all felt too messy.

So I figured trying to implementing pathogen setup/update into the existing commands would reveal if it was actually feasible or not and give more information to help make the choice.

I do think we want a way to list workflows. Haven't quite gotten there yet. I think we can do something other than nextstrain workflow ls if we don't go the nextstrain workflow route.

I don't agree that providing this run interface requires us to do better job of versioning our workflows. […] I do agree that we should do a better job of versioning our workflows independent of how users run them, though!

+1!

[huddlej]

Thanks, @tsibley!

I'm not sure about auto-generating CLI options based on the workflow's config.

That's fair. My main hope was that we could include the documentation for the config in the interface somehow, thinking that it's nicer for users if they don't have to leave the command line for a website to figure out how the program works. The autogenerated CLI is just an example of one approach, but I'm open to whatever achieves the end goal.

I figured trying to implementing pathogen setup/update into the existing commands would reveal if it was actually feasible or not and give more information to help make the choice.

Yeah, I can see how the interface gets complicated fast when you start to bifurcate on workflow vs. runtime. As you test this with people, I'd be most interested in whether external users experience any confusion when trying to update a runtime vs. a workflow. It's obvious to us that these are separate nouns, but if users rely on the verb to signal the noun that's being operated on, it could get confusing.

[genehack]

Thinking out loud about implementation edge cases around fetching workflows:

• For the workflow fetching — whether that's setup or something else¹ — is the idea that a "bare" noun would always get https://github.com/nextstrain/ prepended?
• What does it look like if somebody wants to fork measles into their own org and then fetch that? (Options I see are "if there's a slash in the noun, only prepend https://github.com" and "if it's not a 'bare' noun it has to be a fully specified URL that will work with git clone")
• What does the on-disk storage look like if somebody forks measles into their own organization but wants to nextstrain setup measles AND nextstrain setup my-org/measles?

¹ registering my (probably minority) vote for "something else"; having one verb do two completely different things seems like a bad idea to me…

[tsibley]

@genehack— Good questions, I've thought thru these previously (though not articulated them outside of my notes yet) and so have some answers. That said, I welcome others to think thru them too!

For the workflow fetching — whether that's setup or something else¹ — is the idea that a "bare" noun would always get https://github.com/nextstrain/ prepended?

I anticipate the constructed source URL would not be exactly that (see below), but in concept, yes.

What does it look like if somebody wants to fork measles into their own org and then fetch that? (Options I see are "if there's a slash in the noun, only prepend https://github.com" and "if it's not a 'bare' noun it has to be a fully specified URL that will work with git clone")

If it's not a bare word ("no slashes"), I'm inclined to require it be a URL for maximum explicitness/self-explanation to a reader, but I'd consider the "single slash" case of user/repo to mean GitHub. I haven't been sure about it and would welcome thoughts on it. Beyond favoring explicitness, I also don't like to favor a single commercial service like that. Comparisons to consider are how nextstrain remote works (explicit URLs are required; actually even for nextstrain.org when it's not groups/…) and nextclade dataset (canonical name is always scope/name, e.g. nextstrain/measles or community/my-org/measles, but aliases allow just measles to mean ours).

Notably, I don't plan to run git because I don't want to require it's installed separately. And I don't think there's a no-deps git reimplementation available for Python that I could pick up today and use. I know of a couple projects (in Rust, in Go) that I could pick up and use for that purpose, but then I'd also have to do the work to bundle them correctly into Nextstrain CLI. I don't really feel like doing that right now. I guess I could use our own runtimes to run git, but that feels… weird? Also, requiring the URL be a Git repo feels wrong in and of itself; too "techy".

Instead, my plan is to have the fully-resolved URL return a ZIP, while leaving it open to support more container formats (e.g. tarballs, Git, etc.) in the future.

What does the on-disk storage look like if somebody forks measles into their own organization but wants to nextstrain setup measles AND nextstrain setup my-org/measles?

My thinking is that nextstrain setup measles expands behind the scenes to something like

nextstrain setup measles=https://api.github.com/repos/nextstrain/measles/zipball
nextstrain setup measles --using https://api.github.com/repos/nextstrain/measles/zipball
nextstrain setup https://api.github.com/repos/nextstrain/measles/zipball --as measles

That is, nextstrain setup not just downloads stuff but is also is the conceptual point where you assign a local name to the pathogen/collection of workflows. So in your example, you'd assign a name to each, e.g. maybe

nextstrain setup measles-nextstrain=measles
nextstrain setup measles-my-org=my-org/measles

Trying to assign both the same name would be an error.

In most common usage, of course, the name will be defaulted (i.e. measles alone will serve as both the name and implicit URL).

Current on-disk storage plan is something like:

~/.nextstrain/pathogen/<name>@<versionX>/
~/.nextstrain/pathogen/<name>@<versionY>/

The "current"/default version to use will be stored in ~/.nextstrain/config, à la runtimes. Unlike runtimes (at least as it stands now), you'll have the option to keep old versions around instead of pruning them on nextstrain update.

Names and versions and full URLs of installed pathogens will be reported in nextstrain version --verbose.

You could set up and use specific versions like:

nextstrain setup measles@v3
nextstrain run measles@v3 …

This probably means you could handle the nextstrain/measles and my-org/measles case a different way if you wanted:

nextstrain setup measles@v3
nextstrain setup measles@my-org=my-org/measles

¹ registering my (probably minority) vote for "something else"; having one verb do two completely different things seems like a bad idea to me…

Nod. The way I see it is that it does two completely different things under the hood, but to users who don't know/care about the implementation, it does the same thing: sets up X for use. I don't think users will care if X is a "runtime" or a "pathogen"; it's all just how they get bits of Nextstrain ready for use on their computer. (And at some point, we may blur the lines between "runtime" and "pathogen" if we start bundling the runtimes into the pathogen to support pathogen-specific arbitrary deps. This is the other half of "workflows as programs", though it may not come to pass.)

And if we do find it's confusing to users, we can do "something else" later, but if we do "something else" now, it'd be hard to put the horse back in the barn later.

(And also, it still may be that I end up finding it too confusing to explain once I spend more time making it a reality.)

⁂ ⁂ ⁂

There's much more to write about that I've noodled on, but this feels like a good stopping point to let some discussion happen.

[tsibley]

@huddlej

…thinking that it's nicer for users if they don't have to leave the command line for a website to figure out how the program works.

I'm very sympathetic to this, but I suspect that while you or I would like to not have to leave the command line to figure out how a program works, we a) are in the minority these days by far, especially among our users and b) already leave the command line for the web browser all the gd time because getting high-quality information (beyond basic reference material) available on the command line is Hard.

(Don't tempt me with the good time of arranging to render our rST docs to nroff for use with man. Or rendering to Python's rich/textual libraries. It's high-effort high-polish I'd love, but that I'm highly dubious will be appreciated more broadly. Maybe you want to convince me otherwise‽)

…but if users rely on the verb to signal the noun that's being operated on, it could get confusing.

Totally. My thinking is that many (most?) users will do the one-time setup of a runtime and a pathogen or two by closely following docs, where understanding the nuance between the nouns is not crucial nor important, and then periodically run nextstrain update without qualification to update the runtime and all the pathogens together without thinking much of it.

[genehack]

Heavily redacting to just comment on the bits I want to comment on 😁

If it's not a bare word ("no slashes"), I'm inclined to require it be a URL for maximum explicitness/self-explanation to a reader, but I'd consider the "single slash" case of user/repo to mean GitHub. I haven't been sure about it and would welcome thoughts on it. Beyond favoring explicitness, I also don't like to favor a single commercial service like that.

Yep, I would be in favor of requiring a full URL for exactly that reason.

Instead, my plan is to have the fully-resolved URL return a ZIP, while leaving it open to support more container formats (e.g. tarballs, Git, etc.) in the future.

I hear you about the lack of a decent Git implementation to do the work, but I just tested the zipball download from Github and the thing that gets downloaded does not expand into a Git repo, and there's a tiny voice in the back of my head that says we're gonna end up regretting that.

Also means that the nextstrain setup usage pattern would not be something we ourselves would use all that often, maybe not at all.

The tiny voice is pretty quiet so maybe it's okay to ignore it.

Current on-disk storage plan is something like:

~/.nextstrain/pathogen/<name>@<versionX>/
~/.nextstrain/pathogen/<name>@<versionY>/

...and by ~/.nextstrain you mean $NEXTSTRAIN_HOME, right? 😁 (says the guy that hates things buried in dot-directories and has export NEXTSTRAIN_HOME=/opt/nextstrain-env set...

¹ registering my (probably minority) vote for "something else"; having one verb do two completely different things seems like a bad idea to me…

Nod. The way I see it is that it does two completely different things under the hood, but to users who don't know/care about the implementation, it does the same thing: sets up X for use.

Sure, I get it — but there's a point where "ease of use" crosses into "obscuring actual functional differences", and for me this is on the wrong side of that line.

This is probably the same tiny voice as before though...

There's much more to write about that I've noodled on, but this feels like a good stopping point to let some discussion happen.

++1

[tsibley]

Heavily redacting to just comment on the bits I want to comment on 😁

Sounds ████████, thanks.

Yep, I would be in favor of requiring a full URL for exactly that reason.

👍

the thing that gets downloaded does not expand into a Git repo, and there's a tiny voice in the back of my head that says we're gonna end up regretting that.

Hmm. Can you say more about why you think we'd regret that? I get the inclination, but I've found myself unable to identify a concrete reason we'd want/need these copies to actually be Git repos.

Also means that the nextstrain setup usage pattern would not be something we ourselves would use all that often, maybe not at all.

Why not? I don't follow. There's nothing preventing the use of a Git repo at $NEXTSTRAIN_HOME/pathogen/<name>@<versionX>/ (i.e. the presence of .git/ there won't break anything), so we could manually symlink/git clone/git worktree things into there if we wanted.

And one usage pattern I didn't mention here yet (though did in convos with Jover) would be potentially supporting that symlink/git clone/git worktree approach for local development, e.g. nextstrain setup measles=. from within measles/, akin to pip install -e ., or nextstrain setup measles=~/nextstrain/measles/ from elsewhere. This is totally doable. I have reservations about doing it though, since ln -s is sitting right there. But if setup involves a few other small steps or it really is more convenient/explainable, it might make sense to officially support it. (Esp. for external workflow authors.)

...and by ~/.nextstrain you mean $NEXTSTRAIN_HOME, right? 😁

o(f)c. :-P

Nod. The way I see it is that it does two completely different things under the hood, but to users who don't know/care about the implementation, it does the same thing: sets up X for use.

Sure, I get it — but there's a point where "ease of use" crosses into "obscuring actual functional differences", and for me this is on the wrong side of that line.

Nod. Hmm. The functional differences being "makes --docker or --conda usable" vs. "makes nextstrain run usable"? Or something else?

I'm sympathetic to not wanting to use nextstrain setup for both, but given the alternatives I've thought thru and their implications, it's the best choice IMO. If you can suggest some other alternative (or reasoning that attenuates the downsides of existing identified alternatives) that's compelling, I'd be interested and open to it!

[genehack]

the thing that gets downloaded does not expand into a Git repo, and there's a tiny voice in the back of my head that says we're gonna end up regretting that.

Hmm. Can you say more about why you think we'd regret that? I get the inclination, but I've found myself unable to identify a concrete reason we'd want/need these copies to actually be Git repos.

As soon as somebody wants to collaborate with us on something, we're gonna want the data in Git. I agree that it might not be a requirement but there are a lot of troubleshooting things that are going to be way harder, I suspect. (I.e., somebody contacts us, they nextstrain setupd something, made some changes/added some data, it was working for a while, now it's broken — how is our first question NOT "what's the diff between the original and what you've got now?", or something similar, which will be much easier to ask even if they haven't been checking their changes in.

Also means that the nextstrain setup usage pattern would not be something we ourselves would use all that often, maybe not at all.

Why not? I don't follow. There's nothing preventing the use of a Git repo at $NEXTSTRAIN_HOME/pathogen/<name>@<versionX>/ (i.e. the presence of .git/ there won't break anything), so we could manually symlink/git clone/git worktree things into there if we wanted.

But if you're manually symlinking/git clone/whatevering, aren't you by definition not using nextstrain setup? You're just sticking the data under $NEXTSTRAIN_HOME yourself.

Sure, I get it — but there's a point where "ease of use" crosses into "obscuring actual functional differences", and for me this is on the wrong side of that line.

Nod. Hmm. The functional differences being "makes --docker or --conda usable" vs. "makes nextstrain run usable"? Or something else?

The difference being runtimes aren't datasets, and I don't see any gain we get out of obscuring that distinction.

I'm sympathetic to not wanting to use nextstrain setup for both, but given the alternatives I've thought thru and their implications, it's the best choice IMO. If you can suggest some other alternative (or reasoning that attenuates the downsides of existing identified alternatives) that's compelling, I'd be interested and open to it!

Yeah, I don't think I have anything better than everything you've already decided against. 🤷 nextstrain install-dataset, maybe?

[tsibley]

they nextstrain setupd something, made some changes/added some data, it was working for a while, now it's broken — how is our first question NOT "what's the diff between the original and what you've got now?"

Ah! I think I see our parity mismatch now: you're expecting nextstrain setup to be how someone gets a pathogen to hack on. I'm expecting it to be how someone gets a pathogen to use with nextstrain run (and nextstrain update). A "managed" pathogen; sort of a "no user-serviceable parts inside" thing (although, of course, we wouldn't prevent tinkering). This is why nextstrain setup puts things "hidden" inside of $NEXTSTRAIN_HOME instead of a chosen location like git clone does. It'd be akin to the Conda or Singularity runtimes, which maintain data in $NEXTSTRAIN_HOME that they expect to have full management/control over (but which nothing stops you from tinkering with).

If someone's at the "hacking on the source" stage of usage, I fully expect them to be managing their own git clone outside of nextstrain setup.

The difference being runtimes aren't datasets, and I don't see any gain we get out of obscuring that distinction.

In my thinking, the user-visible distinction of runtimes vs. pathogens/workflows exists solely as an accidental/incidental implementation detail. Ideally, the whole runtime thing would be subsumed into pathogens/workflows and while it wouldn't be hidden from users, it'd need not be so visible.

I guess put in your terms, I see it the other way: what do we gain from highlighting the (current) distinction?

(Also, I think we're talking about "Nextstrain pathogens", not "datasets".)

aren't you by definition not using nextstrain setup?

Yes, sorry, I elided some thinking there where nextstrain setup stood in for the larger thing it enables: nextstrain run. I think it's more important for us to be routinely using nextstrain run, compared to nextstrain setup, and so the thinking is that we'd manually imitate nextstrain setup sometimes in order to use nextstrain run.

I expect that for development (incl. probably CI), we'd be using either a) nextstrain build or b) nextstrain run + manually-imitated-nextstrain setup. However, I also expect that we'd still routinely use nextstrain run + nextstrain setup for ad-hoc usage of pathogen workflows outside of development.

And like I said, if we didn't want to manually-imitate nextstrain setup, we could have it explicitly support the symlink/git clone/git worktree approach for local development. (Yes, this is not the exact same as exercising the "download a ZIP file" code path, but I don't think it would go completely unexercised amongst our use.)

Yeah, I don't think I have anything better than everything you've already decided against.

How about any reasons why the downsides I described for those aren't actually as downside-y as I think they are?

nextstrain install-dataset, maybe?

I'd considered nextstrain install, but rejected it because having both a setup and install command felt like having an upgrade and update command. Too nuanced a difference, just confusing.

Qualifying it with -dataset (or really, -pathogen?) helps mitigate that, but it feels a) too much like splitting hairs while b) sharing the problems of nextstrain pathogen setup.

[victorlin]

Adding another option for command naming: nextstrain pull. Analagous to docker pull, it could be used for both setup and update/downgrade.

[tsibley]

Adding another option for command naming: nextstrain pull. Analagous to docker pull, it could be used for both setup and update/downgrade.

Hmm. Intriguing. I think pull only works slightly more than install because it feels semantically further from setup. But I doubt the meaning of "pull" is as widely understood as "install" or "setup".

Note that I do want to support not just a single version that you can freely upgrade/downgrade, but multiple concurrent versions.

[victorlin]

Re: install, our docs are currently phrased so that "installing Nextstrain" encompasses installing the CLI as well as installing other tools through setup of a runtime. I don't think install/setup/pull of a pathogen repo should be considered part of "installing Nextstrain". If we choose nextstrain install or something similar to represent installation of a pathogen repo, we would need to reword docs to clarify what "install" means. I can see this causing confusion for users.

We already use other technical terms that are not widely understood (e.g. "runtime", "build"), so I don't see the new terminology "pull" as much of a negative.

[tsibley]

Yes, agreed we shouldn't use nextstrain install.

I don't think install/setup/pull of a pathogen repo should be considered part of "installing Nextstrain".

Ah, I don't think I agree. I think for many people "installing Nextstrain" would naturally include a pathogen of interest to them.

We have a nuanced understanding of all the bits and bobs that are "Nextstrain"¹, but experience suggests to me that most users do not. For example, we very often see folks who (understandably) conflate "Nextstrain" with some specific part of it, e.g. Nextclade. IMO, they shouldn't have to care much about it a lot of the time.

¹ and try to communicate some of that in parts of a whole

We already use other technical terms that are not widely understood (e.g. "runtime", "build"), so I don't see the new terminology "pull" as much of a negative.

Those existing terms are often sources of confusion and require explanation. It'd be nice not to add to that if we have other options.

[victorlin]

Interesting points. I see what you mean and it makes sense under the "workflows as programs" phrasing, but it's a new perspective for me. I'll let the thoughts marinate...

[tsibley]

but it's a new perspective for me. I'll let the thoughts marinate...

👍 🧑‍🍳

⁂

To followup on my own statement:

Yes, agreed we shouldn't use nextstrain install.

At least, shouldn't use both nextstrain install and nextstrain setup as separate and different commands. I'd be fine with some sort of migration from the latter to the former if we think that would better match user intuitions and the way we talk about it in documentation elsewhere.