Support for parsl in Galaxy?

[bernt-matthias]

For my reference https://parsl.readthedocs.io/en/latest/userguide/execution.html

We need to explore if the data-paralellism offered by Galaxy (via collection) is sufficient for the more compute intensive jobs coming with metagenomics. Or if it is possible to support parsl jobs in Galaxy.

@ebolyen can you give me a link for docs how the configuration, e.g. for an execution in SLURM would look like? You mentioned toml based config files.

[ebolyen]

Hey @bernt-matthias,

We use toml internally but the way parsl is generally configured would be as a Python object that is initialized and represents the config.

The relevant docs are here:
https://parsl.readthedocs.io/en/latest/userguide/configuring.html#how-to-configure
https://parsl.readthedocs.io/en/latest/stubs/parsl.providers.SlurmProvider.html#parsl.providers.SlurmProvider
https://develop.qiime2.org/en/latest/framework/how-to-guides/parallel-configuration.html

---

Another approach is to use our partition methods directly and then use Galaxy's built in collection system, rather than trying to get parsl and galaxy's various backends to play together.

[bernt-matthias]

use Galaxy's built in collection system

Seems much simpler. Lets see how far we get with this.

[ebolyen]

I've created a new issue to make sure that the way we implement these parallel pipeline (via hidden actions) will be accessible to Galaxy users so they can do the map operation via Galaxy directly:
qiime2/q2galaxy#57