CWL/common workflow language

[crkrenn]

@FrankD412 @jsemler @daub

I just read about cwl in the acm communications.

What are your thought about supporting this with maestro?

Chris

https://m-cacm.acm.org/magazines/2022/6/261172-methods-included/fulltext

https://github.com/common-workflow-language/common-workflow-language

[doutriaux1]

@crkrenn I think it would be a great addition to Maestro and we should try as hard as possible to achieve this. @FrankD412 @jwhite242 what do you think?

[FrankD412]

@crkrenn @doutriaux1 -- Thanks for kicking off this conversation. I've been thinking on this for a few days and so far I've come up with the following:

1. As Maestro is designed, I'm not sure there is a benefit of directly supporting CWL. Maestro's design would have to be changed in order to implement a backend conductor that would parse CWL, plus it would then have to be supported if the standard changed. By nature, Maestro was meant to be a lightweight orchestration with a bit of minimal infrastructure and a nice user-facing specification. The CWL very much seems like a standard aiming for portability and by nature is aiming for tools with backend infrastructure.

   • Maestro is in this weird in-between where you have the specification -> DAG -> orchestration, but the orchestration is coupled to the DAG and not to general APIs.
   • I could imagine a conversion script that could take a Maestro spec and create a CWL spec from it, and vice versa. I could imagine the CWL being used by some frontend GUI perhaps that wanted to bridge multiple workflow tools, but for Maestro's current system structure I have a hard time seeing where it truly fits in.

2. The number of workflow systems that support the CWL isn't that large. It currently seems that there might not be a large amount of benefit to implement this short term.

3. Some striking limitations that exist in the CWL that Maestro would like to support make me hesitant (the following are listed right in the article):

   • Advanced control-flow techniques beyond conditional steps.
   • Workflows that contain cycles: "Repeat this step or sub-workflow a specific number of times" or "Repeat this step or sub-workflow until a condition is met."

I'm not saying there aren't any benefits to implementing it, but to me the benefits are outweighed by the potential effort and return on that investment. These are just my initial thoughts to kick off the discussion, so happy to talk further.

[danlaney]

LANL’s BEEFlow workflow manager uses CWL and they worked with the CWL team to add features needed for MPI invocations and containerized apps I believe. They also have a front-end that takes in CWL and tracks their workflows in a Neo4Js graph database and I think they can leverage some CWL tools to visualize DAGs and such. Not sure how production ready the later capabilities are. Dan From: crkrenn ***@***.***> Reply-To: LLNL/maestrowf ***@***.***> Date: Saturday, June 11, 2022 at 7:23 AM To: LLNL/maestrowf ***@***.***> Cc: Subscribed ***@***.***> Subject: [LLNL/maestrowf] CWL/common workflow language (Discussion #401) @FrankD412<https://urldefense.us/v3/__https:/github.com/FrankD412__;!!G2kpM7uM-TzIFchu!jR4fa3Ms2Th_d85d9FRfuoP5FIg1cPuZF_4iJwuuFbtu_nwrnVjnnW36bDE9UHSR$> @jsemler<https://urldefense.us/v3/__https:/github.com/jsemler__;!!G2kpM7uM-TzIFchu!jR4fa3Ms2Th_d85d9FRfuoP5FIg1cPuZF_4iJwuuFbtu_nwrnVjnnW36bNImxHNB$> @daub<https://urldefense.us/v3/__https:/github.com/daub__;!!G2kpM7uM-TzIFchu!jR4fa3Ms2Th_d85d9FRfuoP5FIg1cPuZF_4iJwuuFbtu_nwrnVjnnW36bPOek7yF$> I just read about cwl in the acm communications. What are your thought about supporting this with maestro? Chris https://m-cacm.acm.org/magazines/2022/6/261172-methods-included/fulltext<https://urldefense.us/v3/__https:/m-cacm.acm.org/magazines/2022/6/261172-methods-included/fulltext__;!!G2kpM7uM-TzIFchu!jR4fa3Ms2Th_d85d9FRfuoP5FIg1cPuZF_4iJwuuFbtu_nwrnVjnnW36bDIk53bX$> https://github.com/common-workflow-language/common-workflow-language<https://urldefense.us/v3/__https:/github.com/common-workflow-language/common-workflow-language__;!!G2kpM7uM-TzIFchu!jR4fa3Ms2Th_d85d9FRfuoP5FIg1cPuZF_4iJwuuFbtu_nwrnVjnnW36bI1tNR5B$> — Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https:/github.com/LLNL/maestrowf/discussions/401__;!!G2kpM7uM-TzIFchu!jR4fa3Ms2Th_d85d9FRfuoP5FIg1cPuZF_4iJwuuFbtu_nwrnVjnnW36bK9Do2nn$>, or unsubscribe<https://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AAJ6RDNBR6NA7XNATGC24JDVOSOMNANCNFSM5YQFB2WQ__;!!G2kpM7uM-TzIFchu!jR4fa3Ms2Th_d85d9FRfuoP5FIg1cPuZF_4iJwuuFbtu_nwrnVjnnW36bOP0Lw2r$>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>