Suggestions for launching/managing multiple processes with maestro?

[bryan-garcia]

Hi all,

I am working with a tool that is itself a workflow manager that manages simulation execution and training with simulation results. The simulation & training cycle repeats within the lifespan of the workflow tool which lives on a login node, and the total run time of the various cycles can cumulatively take well over the allotted compute time.

My question is: can maestro launch multiple processes (rather than jobs) in order for me to utilize my workflow tool in a parallel fashion? I've been able to use the 'batch' entry within the maestro file to coordinate the launch of multiple jobs which run my workflow tool on compute nodes. That works for small simulations, but running out of compute time is bound to happen as I scale.

Thanks for any suggestions, and I'd be happy to clarify anything that isn't clear!

[FrankD412]

Hi Bryan! -- Sorry lost track of this! I'm planning to respond soon, just wanted to ping that I found this.