Pre/post usage?

[bryan-garcia]

Hi all,

I've been using maestrowf for a couple of weeks and noticed while poking through the repo there are pre and post step
entries in the YAML study schema. Are these features available for use?

In pseudocode, I would like to do something like:

run:
  pre: Get latest model data from previous combination of parameters.
  cmd: Train lastest model with current combination of parameters.
  post: Update latest model for next combination of parameters.

Since my cmd is a non-blocking operation, one can see how this post step would be useful.

Thanks for any help or suggestions!

[FrankD412]

Hi Bryan,

At the moment those features are not available. When they are implemented, they are intended to be run locally on a login/head node and should be lightweight operations. It sounds like you're intending the post to run right after execution of the cmd and to overlap with its execution (if I'm understanding correctly)?

If you're looking for a way to do it in sequence, you can break that down to the following:

- name: get_data
  description: Get latest model data from previous combination of parameters.
  run:
    cmd: <run data scrape>
- name: train
  description: Train lastest model with current combination of parameters.
  run:
    cmd: <train model>
    depends: [get_data]
- name: update_params
  description: Update latest model for next combination of parameters.
  run:
    cmd: <update parameters>
    depends: [train]

If that's not 100% what you're looking for, happy to discuss requirements for the pre and post steps.

[bryan-garcia]

Thanks for the reply, Frank.

I would like post to run right after cmd with the current set of parameter combinations. Similarly, I would like pre to run right before cmd with the next set of parameters obtained from the generator. If I'm not mistaken, this sounds like the intended implementation of the features.

From my understanding of the study layout and the depends entry, step get_data would only be run once before the execution of the train step , and update_params would only run after the train step, rather than before and after each execution of cmd with the combination of parameters yielded from my pgen generator. Currently, this is not exactly what I need for this particular study.

In summary: my goal is to obtain a single model trained on data from all "trials" generated from my pgen. A pre/post step would allow me to ensure the next parameter combination of a step will have access to the most recent model.

[FrankD412]

Hi Bryan,

pre and post would be intended as small, local, scripts that would run before and after each parameterized step. The post capability would also be used to possibly drive workflow decisions based on a return code driven standard. So, for example, if a job completed and you could easily search or parse something to see if a simulation actually finished you could then return a code that says it's done or restart it. I think in terms how how you describe pre and post you're expecting that they'd run once for a step for all parameters?

From your last post, I think you're misunderstanding how parameters are applied a little bit. I'm also confused because I don't know if you're dealing with get_data on a data set level, or data on a single point. So I'll try and address both. As an example, I will point to the LULESH sample specification in the repository. In that case, the behavior is that we run make-lulesh once, followed by a number of parameterized versions of run-lulesh followed by some other steps that can be varied based on new or old parameters. For your purposes, let's look at the behavior of make-lulesh and run-luslesh.

If you are pulling a singularly large dataset and know what data is in the set, you can do a get_data step (no parameters), train_model (applying each parameter combination), and then update_model (again, each combination applied). This would look like

- name: get_data
  description: description: Get latest model data from previous combination of parameters.
  run:
    cmd: <run data scrape>
- name: train_model
  description: Train lastest model with current combination of parameters.
  run:
    cmd: <train model> $(P1)
    depends: [get_data]
- name: update_model
  description: Update latest model for next combination of parameters.
  run:
    cmd: <update parameters> $(P1)
    depends: [train_model]

global.parameters:
   P1:
        values  : [1, 2, 3, 4, 5, 6, 7, 8, 9]
        label   : P1.%%

In this case, get_data will be run once with multiple train_model steps following it for each value of P1. You then would use P1 again in the update_model step to apply that to your model (assuming it's process-safe). This would create chains for each value of P1 where you would form a workflow tree with a root of get_data and then two step branches for each value. NOTE: I applied $(P1) to update_model but you can use $(train_model.workspace) to get the parameterized workspace. In both cases the graph is the same as below:

If you need to pull multiple points from a reference set of data and then train your model, you can do something like the following:

- name: get_data
  description: description: Get latest model data from previous combination of parameters.
  run:
    cmd: |
       <run data scrape> $(P1)
       echo "DONE"
- name: train_model
  description: Train lastest model with current combination of parameters.
  run:
    cmd: <train model> $(get_data.workspace)
    depends: [get_data_*]

global.parameters:
   P1:
        values  : [1, 2, 3, 4, 5, 6, 7, 8, 9]
        label   : P1.%%

In this case, you're getting a bunch of data points and then singularly training a model. A few things to note:

1. _* denotes a "funnel" -- this means that the step is dependent on all combination of the specified step and not just a single one. This will hold up training until all data is gathered.
2. $(get_data.workspace) will return to you the workspace that contains all of the parameterized directories containing the data. In this case, you will have to write a script or bash to iterate through the data.
3. The echo "DONE" at the end of the get_data step is required to guard against failures in retrieving data -- Maestro determines success by return code or scheduler completion status. When using _* if any step fails, it will prevent the dependent step from running so the echo guarantees a successful return and makes it so that your custom code has to handle failures.

How this ends up looking:

Hopefully this makes sense. I'm going to be away handling something and might not be able to respond immediately. If you need specific help or anything is unclear here, please let me know and we can schedule something to talk through your use case.