Status filtering design

[jwhite242]

This is meant to kickstart some design discussions on a simple UI for applying filters to the status outputs to better deal with large studies. First, some high level features to enable:

• sorting
• sub-selecting
• ??

Next, a bit on options for those features:

• control ascending/descending ordering of sorts
• selection options:
  • by step names: list of base (un-parameterized) names
  • by param values/ranges
  • job status

Now for some proposed syntax options.

• SQL: Certainly accomplishing these things would be doable via some existing dsl like sql. That however seems like a far bigger/unwieldy hammer than is actually needed for this job, and one with a rather large potential learning curve.
• Maestro specific DSL: This could lean a bit on SQL, but focused more specifically on the terminology specific to Maestro workflows. Potential ideas for an ~intuitive natural language based UI via examples:
  • sort by +step sort in ascending order by step name (unparameterized)
  • sort by -step sort in descending order by step name
  • sort by exec_order sorts based on order of execution
  • sort by step_param sort by parameterized step names -> unsure of operand step_param here..
  • sort by +param_name sort in ascending order by values of param_name
  • sort by status
  • select steps <list of step names> show only listed step names (unparameterized)
  • select param_name by range <min_val> <max_val> min/max vals for strings would be based on their sort order, capturing all indices in between
  • select param_name >= param_val supports all bool operators
  • select param_name by value <list of values>
  • ...

There's a question of what the default should be. The current default would be execution order but could be changed and configurable with filtering turned on. Said config would likely be more yaml, which could also be an alternate to verbose text input on every call, maybe even borrowing from git's aliasing of log formatting to set your own defaults.

This is meant to be a discussion, so please leave any suggestions/requests/comments below.

[jwhite242]

@FrankD412, @jsemler, @kcathey, @doutriaux1
Just a ping to get some more eyeballs on this. I'm happy to whip of a prototype for a subset of this in a dev branch to try out as well.

[nghiem2]

anything more than sort by alphabetical and execution order is a nice bonus in my eyes :)

[FrankD412]

@jwhite242 -- Thanks for putting this together. For the sorting functionality, I think there are two options (since you mentioned YAML) in my eyes:

1. A simplified set of filtering options to the maestro status command that would simply parse the CSV file in a different order. This might be something like maestro status --key step --descending where key could be step, status, exec_order, and etc. This could be complimented by a --layout csv where it prints the csv contents right to the screen for easier piping to a file.
2. I think the other one is what you were suggesting, where you have a YAML configuration file that is meant to "permanently" alter the order of the status print out. This one could be done in conjunction with the command line options.

For the querying language, I feel like that might be a little overkill considering these aren't really in a database and that gets complex super quickly. Of course, this could also be motivation to finally port to a backend like that, but that's a considerable amount of effort.

These are just some initial thoughts, so just putting these down before they get lost. Happy to discuss further.

[jwhite242]

So, the querying language in the design I have in mind isn't tied to the csv file or an eventual database; much as I want to jump on that, it is indeed a lot of work and beyond the scope of this. The query language is something that will be layered on top of arbitrary sources of status information to make the filtering easier/faster to use by tailoring it to the specific domain of maestro study status info. While a database of course enables more sophisticated options, that would basically expose raw SQL on the command line without some simpler query language layered on top, which sounds not awesome.

Underneath this filtering DSL, the only operations this current proposal intends to do boil down to rearranging and/or selecting subsets of rows from the status table and thus doesn't yet need the power of a database backing it. If there are other use cases we'd want down the line that I've missed here that go beyond that then that's something I'd like to capture with this discussion as that would perhaps motivate changes to any DSL we'd put into the cli (or make accessible via other means).

As to the mechanics, in some other current experiments with the status UI I already have most of the sub-selecting working with only using that filter_dict in the existing status layouts (https://github.com/LLNL/maestrowf/blob/develop/maestrowf/__init__.py#L105). Of course upon implementing this it's become clear this isn't the right place to process said dictionary, but that's a different problem. I think the only feature here that is looking awkward without a status/metadata refactor is having to get the step base name out of the metadata; so far everything else is looking easy to do on the actual csv data filtering side.

Finally, on your first point, I'm not entirely clear what you want to do with piping the contents of the csv (to a file or arbitrary things in the shell) that wouldn't better be handled via directly opening the csv file? The #400 issue ticket prompting this is basically doing that already and just looking for some of the easy options baked into the cli directly.

[crkrenn]

All, (cc @jsemler @daub1 @antimatterhorn)

I finally responded to some changes Frank requested on #381.

It adds two new layouts ("summary" (see below) and "narrow2"), but I'm also thinking that users might want even more flexibility.

What do you all think of an option for maestro "--renderers " that would import one or more renderer classes from an external file?

V/r,

-Chris

class SummaryStatusRenderer(FlatStatusRenderer):
    """Flat, simple summary table layout"""

    layout_type = "summary"        # Defines name in factory/cli

    def update_theme(self):
        self._theme_dict["Step Prefix"] = "blue"
        self._theme_dict["State"] = "bold red"
        self._theme_dict["Count"] = "blue"

    def process_status_data(self, status_data):
        """Construct the summary dictionary"""
        summary_data = {
            "Step Prefix": [],
            "State": [],
            "Count": [],
        }

        status_info = zip(status_data["Step Name"], status_data["State"])
        status_info = sorted(status_info, key=lambda x: x[1])
        working_data = defaultdict(list)
        for step_name, state in status_info:
            prefix = step_name.split("-")[0]
            working_data[prefix].append(state)

        for prefix, states in working_data.items():
            counts = Counter(states)
            summary_data["Step Prefix"] += ([prefix] * len(counts))
            summary_data["State"] += counts.keys()
            summary_data["Count"] += counts.values()

        return summary_data

[jwhite242]

So one distinction that's a bit muddled here since filtering isn't available yet: these layouts appear to just be filters, i.e. sub-selecting the columns to return. Contrast that with the other 'narrow' layout which pivots the columns in each row to make things with long paths/many variables viewable on terminals that don't span ultra-wide monitors. In light of that for future uses/additions to the status outputs, are there other layouts/views that you think would be useful in addition to just sub-selecting the data that goes into the table?

[crkrenn]

@jwhite242 (cc @FrankD412),

I believe that the two "layouts"/"filters" I proposed (#381) would be very valuable for both our forensics team and for other users running large studies, but I also think that it is really hard to predict what users will want.

So, I think that a "plugin" solution for status display (like we have for --pgen) would be a valuable addition and offer a great deal of "bang for the buck". These plugins could do filtering, sorting, and formatting for "power" users, and these users could save and modify these formats without touching the maestro source code, or even learning a bunch of new command line options.

I have no objections to also adding sorting and filtering options, and I think that these options could easily work both independently and cooperatively with any status plugin options that are added.

I'm also a little torn... I'd like the "--layout summary" and "--layout narrow2" in #381 to be added to maestro, but I also understand if it might make more sense for this integration to wait for a plugin architecture and for these filters to be put in an examples or sheet music directory.

Our forensics team does agree that the current maestro status format is unreadable with large studies using multiple steps and multiple sampling variables, though piping maestro status output into grep was useful for some users.

Does this make sense?

-Chris

[jwhite242]

Sure, and I didn't mean to suggest we don't push those out given they seem to be more immediately useful. My comment was aimed more at future work. The way it's setup now there's a nice place for the filtering in between the status file reading and the sending of that data to the layout renderers and thus the renderers can be mostly unaware of the data that's sent into them instead of doing the filtering themselves; granted this only works on the filtering prototype right now and thus I don't see a good reason to hold these up.

I do like your idea for user definable layouts and think it's a great feature to add for this. I just want to get the api's/guidelines for it ironed out so it's easy for users to define some of those without having to fully understand the inner machinery in the filtering and the layouts (and worse any coupling points between them) to get something working. If we're not careful that extra flexibility will end up too cumbersome to use/too fragile.

But for the question of layouts, that is aimed more at your last point about large studies being difficult to work with on the status -> what is the pain point(s) there that makes it so? Is it just long named things that wrap in the columns or something? The flat table layout that we currently have does seem pretty limiting there even with reducing the numbers of columns given long names in certain fields will still be a problem. But, I'd also be interested in hearing if there are other problems there as I'm sure I've not run into all of them myself; perhaps there are other layouts that can make that better in a manner orthogonal to any filtering -> e.g. a brief version of the narrow layout with less info, or even an alternate naming convention for things in the status similar to your make-links idea in #329. So there's two concerns I see here: filtering to help sort through the data (i.e. look at completed stuff only, failed only, certain params only, ...) and alternate layouts/formats to make it easier to read that set of data (long path names/hashing being of concern here).

[crkrenn]

Hi Jeremy (cc: @daub1, @jsemler @antimatterhorn)

I can try to send you some examples of our pain points on the "red" system.

The problem for us, using the device assessment framework and codepy is both long names per combination and many steps per combination, many of which (like setting permissions and post-processing) are not of primary interest to the user because they are completed very quickly.

Filtering will help with trimming the number of steps.

I'd like to hope that the "--make-links" that I'm working on will help with the long names problem. The links can serve as aliases that status and other post-processing tools (like codepy table) can optionally use.

The "summary" layout, combined with step filtering, should help solve the problem of monitoring 300 combinations at once, which is also a common use case.

-Chris

PS. Random numbers in our combinations also definitely increase the length of labels that are squeezed into columns.

[{'X1': 20, 'X2': 'foo', 'X3': 9.579442103034559, 'X4': 6.086368664670074}, {'X1': 20, 'X2': 'foo', 'X3': 5.9697877768415255, 'X4': 6.475883924108136}, {'X1': 20, 'X2': 'foo', 'X3': 7.930316084131929, 'X4': 7.97859273265832}, {'X1': 20, 'X2': 'foo', 'X3': 9.52247269430977, 'X4': 9.145262638361192}, {'X1': 20, 'X2': 'foo', 'X3': 9.869335024338316, 'X4': 9.091055669151563}]