Refactoring for Hvplot batch plotting #158

asewnath · 2023-09-29T19:48:58Z

Description

We would like to use eva to create batches of hvplot figures for future web development work. In order to do this, the figure driver needs to be generalized to take in multiple backends. Then, the emcpy and hvplot backends would be selected based on the plotting_backend field in the config yaml. A handler was created to direct each backend to it's associated CreatePlot and CreateFigure classes.

Please note the new test yaml: testHvplotIodaObsSpaceAmsuaN19.yaml. The hvplot backend will expect the same yaml configurations that emcpy does.

Currently, only the scatter plot works for the hvplot backend. We've decided that it would be best to get feedback on the restructuring before we moved on with further hvplot development considering there are a lot of changes.

The new directory structure has two main directories under plotting: batch and interactive. Under the batch directory will be a base directory and backend directories that will inherit from the base directory. In the future, all diagnostics will inherit data prepping/processing from the base diagnostic classes.

danholdaway · 2023-09-29T20:02:57Z

I probably wont have time to review before furlough. But just one quick spot and comment. Since the YAMLs do have to change I think we might as well make the backend a required argument. I.e. add EMCpy to all the test YAMLs.

CoryMartin-NOAA · 2023-09-29T20:45:13Z

Ditto for me with what Dan said :-( here's hoping I see GitHub again soon

danholdaway · 2023-10-02T21:10:35Z

Could you update the requirements.txt to include these new packages? I think we should have three requirements packages:

requirements_github.txt
requirements.txt
requirements_emc.txt

The first one used by github stays as is. The second one should be the same as the first one but with >= instead of == and not include the things needed to build the docs. The third _emc would be the same as the current requirements.txt. I.e. not have seaborn, bokeh, hvplot etc. Would that arrangement work for you @CoryMartin-NOAA?

Would it be possible to create a situation where if the chosen backend is hvplot but this package is not available we get a sensible error?

try:
    import hvplot
except ImportError:
    logger.abort("The hvplot backend is not available since hvplot is not in the environment.")

CoryMartin-NOAA · 2023-10-03T12:10:41Z

Yes, @danholdaway that sounds perfect.

danholdaway

Just a couple of very minor comments. Overall this looks great. I like that you have have essentially the same YAML with just one key different and can make the plots using EMCpy and HVplot. Thanks for working on this.

src/eva/plotting/batch/base/plot_tools/figure_driver.py

danholdaway · 2023-10-05T14:48:40Z

One other thing. Can you make a minor version bump in setup.py to 1.6.0? Then once we have a fully working hvPLot backend perhaps we make it eva 2.0 since that would be a significant new feature.

danholdaway · 2023-10-05T15:26:27Z

I just pushed a little clean up to the requirements. There were still some lingering things in setup.py that are replaced by what's in all the requirements.txt files. I figure we should minimize the number of places we have these dependencies. I also added a requirements file that is used when installing with spack-stack 1.4

CoryMartin-NOAA

You've got a swap file in here.

setup.py

src/eva/tests/config/testCubedSphereRestart.yaml

CoryMartin-NOAA

A few extraneous swapfiles merged in, but beyond that (and figuring out the Actions failure) this looks good to me.

asewnath · 2023-10-05T17:40:18Z

You've got a swap file in here.

This reverts commit 9214290.

asewnath · 2023-10-05T18:19:04Z

@danholdaway I just pushed a branch that deleted your changes, and I tried to revert but I don't see files like the spack stack requirements file anymore. Can you revert back to the changes you made and then I can try again with the review changes?

…internal/eva into feature/bokeh_batch_plots

danholdaway · 2023-10-05T19:33:21Z

@danholdaway I just pushed a branch that deleted your changes, and I tried to revert but I don't see files like the spack stack requirements file anymore. Can you revert back to the changes you made and then I can try again with the review changes?

I think they are still there when I look at https://github.com/JCSDA-internal/eva/tree/feature/bokeh_batch_plots

asewnath · 2023-10-05T19:52:12Z

@danholdaway I just pushed a branch that deleted your changes, and I tried to revert but I don't see files like the spack stack requirements file anymore. Can you revert back to the changes you made and then I can try again with the review changes?

I think they are still there when I look at https://github.com/JCSDA-internal/eva/tree/feature/bokeh_batch_plots

I think something is going on on my end because I'm not seeing your commit in my commit history. Do you mind sharing the commit id?

danholdaway · 2023-10-05T20:02:52Z

Can you

@danholdaway I just pushed a branch that deleted your changes, and I tried to revert but I don't see files like the spack stack requirements file anymore. Can you revert back to the changes you made and then I can try again with the review changes?

I think they are still there when I look at https://github.com/JCSDA-internal/eva/tree/feature/bokeh_batch_plots

I think something is going on on my end because I'm not seeing your commit in my commit history. Do you mind sharing the commit id?

Can you try doing a git pull? Otherwise perhaps just reclone the repo? I'm at commit d54420c

danholdaway · 2023-10-05T20:03:11Z

Oh maybe I hadn't pushed everything I had locally. I just did a push and something went.

asewnath · 2023-10-05T20:06:08Z

Oh maybe I hadn't pushed everything I had locally. I just did a push and something went.

I don't know why, but whatever you did made all of your commits reappear in my commit history (like the one from a few hours ago). Odd github behavior, but we're back on track now

CoryMartin-NOAA

Looks like the tests pass now with that requirements change, but we still have loitering swapfiles committed.

asewnath · 2023-10-05T20:10:15Z

Looks like the tests pass now with that requirements change, but we still have loitering swapfiles committed.

Yep, those will be taken care of in just a moment

## Description This PR expands upon the hvplot backend introduced in #158. All other diagnostic types are reformatted to have a base class and child classes for each backend. Each base class includes two methods: `data_prep` and `configure_plot`. `configure_plot` is a virtual method that is resolved once a child class for either emcpy or hvplot is instantiated. `figure_driver` is updated to reflect these changes

asewnath added 9 commits September 11, 2023 16:12

bokeh batch plot

a47ecce

bokeh components

0b92c20

first pass diag inherit

0c70334

adding figure handler

5713088

bokeh scatter plot

e401068

more hvplot changes

9bb34a3

adding figure_list to tests

0214ed8

adding in new test for hvplot

6e48b50

python code style

63d7112

Merge branch 'develop' into feature/bokeh_batch_plots

36679e3

asewnath requested review from CoryMartin-NOAA, ADCollard, EdwardSafford-NOAA, kevindougherty-noaa and danholdaway September 29, 2023 20:42

enforcing plotting backend in yamls

7d891ae

asewnath added 2 commits October 3, 2023 14:04

requirements

05bbe21

coding norms

4c8a783

danholdaway approved these changes Oct 5, 2023

View reviewed changes

src/eva/plotting/batch/base/plot_tools/figure_driver.py Outdated Show resolved Hide resolved

src/eva/plotting/batch/base/plot_tools/figure_driver.py Show resolved Hide resolved

src/eva/plotting/batch/base/plot_tools/figure_driver.py Show resolved Hide resolved

Sort out requirements files a bit

971fd1d

CoryMartin-NOAA reviewed Oct 5, 2023

View reviewed changes

setup.py Show resolved Hide resolved

CoryMartin-NOAA reviewed Oct 5, 2023

View reviewed changes

src/eva/tests/config/testCubedSphereRestart.yaml Show resolved Hide resolved

CoryMartin-NOAA reviewed Oct 5, 2023

View reviewed changes

emcpy in github requirements

b644d46

review changes

9214290

asewnath force-pushed the feature/bokeh_batch_plots branch from b644d46 to 9214290 Compare October 5, 2023 18:07

Revert "review changes"

9d20f08

This reverts commit 9214290.

Merge branch 'feature/bokeh_batch_plots' of https://github.com/jcsda-…

d54420c

…internal/eva into feature/bokeh_batch_plots

danholdaway mentioned this pull request Oct 5, 2023

Bring YAMLs up to date with new eva and parallelize the EvaObservation task GEOS-ESM/swell#255

Merged

1 task

CoryMartin-NOAA reviewed Oct 5, 2023

View reviewed changes

review changes again

797a005

CoryMartin-NOAA approved these changes Oct 5, 2023

View reviewed changes

danholdaway merged commit 816f162 into develop Oct 5, 2023
5 checks passed

danholdaway deleted the feature/bokeh_batch_plots branch October 5, 2023 20:27

asewnath mentioned this pull request May 16, 2024

Adding the hvplot backend #186

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactoring for Hvplot batch plotting #158

Refactoring for Hvplot batch plotting #158

asewnath commented Sep 29, 2023 •

edited

Loading

danholdaway commented Sep 29, 2023

CoryMartin-NOAA commented Sep 29, 2023

danholdaway commented Oct 2, 2023 •

edited

Loading

CoryMartin-NOAA commented Oct 3, 2023

danholdaway left a comment

danholdaway commented Oct 5, 2023

danholdaway commented Oct 5, 2023

CoryMartin-NOAA left a comment

CoryMartin-NOAA left a comment

asewnath commented Oct 5, 2023

asewnath commented Oct 5, 2023

danholdaway commented Oct 5, 2023

asewnath commented Oct 5, 2023

danholdaway commented Oct 5, 2023

danholdaway commented Oct 5, 2023

asewnath commented Oct 5, 2023

CoryMartin-NOAA left a comment

asewnath commented Oct 5, 2023

Refactoring for Hvplot batch plotting #158

Refactoring for Hvplot batch plotting #158

Conversation

asewnath commented Sep 29, 2023 • edited Loading

Description

danholdaway commented Sep 29, 2023

CoryMartin-NOAA commented Sep 29, 2023

danholdaway commented Oct 2, 2023 • edited Loading

CoryMartin-NOAA commented Oct 3, 2023

danholdaway left a comment

Choose a reason for hiding this comment

danholdaway commented Oct 5, 2023

danholdaway commented Oct 5, 2023

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

asewnath commented Oct 5, 2023

asewnath commented Oct 5, 2023

danholdaway commented Oct 5, 2023

asewnath commented Oct 5, 2023

danholdaway commented Oct 5, 2023

danholdaway commented Oct 5, 2023

asewnath commented Oct 5, 2023

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

asewnath commented Oct 5, 2023

asewnath commented Sep 29, 2023 •

edited

Loading

danholdaway commented Oct 2, 2023 •

edited

Loading