Total PowerAnalytics Redesign: `ComponentSelector`s and `Metric`s #24

GabrielKS · 2024-09-30T07:15:53Z

See https://github.nrel.gov/gkonars/PowerAnalytics-demos/blob/main/new_pa_pr_demo.ipynb for a full demo. Depends on NREL-Sienna/InfrastructureSystems.jl#342 and NREL-Sienna/PowerSystems.jl#1197.

This is the first commit in a major PowerAnalytics redesign.

Previously, the test simulations only contained one RenewableDispatch component and did not dispatch any thermal generation. Now there is a solar RenewableDispatch component, and nonzero thermal generation for 2/5 generators. The old PowerAnalytics test still work.

Additional changes in this commit: - Generalize compute_all to take Components - Make create_problem_results_dict return SortedDict

Rather than setting `agg_fn`/`reduce_fn` when calling `compute` or `aggregate_time`, metrics now specify how they should be aggregated across Entities and across time.

- Add and test the ability to have `entity_agg_fn` and `time_agg_fn` use column element-wise "aggregation metadata" for things like weighted averages - Add options for how to aggregate the aggregation metadata itself (insufficiently tested) - Add option to specify what the result of `compute` looks like when there are no components (insufficiently tested) - Fix built-in curtailment fraction and integration calculations using weighted_average

I'm adding myself as the first author to reflect the fact that this redesigned version of the package is meant to be the main one in use going forward.

jd-lara · 2024-10-08T21:00:23Z

src/metrics.jl

+is_available_for_analytics(comp::Component) =
+    try
+        PSY.get_available(comp)
+    catch e  # A bit hacky but the alternative is hardcoding which component types don't have a `get_available` (or using `hasmethod`, which is probably worse)
+        (e isa MethodError) && return true
+        rethrow(e)
+    end
+


Suggested change

is_available_for_analytics(comp::Component) =

try

PSY.get_available(comp)

catch e # A bit hacky but the alternative is hardcoding which component types don't have a `get_available` (or using `hasmethod`, which is probably worse)

(e isa MethodError) && return true

rethrow(e)

end

function is_available_for_analytics(comp::T)

if hasmethod(PSY.get_available, Tuple{T})

return PSY.get_available(comp)

else

return true

end

My impression was that hasmethod isn't really supposed to be used in production code like this (see e.g. here) but happy to switch if the consensus is it's less bad than the try/catch

It's an exported function with proper docstrings! I'm not sure about that comment. Anyway, we could implement get_available(x::Component) = true. I'll defer to @jd-lara if that is a reasonable default. It has the same outcome here.

While hasmethod certainly isn't ideal, it is faster than try/catch: https://discourse.julialang.org/t/performance-of-hasmethod-vs-try-catch-on-methoderror/99827.

I agree with what that commenter would likely say...this is ill-formed dynamism. A default get_available is obviously still better.

Agree that the best option for this use case is to define get_available on all Components — @jd-lara does that work for you?

Without objection, moving forward with defining

get_available(::Component) = true

in PSY to eliminate this.

daniel-thom · 2024-10-09T12:05:44Z

src/PowerAnalytics.jl

+export mean, weighted_mean, unweighted_sum
+
+# IMPORTS
+import Base: @kwdef


It's already public.

Not in older minor versions we commit to supporting (see NREL-Sienna/InfrastructureSystems.jl#390)

I see. @jd-lara Is it time to force users to upgrade? 1.11 is out. If people haven't upgraded to at least 1.9, they probably should.

daniel-thom · 2024-10-09T12:06:39Z

src/PowerAnalytics.jl

 const PSY = PowerSystems
 const IS = InfrastructureSystems
 const PSI = PowerSimulations

+# INCLUDES
+# Old PowerAnalytics


Does there need to be an old and new?

I'm trying to maintain a strict separation between the two interfaces until we decide what to do with the old one (e.g., should it be deprecated once PowerGraphics no longer depends on it)?

Can this be a 1.0 vs 2.0 kinda thing? Basically, do we need to maintain support for the old methods in the new version if those users can continue to use the old version?

One issue is that the current latest version of PowerGraphics depends on old PowerAnalytics and I don't want to keep PowerGraphics users from using new PowerAnalytics. I'm thinking something like this:

Release new PA, keep old PA as is

Develop a user base on new PA, fix some bugs

Deprecate old PA (i.e., throw warnings) but keep it so PG can use it

Revamp PG to only depend on new PA (I would be interested in supervising an intern to do this)

Remove old PA

but not committed to that, open to discussion.

Sounds good to me.

daniel-thom · 2024-10-09T12:07:17Z

src/PowerAnalytics.jl

+# SUBMODULES
+using .Selectors
+using .Metrics
+
 greet() = print("Hello World!")


Can we delete this?

Yeah, will do. Surprised that this is the only package of ours that still has it

daniel-thom · 2024-10-09T12:09:01Z

src/builtin_component_selectors.jl

+
+# Parse the strings in generator_mapping.yaml into types and enum items
+function parse_fuel_category(category_spec::Dict)
+    # Use reflection to look up the type corresponding the generator type string (this isn't a security risk, is it?)


This is fine. Comment not needed.

daniel-thom · 2024-10-09T12:27:48Z

src/builtin_component_selectors.jl

+    end
+
+    # Create a nice name that is guaranteed to never collide with fully-qualified component names
+    selector_name = join(ifelse.(isnothing.(parse_results), "", string.(parse_results)),


So if prime_mover and fuel_category are both nothing, the name will be "ThermalStandard____"? That will look like a bug to the user.

Secondly, calling string(gen_type) will produce different results depending on whether you use import or using to load the package. Someone could decide to change the method in this package and mysteriously cause these names to change.

daniel-thom · 2024-10-09T14:05:41Z

test/test_input.jl

+decision_problem_names = ("UC", "ED")
+my_results_sets = get_decision_problem_results.(Ref(sim_results), decision_problem_names)
+
+# TODO is there a better way?


compare_values?

daniel-thom · 2024-10-09T14:14:30Z

test/test_input.jl

+    end
+end
+
+# Reimplements Base.Filesystem.cptree since that isn't exported


It seems crazy that Julia does not provide a way to do this. Since this is just a test, I would have just called the internal function, but am not asking for a change.

Oh yeah, this is the basic filesystem function that Julia doesn't provide that I was trying to think of in a recent discussion of ours

daniel-thom · 2024-10-09T14:17:52Z

test/test_metrics.jl

+@assert all(
+    in.("ActivePowerVariable__ThermalStandard", list_variable_names.(values(resultses))),
+) "Expected all results to contain ActivePowerVariable__ThermalStandard"
+comp_results = Dict()  # Will be populated later


Does this introduce risks of tests colliding with each other?

It does seem like a kind of bad use of global state, I'll look into changing this

daniel-thom · 2024-10-09T14:22:30Z

test/setuptests.jl

+const BASE_DIR = dirname(dirname(pathof(PowerAnalytics)))
+const TEST_DIR = joinpath(BASE_DIR, "test")
+const TEST_OUTPUTS = joinpath(BASE_DIR, "test", "test_results")
+!isdir(TEST_OUTPUTS) && mkdir(TEST_OUTPUTS)


Can you use a tmpdir for all outputs?

This is roughly what PA had before and it's been useful to cache these results so we don't have to regenerate them every time we run the tests

Ah, it's for caching. Can you add a comment?

daniel-thom · 2024-10-09T14:23:37Z

test/test_builtin_metrics.jl

@@ -0,0 +1,121 @@
+# For now we are mostly just testing that all the metrics can be called without error,


Is this still correct? I see plenty of places where you are checking the values returned by functions like aggregate_time. Are there any important functions where you aren't validating the returned values?

The "infrastructure," including everything in metrics.jl, output_utils.jl, etc., gets tested pretty thoroughly; this comment only applies to the built-in metrics in PowerAnalytics.Metrics — like, we aren't testing that calc_active_power is actually calculating the active power

GabrielKS added 30 commits July 17, 2024 13:25

Implement ComponentEntity

bb3dea6

This is the first commit in a major PowerAnalytics redesign.

Implement ListEntitySet

c13c836

Implement SubtypeEntitySet

c7a9729

Implement TopologyEntitySet

0bdbd2c

Implement FilterEntitySet

37423ae

Implement the very basics of Metrics

c352903

Refactor test setup into its own file

1fc29ea

Implement some input convenience functions

438661a

Minor create_problem_results_dict bugfix

53809a5

Add metrics wrapper functions and compute_all

b687bd1

Make entities only return available components

b0f7b49

More gracefully handle missing data (draft)

233f439

Add some Metric creation infrastructure and built-in Metrics

ea269c0

Add metadata columns feature

01e2041

Add aggregate_time function

1c61537

Add hcat_timed and calc_discharge_cycles

1570519

Add SystemTimedMetrics

7840602

Add some more built-in metrics

9fe0c33

Add ResultsTimelessMetrics

0bc3769

Add CustomTimedMetric (untested), more built-in metrics

04d87d8

Additional changes in this commit: - Generalize compute_all to take Components - Make create_problem_results_dict return SortedDict

Implement compose_metrics

afbd015

Give metrics control over aggregation functions

b8e54ef

Rather than setting `agg_fn`/`reduce_fn` when calling `compute` or `aggregate_time`, metrics now specify how they should be aggregated across Entities and across time.

Add more built-in metrics!

4228377

Add another way to call compute_all

396e95b

Fix calc_capacity_factor

e731c6e

Add author

36840da

I'm adding myself as the first author to reflect the fact that this redesigned version of the package is meant to be the main one in use going forward.

Implement compute_set

214537d

Resolve domain knowledge issues in builtin_metrics.jl

42ec449

GabrielKS added 18 commits July 17, 2024 15:17

Only compute on available subselectors, use PSY4 storage interface

be6d683

Prototype ComponentSelector grouping (tests not updated)

d09ee93

Prototype Metric grouping and callability (tests not updated)

ea9e418

Fix bug in calling repeat

2779e9a

Rename select_components to make_selector

50b9ecf

Add ability to disable test files like in other Sienna packages

d195534

Accommodate upstream refactors

036f57f

Wrap built-in selectors in their own submodule

0245c92

Rename input.jl to input_utils.jl

edf9e34

Cleanup: move helper functions to output_utils.jl, use @kwdef

ac386ee

Use kwargs... instead of start_time, len

ee503dc

Remove Metric description field in favor of docstrings

ba5dc48

Cleanup: move more code to utils files, run formatter

f27bfdd

Make minor adjustments, finish compute interface change

889aad9

Prototype Metrics submodule

5569eaf

Replace with_component_agg_fn, etc. with rebuild_metric

818ee94

Make tests pass, add necessary imports, adjust documentation

c43c5e4

Add tests for built-in metrics; fix bugs thereby uncovered

d0c7e1f

GabrielKS requested review from daniel-thom and jd-lara October 1, 2024 03:50

GabrielKS marked this pull request as ready for review October 1, 2024 04:03

GabrielKS added 2 commits October 4, 2024 11:43

Accommodate filterby -> scope_limiter rename

7978528

Bugfix: add fallback for when get_available is not defined

db54809

jd-lara reviewed Oct 8, 2024

View reviewed changes

daniel-thom requested changes Oct 9, 2024

View reviewed changes

Minor edits from PR comments 1

a512703

GabrielKS mentioned this pull request Oct 9, 2024

Slowly deprecate old PowerAnalytics #28

Open

GabrielKS added 3 commits October 30, 2024 00:08

Accommodate ComponentSelector API changes

b37a78d

Refactor generator_mapping.yaml, prototype new built-in selectors

e139fa9

Clean up generator_mapping.yaml selectors implementation, test

3173796

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Total PowerAnalytics Redesign: `ComponentSelector`s and `Metric`s #24

Total PowerAnalytics Redesign: `ComponentSelector`s and `Metric`s #24

GabrielKS commented Sep 30, 2024 •

edited

Loading

jd-lara Oct 8, 2024

GabrielKS Oct 8, 2024

daniel-thom Oct 8, 2024

daniel-thom Oct 8, 2024 •

edited

Loading

GabrielKS Oct 9, 2024

GabrielKS Oct 24, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

daniel-thom Oct 9, 2024

daniel-thom Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024

daniel-thom Oct 9, 2024

daniel-thom Oct 9, 2024

GabrielKS Oct 9, 2024 •

edited

Loading

		@@ -0,0 +1,121 @@
		# For now we are mostly just testing that all the metrics can be called without error,

Total PowerAnalytics Redesign: ComponentSelectors and Metrics #24

Are you sure you want to change the base?

Total PowerAnalytics Redesign: ComponentSelectors and Metrics #24

Conversation

GabrielKS commented Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-thom Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GabrielKS Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Total PowerAnalytics Redesign: `ComponentSelector`s and `Metric`s #24

Total PowerAnalytics Redesign: `ComponentSelector`s and `Metric`s #24

GabrielKS commented Sep 30, 2024 •

edited

Loading

daniel-thom Oct 8, 2024 •

edited

Loading

GabrielKS Oct 9, 2024 •

edited

Loading