Update & improve reporting

.github/workflows/pytest.yaml

@@ -28,8 +28,7 @@ jobs:
         - ubuntu-latest
         - windows-latest
         python-version:
-        - "3.7"  # Earliest version supported by message_ix
-        - "3.8"
+        - "3.8"   # Earliest version supported by message_ix
         - "3.9"
         - "3.10"
         - "3.11"  # Latest version supported by message_ix

RELEASE_NOTES.rst

@@ -4,6 +4,9 @@ Next release
 All changes
 -----------
 
+- Drop support for Python 3.7, which `reached end-of-life on 2023-06-27 <https://peps.python.org/pep-0537/#lifespan>`__ (:pull:`738`).
+  :mod:`message_ix` now requires Python version 3.8 or greater.
+- New reporting operator :func:`.model_periods` and automatic keys ``y::model`` and ``y0`` (:pull:`738`).
 - Improve readability of LaTeX equations in docs (:pull:`721`).
 - Bugfix: :meth:`.Scenario.add_macro` would not correctly handle configuration that mapped a MESSAGE (commodity, level) to MACRO sector when the commodity and sector names were different (:pull:`719`).
 - Expand :doc:`macro` documentation, particularly code documentation (:issue:`315`, :pull:`719`).

doc/conf.py

@@ -73,6 +73,8 @@
 
 .. role:: underline
 
+.. role:: py(code)
+   :language: python
 """  # noqa: E501
 
 # -- Options for HTML output ----------------------------------------------

doc/reporting.rst

@@ -35,12 +35,11 @@ Each layer of this “stack” builds on the features in the level below:
      - —
 
 These features are accessible through :class:`.Reporter`, which can produce multiple **reports** from one or more Scenarios.
-A report is identified by a **key** (usually a string), and may…
+A report and the quantities that enter it is identified by a **key**, and may…
 
 - perform arbitrarily complex calculations while intelligently handling units;
-- read and make use of data that is ‘exogenous’ to (not included in) a
-  Scenario;
-- produce output as Python or R objects (in code), or to files or databases;
+- read and make use of data that is ‘exogenous’ to (not included in) a Scenario;
+- produce output as Python or R objects (in code), or write to files or databases;
 - calculate only a requested subset of quantities; and
 - much, much more!
 
@@ -62,7 +61,7 @@ In :mod:`message_ix.reporting`:
 - For example, the |MESSAGEix| parameter ``resource_cost``, defined with the dimensions (node `n`, commodity `c`, grade `g`, year `y`) is identified by the key ``resource_cost:n-c-g-y``.
   When summed across the grade/`g` dimension, it has dimensions `n`, `c`, `y` and is identified by the key ``resource_cost:n-c-y``.
 - :meth:`.Reporter.from_scenario` automatically sets up keys and tasks (such as ``resource_cost:n-c-g-y``) that simply retrieve raw/unprocessed data from a :class:`~message_ix.Scenario` and return it as a :any:`genno.Quantity`.
-- Computations are defined as functions in modules including:
+- Operators are defined as functions in modules including:
   :mod:`message_ix.reporting.computations`,
   :mod:`ixmp.reporting.computations`, and
   :mod:`genno.computations`.
@@ -114,7 +113,7 @@ The method :meth:`.Reporter.add` can be used to add *arbitrary* Python code that
     rep.get("custom")
 
 In this example, the function ``my_custom_report()`` *could* run to thousands of lines; read to and write from multiple files; invoke other programs or Python scripts; etc.
-In order to take advantage of the performance-optimizing features of the Reporter, such calculations can instead be composed from atomic (i.e. small, indivisible) computations.
+In order to take advantage of the performance-optimizing features of the Reporter, such calculations can instead be composed from atomic (i.e. small, indivisible) operators or functions.
 See the :mod:`genno` documentation for more.
 
 API reference
@@ -146,23 +145,26 @@ Their documentation is repeated below for convenience.
    configure
 
 
-:meth:`ixmp.Reporter.from_scenario <ixmp.reporting.Reporter.from_scenario>` automatically adds keys based on the contents of the :class:`.Scenario` argument.
+:meth:`ixmp.Reporter.from_scenario <ixmp.reporting.Reporter.from_scenario>` automatically adds keys based on the contents of the :class:`.Scenario` argument;
+that is, every :mod:`ixmp` set, parameter, variable, and equation available in the Scenario.
 :meth:`message_ix.Reporter.from_scenario <.Reporter.from_scenario>` extends this to add additional keys for derived quantities specific to the MESSAGEix model framework.
 These include:
 
-- ``out`` - ``output`` × ``ACT``; that is, the product of ``output`` (output efficiency) and ``ACT`` (activity)
+.. tip:: Use :meth:`~.Computer.full_key` to retrieve the full-dimensionality :class:`Key` for any of these quantities.
+
+- ``out``          = ``output`` × ``ACT``; that is, the product of ``output`` (output efficiency) and ``ACT`` (activity)
 - ``out_hist``     = ``output`` × ``ref_activity`` (historical reference activity)
 - ``in``           = ``input`` × ``ACT``
 - ``in_hist``      = ``input`` × ``ref_activity``
 - ``emi``          = ``emission_factor`` × ``ACT``
 - ``emi_hist``     = ``emission_factor`` × ``ref_activity``
 - ``inv``          = ``inv_cost`` × ``CAP_NEW``
 - ``inv_hist``     = ``inv_cost`` × ``ref_new_capacity``
-- ``fom``          = ``fix_cost`` × ``CAP``
+- ``fom``          = ``fix_cost`` × ``CAP``; the name refers to "Fixed Operation and Maintenance costs"
 - ``fom_hist``     = ``fix_cost`` × ``ref_capacity``
-- ``vom``          = ``var_cost`` × ``ACT``
+- ``vom``          = ``var_cost`` × ``ACT``; "Variable Operation and Maintenance costs"
 - ``vom_hist``     = ``var_cost`` × ``ref_activity``
-- ``tom``          = ``fom`` + ``vom``
+- ``tom``          = ``fom`` + ``vom``; "Total Operation and Maintenance costs"
 - ``land_out``     = ``land_output`` × ``LAND``
 - ``land_use_qty`` = ``land_use`` × ``LAND``
 - ``land_emi``     = ``land_emission`` × ``LAND``
@@ -174,13 +176,14 @@ These include:
 - ``addon potential`` = ``addon up`` × ``addon ACT``, the maximum potential activity by add-on technology.
 - ``price emission``, the model variable ``PRICE_EMISSION`` broadcast across emission species (`e`) *and* technologies (`t`) rather than types (`type_emission`, `type_tec`).
 
-.. tip:: Use :meth:`~.Computer.full_key` to retrieve the full-dimensionality :class:`Key` for any of these quantities.
-
 Other added keys include:
 
 - :mod:`message_ix` adds the standard short symbols for |MESSAGEix| dimensions (sets) based on :data:`DIMS`.
-  Each of these is also available in a Reporter: for example ``rep.get("n")`` returns a list with the elements of the |MESSAGEix| set named "node".
-  These keys can be used as input
+  Each of these is also available in a Reporter: for example :py:`rep.get("n")` returns a list with the elements of the |MESSAGEix| set named "node";
+  :py:`rep.get("t")` returns the elements of the set "technology", and so on.
+  These keys can be used as input to other computations.
+- ``y0`` = the ``firstmodelyear`` or :math:`y_0` (:class:`int`).
+- ``y::model`` = only the periods in the `year` set (``y``) that are equal to or greater than ``y0``.
 
 .. _default-reports:
 
@@ -213,17 +216,13 @@ These automatic contents are prepared using:
 
    .. autosummary::
       add
-      add_file
-      add_product
       add_queue
       add_single
-      aggregate
       apply
       check_keys
       configure
-      convert_pyam
       describe
-      disaggregate
+      eval
       finalize
       from_scenario
       full_key
@@ -234,6 +233,13 @@ These automatic contents are prepared using:
       visualize
       write
 
+   .. autosummary::
+      add_file
+      add_product
+      aggregate
+      convert_pyam
+      disaggregate
+
 
 .. autodata:: DERIVED
 .. autodata:: DIMS
@@ -247,29 +253,31 @@ These automatic contents are prepared using:
    :members: ComputationError, Key, KeyExistsError, MissingKeyError, Quantity, configure
 
 
-Computations
-------------
+Operators
+---------
 
 .. automodule:: message_ix.reporting.computations
    :members:
 
-   :mod:`message_ix.reporting` provides a small number of computations.
+   :mod:`message_ix.reporting` provides a small number of operators.
    Two of these (:func:`.plot_cumulative` and :func:`.stacked_bar`) are currently only used in the tutorials to produce simple plots; for more flexible plotting, :mod:`genno.compat.plotnine` is recommended instead.
 
    .. autosummary::
       as_message_df
+      model_periods
       plot_cumulative
       stacked_bar
 
-   Other computations are provided by :mod:`ixmp.reporting`:
+   Other operators are provided by :mod:`ixmp.reporting`:
 
    .. autosummary::
       ~ixmp.reporting.computations.data_for_quantity
       ~ixmp.reporting.computations.map_as_qty
       ~ixmp.reporting.computations.store_ts
       ~ixmp.reporting.computations.update_scenario
 
-   …and by :mod:`genno.computation` and its compatibility modules. See the package documentation for details.
+   …and by :mod:`genno.computations` and its compatibility modules.
+   See the package documentation for details.
 
    .. autosummary::
       ~genno.compat.plotnine.Plot
@@ -280,21 +288,26 @@ Computations
       ~genno.computations.broadcast_map
       ~genno.computations.combine
       ~genno.computations.concat
-      ~genno.computations.disaggregate_shares
       ~genno.computations.div
+      ~genno.computations.drop_vars
       ~genno.computations.group_sum
       ~genno.computations.index_to
       ~genno.computations.interpolate
       ~genno.computations.load_file
       ~genno.computations.mul
       ~genno.computations.pow
-      ~genno.computations.ratio
       ~genno.computations.relabel
       ~genno.computations.rename_dims
+      ~genno.computations.round
       ~genno.computations.select
+      ~genno.computations.sub
       ~genno.computations.sum
       ~genno.computations.write_report
 
+   .. autosummary::
+      ~genno.computations.disaggregate_shares
+      ~genno.computations.product
+      ~genno.computations.ratio
 
 Utilities
 ---------

message_ix/reporting/__init__.py

@@ -1,5 +1,6 @@
 import logging
 from functools import lru_cache, partial
+from operator import itemgetter
 from typing import Mapping, Sequence, Tuple, Union, cast
 
 import genno
@@ -92,6 +93,8 @@
 DERIVED = [
     # Each entry is ('full key', (computation tuple,)). Full keys are not inferred and
     # must be given explicitly.
+    ("y::model", ("model_periods", "y", "cat_year")),
+    ("y0", (itemgetter(0), "y::model")),
     ("tom:nl-t-yv-ya", (genno.computations.add, "fom:nl-t-yv-ya", "vom:nl-t-yv-ya")),
     # Broadcast from type_addon to technology_addon
     (
@@ -196,7 +199,7 @@ def put(*args, **kwargs):
         # Callback function to further collapse other columns into IAMC columns
         collapse_cb = partial(collapse_message_cols, **collapse_kw)
 
-        put("convert_pyam", qty, "pyam", rename=rename, collapse=collapse_cb)
+        put("as_pyam", qty, "pyam", rename=rename, collapse=collapse_cb)
 
     # Standard reports
     for group, pyam_keys in REPORTS.items():

message_ix/reporting/computations.py

@@ -1,4 +1,4 @@
-from typing import Mapping, Union
+from typing import List, Mapping, Union
 
 import pandas as pd
 from ixmp.reporting import Quantity
@@ -7,6 +7,7 @@
 
 __all__ = [
     "as_message_df",
+    "model_periods",
     "plot_cumulative",
     "stacked_bar",
 ]
@@ -57,6 +58,17 @@
     return {name: df} if wrap else df
 
 
+def model_periods(y: List[int], cat_year: pd.DataFrame) -> List[int]:
+    """Return the elements of `y` beyond the firstmodelyear of `cat_year`."""
+    return list(
+        filter(
+            lambda year: cat_year.query("type_year == 'firstmodelyear'")["year"].item()
+            <= year,
+            y,
+        )
+    )
+
+
 def plot_cumulative(x, y, labels):
     """Plot a supply curve.
 

message_ix/tests/test_reporting.py

@@ -5,6 +5,7 @@
 
 import pandas as pd
 import pyam
+import pytest
 import xarray as xr
 from genno import Quantity
 from genno.testing import assert_qty_equal
@@ -73,7 +74,7 @@ def test_reporter_from_scenario(message_test_mp):
 
     # message_ix.Reporter pre-populated with additional, derived quantities
     # This is the same value as in test_tutorials.py
-    assert len(rep.graph) == 13074
+    assert len(rep.graph) == 13076
 
     # Derived quantities have expected dimensions
     vom_key = rep.full_key("vom")
@@ -145,7 +146,7 @@ def test_reporter_from_westeros(test_mp):
     assert_allclose(obs, exp)
 
 
-def test_reporter_convert_pyam(caplog, tmp_path, dantzig_reporter):
+def test_reporter_as_pyam(caplog, tmp_path, dantzig_reporter):
     caplog.set_level(logging.INFO)
 
     rep = dantzig_reporter
@@ -181,7 +182,9 @@ def add_tm(df, name="Activity"):
         return df.drop(["t", "m"], axis=1)
 
     # Use the convenience function to add the node
-    key2 = rep.convert_pyam(ACT, "iamc", rename=rename, collapse=add_tm)
+    with pytest.warns(DeprecationWarning):
+        key2 = rep.convert_pyam(ACT, "iamc", rename=rename, collapse=add_tm)
+    key2 = rep.add("as_pyam", ACT, "iamc", rename=rename, collapse=add_tm)
 
     # Keys of added node(s) are returned
     assert ACT.name + "::iamc" == key2
@@ -223,8 +226,16 @@ def add_tm(df, name="Activity"):
 
     # Use a name map to replace variable names
     replacements = {re.escape("Activity|canning_plant|production"): "Foo"}
-    key3 = rep.convert_pyam(
-        ACT, rename=rename, replace=dict(variable=replacements), collapse=add_tm
+    with pytest.warns(DeprecationWarning):
+        key3 = rep.convert_pyam(
+            ACT, rename=rename, replace=dict(variable=replacements), collapse=add_tm
+        )
+    key3 = rep.add(
+        "as_pyam",
+        ACT,
+        rename=rename,
+        replace=dict(variable=replacements),
+        collapse=add_tm,
     )
     df3 = rep.get(key3).as_pandas()
 
@@ -236,16 +247,24 @@ def add_tm(df, name="Activity"):
 
     # Now convert variable cost
     cb = partial(add_tm, name="Variable cost")
-    key4 = rep.convert_pyam("var_cost", rename=rename, collapse=cb)
+    with pytest.warns(DeprecationWarning):
+        key4 = rep.convert_pyam("var_cost", rename=rename, collapse=cb)
+    key4 = rep.add("as_pyam", "var_cost", rename=rename, collapse=cb)
+
     df4 = rep.get(key4).as_pandas().drop(["model", "scenario"], axis=1)
 
     # Results have the expected units
     assert all(df4["unit"] == "USD / case")
 
     # Also change units
-    key5 = rep.convert_pyam(
-        "var_cost", rename=rename, collapse=cb, unit="centiUSD / case"
+    with pytest.warns(DeprecationWarning):
+        key5 = rep.convert_pyam(
+            "var_cost", rename=rename, collapse=cb, unit="centiUSD / case"
+        )
+    key5 = rep.add(
+        "as_pyam", "var_cost", rename=rename, collapse=cb, unit="centiUSD / case"
     )
+
     df5 = rep.get(key5).as_pandas().drop(["model", "scenario"], axis=1)
 
     # Results have the expected units

message_ix/tests/test_tutorials.py

@@ -22,10 +22,6 @@
         condition=GHA and sys.platform == "darwin",
         reason="Always fails on GitHub Action macOS runners",
     ),
-    pytest.mark.xfail(  # 2
-        condition=sys.version_info.minor >= 11,
-        reason="pyam-iamc does not support Python 3.11; see IAMconsortium/pyam#744",
-    ),
 ]
 
 # Affects all tests in the file
@@ -87,7 +83,7 @@ def _t(group: Union[str, None], basename: str, *, check=None, marks=None):
     _t("w0", f"{W}_historical_new_capacity"),
     _t("w0", f"{W}_multinode"),
     # NB this is the same value as in test_reporter()
-    _t(None, f"{W}_report", check=[("len-rep-graph", 13074)], marks=MARK[2]),
+    _t(None, f"{W}_report", check=[("len-rep-graph", 13076)]),
     _t("at0", "austria", check=[("solve-objective-value", 206321.90625)]),
     _t("at0", "austria_single_policy", check=[("solve-objective-value", 205310.34375)]),
     _t("at0", "austria_multiple_policies"),

pyproject.toml

@@ -20,7 +20,6 @@ classifiers = [
   "Operating System :: OS Independent",
   "Programming Language :: Python",
   "Programming Language :: Python :: 3",
-  "Programming Language :: Python :: 3.7",
   "Programming Language :: Python :: 3.8",
   "Programming Language :: Python :: 3.9",
   "Programming Language :: Python :: 3.10",
@@ -29,11 +28,11 @@ classifiers = [
   "Topic :: Scientific/Engineering",
   "Topic :: Scientific/Engineering :: Information Analysis",
 ]
+requires-python = ">=3.8"
 dependencies = [
   "click",
-  'importlib_metadata; python_version < "3.8"',
   "ixmp >= 3.7.0",
-  "genno[pyam]",
+  "genno[pyam] >= 1.18.1",
   "numpy",
   "pandas >= 1.2",
   "PyYAML",