Fix a MAgPIE run based on existing gdx files

[flohump]

For some applications it would be useful if MAgPIE could be fixed to the values of existing GDX files until a certain year.
Let's assume you have a run that is infeasible in 2050. Now you want to restart the run but keep the results until 2020 from the first run that was infeasible only in 2050 (based on magpie_y1995.gdx ... magpie_y2020.gdx). This makes sure that the results for 1995-2020 are identical to other model runs from the same scenario set. From 2025 onwards the model should use the gdx files from the first run for all years until 2045 as starting point (cfg$gms$s_use_gdx <- 2). This will push the solver to a slightly different trajectory from 2025 onwards and might lead to a feasible solution for 2050 and beyond.

Note 1: Re-running the model for the period 1995-2020 without gdx files (or cfg$gms$s_use_gdx <- 0) should yield the same result but of course needs much more time than just fixing all variables based on existing gdx files.
Note 2: Re-running the model for the period 1995-2020 with gdx files and cfg$gms$s_use_gdx <- 2 produces different results.

[orichters]

I discussed with @mishkos yesterday that this functionality would be also very useful for delayed transition scenarios in the NGFS project.

We have this functionality in REMIND, so it should be possible to copy that to MAgPIE.

Basically, the cozy function create_standard_fixings reads from the gdx file (always input_ref.gdx in the ./output subfolder), creating three files levs.gms (levels), fixings.gms (fixings), and margs.gms (marginals).

There is some zipping and unzipping involved for restarted runs to avoid doing this multiple times.

In loop.gms, we have a unique string cb20140305readinpositionforfinxingfiles (only valid with the typo!) which is changed into the gms files created, which will then be executed by GAMS.

Like that, we fix runs on older runs for all timesteps < cm_startyear.

But maybe there is a more elegant solution for that. I guess @christophbertram might provide more information.

[christophbertram]

Isn't this much easier to implement in MAgPIE, given its recursive solution approach (compared to the intertemporal optimization in REMIND)?
Without knowing what exactly the cfg$gms$s_use_gdx <- 2 is, in my view there should be settings 2020, 2025, ...20xx for this switch that result in a scenario in which the gdxs up to the chosen year are just used from before, and the first year to be solved anew (as the start of the recursive chain) is the next time step. I recall that you argued in the past that this would result in INFES, but I think that the underlying mechanisms are then to blame and need revision, as the real world functions like this (as of now (p0), everything prior to p0, and all decisions that have been made in the past were done with one singular set of assumptions about the then future (part of which has transpassed p < p0, different from expectations, and part of which is still to occur). And no matter what I expected what will happen tomorrow in the past, I'm free to completely readjust my expectation now based on all information I have now.

[tscheypidi]

quick update here: The situation is here quite different in MAgPIE compared to REMIND. The good thing is: In contrast to REMIND we do not need this feature to run delayed action scenarios due to the iterative nature of the model. Bad thing is that the fixing solution from REMIND is also not applicable to MAgPIE.

I did some more digging and revisited the GAMS manual. What GAMS provides is a Save and Restart which goes into the direction of what we would need but is not exactly the same. Hence, I wrote a mail to the GAMS developer team asking for a feature expansion in this direction. What we would need is the option to set restart points in the code, which write a save file every time they reach this point and would serve as an entry point when the model is started from such a savepoint.

[tscheypidi]

PR #512 partly adresses this issue by adding restart support to the model. It allows to interrupt and restart the model and still getting the exact same results like without interruption.
However, using this restart point in combination with a different scenario will most likely not yield the wanted behavior as the restart point contains all information about the run, including the whole parametrization of this run

[orichters]

Any progress on this issue (planned?), @tscheypidi / @pfuehrlich-pik?

[pfuehrlich-pik]

Not on my side, I'm still very busy with RESCUE/OptimESM.

[tscheypidi]

no, no plans

[orichters]

In the delayed transition scenarios of NGFS that are fixing in REMIND on a reference run until 2030, some of the deviations between the reference run and the delayed transition run are quite substantial:

   group                                   variables period            reldiff
 1 Agricultural Demand                             1 2025,2030         0.654 %
42 Fertilizer Use|Nitrogen                         1 2025,2030         2.03 %
54 Land Cover|Forest                               1 2025,2030         32 %
60 Price|Agriculture|Corn|Index                    1 2025,2030         13.4 %
73 Water Withdrawal|Irrigation                     1 2025,2030         20.1 %
74 Yield|Cereal                                    1 2025,2030         7.63 %
75 Yield|Oilcrops                                  1 2025,2030         10.8 %
76 Yield|Sugarcrops                                1 2025,2030         26.2 %

[orichters]

The problem:

• see here a summary of the problem for a delayed transition run: less +514g /p/tmp/oliverr/NGFS_v5/remind-2024-03-27/output/C_d_strain-rem-3/log.txt
• this is a 2°C run (rcp2p6) which follows the trajectory of a Current Policies run (rcp4p5) until 2030.

The REMIND way to achieve consistency (except for EDGE-Transport):

• specify path_gdx_ref to a different run or a fulldata.gdx in the config, and a cm_startyear which is the first free model year
• create_fixing_files, creating a gdxdump of the input_ref.gdx
• create_standard_fixings, basically three .gms file for .L, .M and .FX. There is a list of removals from this file, basically to be able to use older gdx files, avoiding specifying values for variables that are never declared. These files are then included in full.gms.
• see for example less /p/tmp/oliverr/debugging/fixings.gms, a file with a long list of such entries: vm_effGr.FX ('2030','CAZ','inco') = 1 ;
• in the REMIND code, one has to differentiate between ttot (all model timesteps) and t which is basically all ttot >= cm_startyear to avoid that fixings are overwritten. This is a bit of a delicate step because what to use when is not so easy to understand

[orichters]

@merfort added: Something else that might be relevant is the partial foresight that MAgPIE has regarding forestry (there is some price anticipation, right?). If I am not mistaken, here changes in the settings may also affect the historical period (or rather the period until the REMIND start year), even though all inputs from REMIND are the same.

[merfort]

Thanks Olli! For RESCUE there are, unfortunately, quite some differences between the reference NPi run (C_SSP2EU-NPi; red) and a delayed policy run (C_RESCUE-dir-v4p0-EocBudg500-OAE_off, blue), in which in REMIND everything is fixed to NPi for all years < 2035. Differences in Emissions|CO2|Land|+|Land-use Change are in particularly large.
The largest difference seems to be due to the NPi/NDC switch, since the NDC run (C_SSP2EU-NDC) is much closer to the delayed policy run. However, for the NDC run we have no fixing to the REMIND NPi in 2025 and 2030 (cm_startyear = 2025), so the remaining differences might also come from the differences between the respective REMIND runs.

title	magpie_scen	cm_startyear (REMIND)
C_SSP2EU-NPi	SSP2|NPI|nocc_hist|rcp4p5|ForestryExo	2025
C_SSP2EU-NDC	SSP2|NDC|nocc_hist|rcp4p5|ForestryExo	2025
C_RESCUE-dir-v4p0-EocBudg500-OAE_off	SSP2|NDC|nocc_hist|rcp1p9|ForestryExo	2035

Simply overwriting all MAgPIE reporting variables in 2025 and 2030 with the one from the NPi run (which is what I do so far for the emissions reporting), leads to quite a jump between 2030 and 2035. This is particularly problematic, since for RESCUE we are also reporting the land-use data (e.g. land-cover change) and so far we use all data from one the delayed policy run (no overwriting). I guess, that overwriting land-use data on a cluster level is anyways not viable, since the jumps on this lower resolution on the cluster level will be even larger. @tscheypidi @pfuehrlich-pik what are your thoughts on that?

[merfort]

For me, there are three candidates that may lead to differences between the runs

1. Switchting MAgPIE setting from NPi to NDC in the years 2025 and 2030
2. Switchting MAgPIE setting from rcp4p5 to rcp1p9 in the years 2025 and 2030 (should acutally not have an effect, with nocc_hist being activated, though)
3. The (limited) foresight of MAgPIE regarding timber plantations. This is only a wild guess, though, I don't know how the anticipation mechanism regarding new plantations works in MAgPIE. @flohump, do you know if (a) the differing (future) information that MAgPIE gets from REMIND (i.e. different bioenergy demands and GHG prices) between NPi and delayed policy run has an influence on how MAgPIE chooses to set up timber plantations and (b) that this would be even applicable in the ForestryExo case?

[mishkos]

Thanks Leon!

1. The NPI and NDC are consistent in MAgPIE, i.e. NPI last until 2020 when the NDCs start. NDCs end in 2030, so switching to different settings in later timesteps shouldn't make a difference between the runs.
2. Correct, for nocc_hist there shouldn't be any effect. We will see for the cases with cc how can the inputs in MAgPIE be adjusted.
3. Not sure here.

[merfort]

Hi all,
I did some experiments in order to isolate the effects that lead to differences between the runs that should - from a scenario narrative perspective - be identical in early periods.
For my case - the RESCUE scenarios - I want to have a set of delayed policy runs that start to differ from an NPi run only after 2030, i.e. with the first "free" year being 2035.
In the following I try to summarize a bit, what I found.

As I see it, there are two main issues.

1. Some MAgPIE settings differ between the run that REMIND is fixed to (here: NPi) and the delayed policy run. This can be the settings defined in magpie_scen, e.g.,

   • NPI -> NDC
   • rcp4p5 -> rcp1p9

   or the value defined in no_ghgprices_land_until, i.e., the first year, in which GHG pricing in MAgPIE is activated

   • y2150 -> y2030

   The latter is relatively easy to solve, here simply the value needs to be set according to the start year in REMIND (in my specific case it is already good, since the REMIND start year is 2035, so no_ghgprices_land_until <- 2030 is correct, since this mutes GHG prices in all years before and in 2030).
   For differences in between values in magpie_scen, the solution would be to create a mixed scenario, that features a transition from one scenario to the other, e.g. a scenario NPI_until_2030_NDC_from_2035 that has NPI characteristics until 2030 and NDC characteristics starting in 2035. Here, of course, we first need to ask ourselves, what the story behind such a mix is (do the exact same NDCs simply start with a delay of 10 years? Is that reasonable?).
   Such a mixed scenario would need to be created for all magpie_scen entries that differ between the scenarios, i.e. in particular also for the RCP switch, when cc is activated (something like rcp4p5_until_2030_rcp1p9_from_2035). Strangely, though, I also found differences between scenarios that deviate only between RCP (rcp4p5 -> rcp1p9), even though nocc_hist is activated. Do rcp4p5 and rcp1p9 have already a different history, i.e., in years <=2020? With nocc_hist I would have expected that two runs are identical.

2. Some of the MAgPIE output variables, which are used in REMIND, are modified with a low-pass filter. In particular Emissions|CO2|Land|+|Land-use Change, which is one of the most important outputs from MAgPIE to REMIND, as it enters the CO2 budget equation, can - due to the application of the low-pass filter - differ quite substantially between scenarios in years, where it should actually be identical. For example, in my test runs (see Fig. 1) this variable differs between three OS_... (OS = overshoot = delayed policy) scenarios and the NPi run. The raw variable Emissions|CO2|Land RAW|+|Land-use Change RAW (i.e., without low-pass filter, Fig. 2), is identical, though.
   
   
   For this problem, I don't see a straight forward solution. Starting low-pass filtering only after the REMIND start year (i.e. leaving all values before untouched) would of course solve the problem, but then we would have to deal with the LUC emission spikes before.