Running train_nn.py results in KeyError 'no index found for coordinate level'

[ciaron]

I'm trying to follow the instructions to reproduce this run:
python -m src.train_nn -c src/nn_configs/fccnn_3d.yml

with commit 11cfbff (latest as of today) and the current dataset (downloaded today)

After a few seconds though, I get the following exception:

Traceback (most recent call last):
  File "/p/tmp/linstead/envs/weatherbench/lib/python3.9/site-packages/xarray/core/indexing.py", line 69, in group_indexers_by_index
    index = xindexes[key]
KeyError: 'level'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/p/tmp/linstead/envs/weatherbench/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/p/tmp/linstead/envs/weatherbench/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/p/projects/its/benchmarks_2020/application/WeatherBench/src/train_nn.py", line 299, in <module>
    main(
  File "/p/projects/its/benchmarks_2020/application/WeatherBench/src/train_nn.py", line 237, in main
    dg_train = DataGenerator(ds_train, dic, lead_time, batch_size=batch_size)
  File "/p/projects/its/benchmarks_2020/application/WeatherBench/src/train_nn.py", line 44, in __init__
    data.append(ds[var].sel(level=levels))
  File "/p/tmp/linstead/envs/weatherbench/lib/python3.9/site-packages/xarray/core/dataarray.py", line 1315, in sel
    ds = self._to_temp_dataset().sel(
  File "/p/tmp/linstead/envs/weatherbench/lib/python3.9/site-packages/xarray/core/dataset.py", line 2474, in sel
    pos_indexers, new_indexes = remap_label_indexers(
  File "/p/tmp/linstead/envs/weatherbench/lib/python3.9/site-packages/xarray/core/coordinates.py", line 421, in remap_label_indexers
    pos_indexers, new_indexes = indexing.remap_label_indexers(
  File "/p/tmp/linstead/envs/weatherbench/lib/python3.9/site-packages/xarray/core/indexing.py", line 106, in remap_label_indexers
    indexes, grouped_indexers = group_indexers_by_index(
  File "/p/tmp/linstead/envs/weatherbench/lib/python3.9/site-packages/xarray/core/indexing.py", line 80, in group_indexers_by_index
    raise KeyError(f"no index found for coordinate {key}")
KeyError: 'no index found for coordinate level'

[raspstephan]

Hi, I just wanted to quickly reply. Unfortunately, I didn't get around to looking at this this week and will be on vacation next week. Hard for me to say what the problem is from afar. I would check the dataset ds and seen what's up with the level coordinate. Let me know if the problem is still current and I will take a look after next week.

[ciaron]

Hi Stephan, thanks for your reply. The level coordinate seems fine to me:

I can reproduce the error by stepping through the relevant code in a notebook: I get to line 42 in src/train_nn.py (DataGenerator) for the call from line 237 (dg_train). At line 45 ValueError is handled, but I'm not sure if KeyError should be too? (Indeed, adding KeyError to the exception handler gets the run going, but is this valid?)

README mentions that the data format has changed, could this be the problem?

Here is my config file for the run:

# NN config file
datadir: /p/projects/its/benchmarks_2020/data/WeatherBench/5.625deg
model_save_fn: /p/projects/its/benchmarks_2020/application/WeatherBench/baselines/saved_models/cnn_3d.h5
pred_save_fn: /p/projects/its/benchmarks_2020/application/WeatherBench/baselines/cnn_3d.nc
vars: [z, t]
filters: [64, 64, 64, 64, 2]
kernels: [5, 5, 5, 5, 5]
lead_time: 72

[raspstephan]

Hmm, I think think the issue is that while level is a coordinate it is not actually a dimension of the data variables. z and t only have time, lat and lon as dimensions, so selecting a level does not work. A quick fix should be to just remove .sel(level=levels)

[ciaron]

Thanks! I tried that, but level appears in other places: e.g.

File "/p/projects/its/benchmarks_2020/application/WeatherBench/src/score.py", line 22, in load_test_data
    ds = ds.sel(level=500 if var == 'z' else 850)
.
.
.
KeyError: 'no index found for coordinate level'

Anyway, enjoy your vacation, this can certainly wait until after that!

[raspstephan]

Does it work to just remove the .sel from all of these? I think in the version of the data I used for the notebooks, I had a level coordinate with one level, then got rid of it for easier use later but without changing the notebooks.

…

On 6. Aug 2021, at 22:34, Ciaron Linstead ***@***.***> wrote: Thanks! I tried that, but level appears in other places: e.g. File "/p/projects/its/benchmarks_2020/application/WeatherBench/src/score.py", line 22, in load_test_data ds = ds.sel(level=500 if var == 'z' else 850) .. . .. KeyError: 'no index found for coordinate level' Anyway, enjoy your vacation, this can certainly wait until after that! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVIM34A2I3MVWUYKDPQBZTT3RBNXANCNFSM5BIVAYZQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>.

[ciaron]

It seems to have worked, yes. I have a run in progress now.