[Bug]: Failure to generate global time series plot if not starting from year 1 #546

wlin7 · 2024-02-04T16:36:04Z

What happened?

The following error occur when trying to generate global time series plot that does not start from year 1 of a simulation (v3.LR.piControl-spinup). This happens after e3sm_diags and mpas-analysis for the covered period have been completed properly. Same errors also occur for other subperiods, such as global_time_series_0901-1100, global_time_series_1001-1400, regardless of the length of the time series. Does it mean the plotting codes are currently intended only for time series from the beginning of a simulation? Is it easy to make it work for any selected sub-period?

Update time series figures
/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/atm/glb/ts/monthly/10yr/glb.xml
RESTOM
RESSURF
TREFHT
FSNTOA
FLUT
PRECC
PRECL
QFLX
Traceback (most recent call last):
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 651, in <module>
    run_by_region(sys.argv)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 647, in run_by_region
    run(parameters, rgn)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 588, in run
    exp["annual"]["ohc"] = ts.globalAnnual("ohc")
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/readTS.py", line 65, in globalAnnual
    v = cdutil.YEAR(v)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 1416, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 477, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 751, in monthBasedSlicer
TypeError: 'NoneType' object is not subscriptable

Examples can be found under /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts

What machine were you running on?

chrysalis

Environment

v2.3.1rc1

What command did you run?

zppy -c /home/ac.wlin/E3SM/utils/post.20231209.v3.LR.piControl-spinup.chrysalis.cfg.1a

Copy your cfg file

[default]
input = /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis
output = /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis
case = 20231209.v3.LR.piControl-spinup.chrysalis
www = /lcrc/group/e3sm/public_html/diagnostic_output/ac.wlin/E3SMv3
partition = compute
environment_commands = "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh"
campaign = "water_cycle"

[climo]
active = True
years = "0001:1450:50", "0001:1400:100"
walltime = "1:00:00"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  frequency = "monthly"

  [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h3"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "PRECT"
  frequency = "diurnal_8xdaily"

[ts]
active = True
years = "0001:1450:10",
walltime = "00:50:00"

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,ICEFRAC,LANDFRAC,OCNFRAC,PS"

  [[ atm_monthly_glb ]]
  input_subdir = "archive/atm/hist"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb"

  [[ land_monthly ]]
  input_subdir = "archive/lnd/hist"
  input_files = "elm.h0"
  mapping_file = ""
  frequency = "monthly"
  vars = "FSH,RH2M"
  extra_vars = "landfrac"

  [[ rof_monthly ]]
  input_subdir = "archive/rof/hist"
  input_files = "mosart.h0"
  mapping_file = ""
  frequency = "monthly"
  vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"
  extra_vars = 'areatotal2'

[e3sm_diags]
active = True
walltime = "4:00:00"
years = "0001:1450:50", "0001:1400:100"
ts_num_years = 10
ref_start_yr = 1979
ref_final_yr = 2016
multiprocessing = True
num_workers = 8

  [[ atm_monthly_180x360_aave ]]
  short_name = '20231209.v3.LR.piControl-spinup'
  grid = '180x360_aave'
  reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
  obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
  dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
  climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
  climo_diurnal_frequency = "diurnal_8xdaily"
  sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","qbo","diurnal_cycle","zonal_mean_2d_stratosphere","aerosol_budget"
[mpas_analysis]
active = True
walltime = "6:00:00"
parallelTaskCount = 6
anomalyRefYear = 101
ts_years = "101-1400", "101-1450",
enso_years = "101-1400","101-1450",
climo_years = "1351-1400","1401-1450",
mesh = "IcoswISC30E3r5"
shortTermArchive = True

[global_time_series]
active = True
years = "101-1400"
ts_num_years = 10
figstr= "20231209.v3.LR.piControl-spinup"
moc_file = "mocTimeSeries_0101-1400.nc"
experiment_name = "20231209.v3.LR.piControl-spinup"
ts_years = "101-1400",
climo_years = "1351-1400",

What jobs are failing?

global_time_series

What stack trace are you encountering?

Traceback (most recent call last):
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 651, in <module>
    run_by_region(sys.argv)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 647, in run_by_region
    run(parameters, rgn)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/coupled_global.py", line 588, in run
    exp["annual"]["ohc"] = ts.globalAnnual("ohc")
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_series_0101-1400_dir/readTS.py", line 65, in globalAnnual
    v = cdutil.YEAR(v)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 1416, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 477, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdutil-8.2.1-py3.11.egg/cdutil/times.py", line 751, in monthBasedSlicer
TypeError: 'NoneType' object is not subscriptable

The text was updated successfully, but these errors were encountered:

forsyth2 · 2024-03-01T22:16:43Z

Does it mean the plotting codes are currently intended only for time series from the beginning of a simulation? Is it easy to make it work for any selected sub-period?

No, that works. The cfg I use for testing starts at 1850 (https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/test_complete_run_chrysalis.cfg). But I wasn't sure if that was the simulation start date. So, I ran a cfg starting at 1860 instead and it generated plots ok: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/zppy_test_debug_www/546-change-start-yr/v2.LR.historical_0201/global_time_series/global_time_series_1860-1870_results/v2_historical_0201_glb.pdf

cfg:

[default]
case = v2.LR.historical_0201
constraint = ""
dry_run = "False"
environment_commands = ""
input = "/lcrc/group/e3sm/ac.forsyth2//E3SMv2/v2.LR.historical_0201"
input_subdir = archive/atm/hist
mapping_file = "map_ne30pg2_to_cmip6_180x360_aave.20200201.nc"
# To run this test, edit `output` and `www` in this file, along with `actual_images_dir` in test_complete_run.py
output = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_debug_output/546-change-start-yr/v2.LR.historical_0201"
partition = "debug"
qos = "regular"
www = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_test_debug_www/546-change-start-yr"

[ts]
active = True
e3sm_to_cmip_environment_commands = "source /home/ac.forsyth2/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_to_cmip_20240229"
walltime = "00:30:00"
years = "1860:1864:2",

  [[ atm_monthly_glb ]]
  # Note global average won't work for 3D variables.
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  mapping_file = "glb"
  years = "1860:1870:5",

  [[ land_monthly_glb ]]
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  mapping_file = "glb"


[mpas_analysis]
active = True
anomalyRefYear = 1860
climo_years ="1860-1864", "1865-1870",
enso_years = "1860-1864", "1865-1870",
mesh = "EC30to60E2r2"
parallelTaskCount = 6
partition = "compute"
qos = "regular"
ts_years = "1860-1864", "1860-1870",
walltime = "00:30:00"

[global_time_series]
active = True
climo_years ="1860-1864", "1865-1870",
experiment_name = "v2.LR.historical_0201"
figstr = "v2_historical_0201"
moc_file=mocTimeSeries_1860-1870.nc
ts_num_years = 5
ts_years = "1860-1864", "1860-1870",
walltime = "00:30:00"
years = "1860-1870",

forsyth2 · 2024-03-01T22:50:19Z

I'm having a hard time figuring out what's causing this error. TypeError: 'NoneType' object is not subscriptable is not particularly helpful.

Also worth noting is that the error is coming from cdutil which needs to be removed/replaced anyway as part of the CDAT migration (#346/#519)

forsyth2 · 2024-03-02T02:54:57Z

So something is happening in this block, specifically at exp["annual"]["ohc"] = ts.globalAnnual("ohc") :

zppy/zppy/templates/coupled_global.py

Lines 585 to 590 in 4db3e24

    
           # Optionally read ohc 
        
           if exp["ocean"] is not None: 
        
               ts = TS(exp["ocean"]) 
        
               exp["annual"]["ohc"] = ts.globalAnnual("ohc") 
        
               # annomalies with respect to first year 
        
               exp["annual"]["ohc"][:] = exp["annual"]["ohc"][:] - exp["annual"]["ohc"][0]

But I'm not sure why that's failing.

$ cd /lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts
$ grep -v "OK" *status
global_time_series_0051-1400.status:WAITING 465551
global_time_series_0101-1400.status:ERROR (5)
global_time_series_0101-1450.status:WAITING 465658
global_time_series_0901-1100.status:ERROR (5)
global_time_series_0901-1150.status:ERROR (3)
global_time_series_1001-1150.status:ERROR (3)
global_time_series_1001-1200.status:ERROR (5)
global_time_series_1001-1300.status:ERROR (5)
global_time_series_1001-1350.status:ERROR (5)
global_time_series_1001-1400.status:ERROR (5)
global_time_series_1001-1500.status:ERROR (5)
global_time_series_1001-1550.status:ERROR (5)
global_time_series_1001-1700.status:ERROR (5)
global_time_series_1001-1900.status:ERROR (5)
global_time_series_1001-1950.status:ERROR (5)
global_time_series_1001-2000.status:ERROR (5)
global_time_series_1501-1800.status:ERROR (5)
global_time_series_1501-1900.status:ERROR (5)
global_time_series_1501-1950.status:ERROR (3)
global_time_series_1501-2000.status:ERROR (5)
mpas_analysis_ts_0001-1400_climo_1351-1400.status:ERROR (1)
mpas_analysis_ts_0001-1600_climo_1551-1600.status:ERROR (1)
mpas_analysis_ts_0051-1400_climo_1351-1400.status:ERROR (1)
mpas_analysis_ts_0101-1450_climo_1401-1450.status:ERROR (1)
$ grep "OK" global_time_series_*.status
global_time_series_0001-0300.status:OK
global_time_series_0001-0400.status:OK
global_time_series_0001-0450.status:OK
global_time_series_0001-0500.status:OK
global_time_series_0001-0600.status:OK
global_time_series_0001-0650.status:OK
global_time_series_0001-0700.status:OK
global_time_series_0001-0750.status:OK
global_time_series_0001-0800.status:OK
global_time_series_0001-0850.status:OK
global_time_series_0001-0900.status:OK
global_time_series_0001-0950.status:OK
global_time_series_0001-1000.status:OK
global_time_series_0001-1050.status:OK
global_time_series_0001-1100.status:OK
global_time_series_0001-1150.status:OK
global_time_series_0001-1200.status:OK
global_time_series_0001-1250.status:OK
global_time_series_0001-1300.status:OK
global_time_series_0001-1350.status:OK

It does look like only the global_time_series jobs beginning at 0001 ran successfully.

There are two types of errors:
ERROR (5) is what you describe here.

Traceback (most recent call last):
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 651, in <module>
    run_by_region(sys.argv)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 647, in run_by_region
    run(parameters, rgn)
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/coupled_global.py", line 588, in run
    exp["annual"]["ohc"] = ts.globalAnnual("ohc")
  File "/lcrc/group/e3sm2/ac.wlin/E3SMv3/20231209.v3.LR.piControl-spinup.chrysalis/post/scripts/global_time_ser\
ies_0901-1100_dir/readTS.py", line 65, in globalAnnual
    v = cdutil.YEAR(v)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 1416, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 477, in get
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdu\
til-8.2.1-py3.11.egg/cdutil/times.py", line 751, in monthBasedSlicer
TypeError: 'NoneType' object is not subscriptable

Corresponds to

zppy/zppy/templates/global_time_series.bash

Lines 88 to 95 in 4db3e24

    
           echo 'Update time series figures' 
        
           cd ${global_ts_dir} 
        
           python coupled_global.py ${case_dir} ${experiment_name} ${figstr} ${start_yr} ${end_yr} {{ color }} ${ts_num_years} ${atmosphere_only} "{{ plot_names }}" {{ regions }} 
        
           if [ $? != 0 ]; then 
        
             cd {{ scriptDir }} 
        
             echo 'ERROR (5)' > {{ prefix }}.status 
        
             exit 5 
        
           fi

$ grep "ERROR (5)" global_time_series_*.status
global_time_series_0101-1400.status:ERROR (5)
global_time_series_0901-1100.status:ERROR (5)
global_time_series_1001-1200.status:ERROR (5)
global_time_series_1001-1300.status:ERROR (5)
global_time_series_1001-1350.status:ERROR (5)
global_time_series_1001-1400.status:ERROR (5)
global_time_series_1001-1500.status:ERROR (5)
global_time_series_1001-1550.status:ERROR (5)
global_time_series_1001-1700.status:ERROR (5)
global_time_series_1001-1900.status:ERROR (5)
global_time_series_1001-1950.status:ERROR (5)
global_time_series_1001-2000.status:ERROR (5)
global_time_series_1501-1800.status:ERROR (5)
global_time_series_1501-1900.status:ERROR (5)
global_time_series_1501-2000.status:ERROR (5)

ERROR (3)

Traceback (most recent call last):
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/bin/cdscan", line 1843, in <modu\
le>
    main(sys.argv)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/bin/cdscan", line 1309, in main
    vartime[0],
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/cdm\
s2/axis.py", line 2212, in __getitem__
    raise IndexError('index out of bounds')
IndexError: index out of bounds

Corresponds to

zppy/zppy/templates/global_time_series.bash

Lines 67 to 75 in 4db3e24

    
           echo 'Create xml for for ocn' 
        
           export CDMS_NO_MPI=true 
        
           cd ${case_dir}/post/ocn/glb/ts/monthly/${ts_num_years}yr 
        
           cdscan -x glb.xml mpaso.glb*.nc 
        
           if [ $? != 0 ]; then 
        
             cd {{ scriptDir }} 
        
             echo 'ERROR (3)' > {{ prefix }}.status 
        
             exit 3 
        
           fi

$ grep "ERROR (3)" global_time_series_*.status
global_time_series_0901-1150.status:ERROR (3)
global_time_series_1001-1150.status:ERROR (3)
global_time_series_1501-1950.status:ERROR (3)

wlin7 added the semver: bug Bug fix (will increment patch version) label Feb 4, 2024

chengzhuzhang mentioned this issue Feb 5, 2024

[Bug]: e3sm_to_cmip exception running bundles on Unified 1.9.2 #543

Closed

forsyth2 added this to forsyth2 current tasks Feb 5, 2024

github-project-automation bot moved this to Todo in forsyth2 current tasks Feb 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Failure to generate global time series plot if not starting from year 1 #546

[Bug]: Failure to generate global time series plot if not starting from year 1 #546

wlin7 commented Feb 4, 2024

forsyth2 commented Mar 1, 2024

forsyth2 commented Mar 1, 2024

forsyth2 commented Mar 2, 2024 •

edited

Loading

[Bug]: Failure to generate global time series plot if not starting from year 1 #546

[Bug]: Failure to generate global time series plot if not starting from year 1 #546

Comments

wlin7 commented Feb 4, 2024

What happened?

What machine were you running on?

Environment

What command did you run?

Copy your cfg file

What jobs are failing?

What stack trace are you encountering?

forsyth2 commented Mar 1, 2024

forsyth2 commented Mar 1, 2024

forsyth2 commented Mar 2, 2024 • edited Loading

forsyth2 commented Mar 2, 2024 •

edited

Loading