-
Notifications
You must be signed in to change notification settings - Fork 6
ELM on cades, or, condo: elm model customizing examples, with option 'cpl_bypass', and, 'user pfts'
HERE we customize E3SM Land Model to run with option of 'CPL_BYPASS' (coupler_bypass), developed by Dr. Daniel Ricciuto at ORNL. This configuration will run offline ELM about 3 times faster and vey-slightly different results, although not bit-for-bit
- E3SM https://github.com/E3SM-Project/E3SM.git, Branch:
fmyuan/lnd/elm-userpft_v2
There are 4-5 forms of this configuration, upon what meteorological dataset to drive the land model. We will demonstrate 2 of them as following: (1) 'site' - user-provided point metdata ; and (2) 'GSWP3v1_daymet' - GSWP3 v1 adjusted with Daymet
- As for directly run OLMT for 1 global grid only, with modified land domain & surface data, the example script is in this link: https://github.com/dmricciuto/OLMT/blob/Arctic-userpft/site_fullrun_userpft_example.sh
- STEP 1. Create a PTCLM case, with
--compset ICBxxxx
1) cd $CCSI_DIR/proj-shared/models/e3sm/cime/scripts
2) ./create_newcase --case /lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cases/ELM_US-Brw_ICB1850CNPRDCTCBC --res CLM_USRDAT --mach cades --compiler gnu --compset ICB1850CNPRDCTCBC --project ccsi --walltime 00:30:00
- STEP 2. Configure the case created in STEP 1 (user-defined 1 point dataset, namely '1x1pt_US-Brw' from 1985-2015)
3) cd /lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cases/ELM_US-Brw_ICB1850CNPRDCTCBC
4) ./xmlchange, OR, vi env_run.xml
--id SAVE_TIMING --val FALSE
--id CLM_USRDAT_NAME --val 1x1pt_US-Brw
--id LND_DOMAIN_PATH --val /lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cesminput_ngee/share/domains/domain.clm
--id LND_DOMAIN_FILE --val domain.lnd.1x1pt_US-Brw_navy.nc
ADDITIONALLY (not availble in E3SM trunk),
if 'accelerated spinup', you need to edit 'env_run.xml' as following:
--id CLM_ACCELERATED_SPINUP --val on
--id CLM_BLDNML_OPTS --val '-bgc bgc -nutrient cn -nutrient_comp_pathway rd -soil_decomp ctc -methane -nitrif_denitrif'
(i.e. remove P limitation for ad_spinup)
if 'cold startup', you need to edit 'env_run.xml' as following:
--id CLM_FORCE_COLDSTART --val on
if NOT coupling with pflotran, better NOT to include libpflotran.a, otherwise which must exist in indicated directory, by editing 'env_mach_specific.xml' as following:
5) vi env_mach_specific.xml
<!--
<env name="PFLOTRAN_INC"> -I$ENV{CLM_PFLOTRAN_SOURCE_DIR} -I$ENV{PETSC_DIR}/include</env>
<env name="PFLOTRAN_LIB"> -L$ENV{CLM_PFLOTRAN_SOURCE_DIR} -lpflotran -L$ENV{PETSC_DIR}/lib -lpetsc -lmetis -lparmetis</env>
-->
By default, the model run on CADES is upon some pre-configured setting. For an example, the case setup by now will run model on 1 node with 8 MPI Task. So that's not good for 1 PT model, so vi env_mach_pes.xml as following so that model runs on 1 node with 1 MPI Tasks
6) vi env_mach_pes.xml
<entry id="NTASKS">
<type>integer</type>
<values>
<value compclass="ATM">1</value>
<value compclass="CPL">1</value>
<value compclass="OCN">1</value>
<value compclass="WAV">1</value>
<value compclass="GLC">1</value>
<value compclass="ICE">1</value>
<value compclass="ROF">1</value>
<value compclass="LND">1</value>
<value compclass="ESP">1</value>
</values>
<desc>number of tasks for each component</desc>
</entry>
THEN, issue the following command:
7) ./case.setup
(./case.setup --clean FOR cleaning-up)
The above command may or may not be successful (mostly with errors relevant to missing input files). AND it's also missing of metdata and setting for CPL_BYPASS. So, modifying user_nl_clm as following:
metdata_type = 'site'
metdata_bypass = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cesminput_ngee/atm/datm7/1x1pt_US-Brw/cpl_bypass_full'
aero_file = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/ACME_inputdata/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_monthly_1850_mean_1.9x2.5_c090803.nc'
CO2_file = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/ACME_inputdata/atm/datm7/CO2/fco2_datm_1765-2007_c100614.nc'
nyears_ad_carbon_only = 25
spinup_mortality_factor = 10
NOTE: (1)the last 2 options are optional, but which can accelerate spinup much better. (2) Those options are to instruct model read datasets from indicated directory or files. In this case, the 'metdata' is located in directory metdata_bypass with filename all_hourly.nc.
THEN, re-issue command ./case.setup
- STEP 3. Build the case from STEP 1-2
FOR CPL_BYPASS, ONE more editing IS required before code compiling. Edit Macro.make under case directory as following (appending -DCPL_BYPASS into CPPDEFS)
CPPDEFS := $(CPPDEFS) -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DCPL_BYPASS
THEN, build the model:
8) ./case.build
(./case.build --clean #cleaning-up ALL E3SM model components
./case.build --clean lnd #cleaning-up LND component
./case.build --clean-all #cleaning-up ALL model components and External components)
- STEP 4. Run the case
./case.submit
NOTES:
*(a) the run direcory is:$CCSI_DIR/scratch/$USER/ELM_US-Brw_ICB1850CNPRDCTCBC/run
*(b) The run length is 5 days, with monthly output, i.e. NO real data write out EXCEPT for restart files at the end.
*(c) If you'd like what-ever time-length of the simulation, editing env_run.xml, for an example, like the following:
--id STOP_OPTION nyears
--id STOP_N 600
(These editing will let model run 600 years)
II. CASE 2: single-point ELM mode, with user-defined PFT, 1 point site-level GSWP3v1 atm forcing, 6 Ecotypes (land grids) at NGEE-Arctic Kougarok Sites. (CPL_BYPASS metdata_type of 'GSWP3v1_daymet').
- STEP 1. Create a PTCLM case
1) cd $CCSI_DIR/proj-shared/models/e3sm/cime/scripts
2) ./create_newcase --case /lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cases/ELMuserpft_Kougarok_ICB1850CNPRDCTCBC --res CLM_USRDAT --mach cades --compiler gnu --compset ICB1850CNPRDCTCBC --project ccsi --walltime 00:30:00
- STEP 2. Configure case just created in STEP 1: (1) 1 grid atm data (1985-2010); (2) 6 user-defined point land surface datasets for NGEE-Arctic Kougarok Sites; (3) user-defined Arctic PFTs.
3) cd /lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cases/ELMuserpft_Kougarok_ICB1850CNPRDCTCBC
4) ./xmlchange, OR, vi env_run.xml, to modify configurations as following:
--id SAVE_TIMING --val FALSE
--id CLM_USRDAT_NAME --val 1x1pt_kougarok-NGEE
--id LND_DOMAIN_PATH --val /lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cesminput_ngee/share/domains/domain.clm
--id LND_DOMAIN_FILE --val domain.lnd.51x63pt_kougarok-NGEE_TransA_navy.nc
ADDTIONALLY (not availble in E3SM trunk), if 'accelerated spinup', you need to edit 'env_run.xml' as following:
--id CLM_ACCELERATED_SPINUP --val on
--id CLM_BLDNML_OPTS --val '-bgc bgc -nutrient cn -nutrient_comp_pathway rd -soil_decomp ctc -methane -nitrif_denitrif'
(i.e. remove P limitation for accelerated_spinup)
if 'cold startup', you need to edit 'env_run.xml' as following:
--id CLM_FORCE_COLDSTART --val on
if NOT coupling with pflotran, better NOT to include libpflotran.a, otherwise which must exist in indicated directory, by editing 'env_mach_specific.xml' as following:
5) vi env_mach_specific.xml
<!--
<env name="PFLOTRAN_INC"> -I$ENV{CLM_PFLOTRAN_SOURCE_DIR} -I$ENV{PETSC_DIR}/include</env>
<env name="PFLOTRAN_LIB"> -L$ENV{CLM_PFLOTRAN_SOURCE_DIR} -lpflotran -L$ENV{PETSC_DIR}/lib -lpetsc -lmetis -lparmetis</env>
-->
- STEP 3. Configure model run parallelism setting
By default, the model run on CADES is upon some pre-configured setting. For an example, the case setup by now will run model on 1 node with 1 MPI Task. So that's not good, so vi env_mach_pes.xml as following so that model runs on 1 node with 6 MPI Tasks
6) vi env_mach_pes.xml
<entry id="NTASKS">
<type>integer</type>
<values>
<value compclass="ATM">6</value>
<value compclass="CPL">6</value>
<value compclass="OCN">6</value>
<value compclass="WAV">6</value>
<value compclass="GLC">6</value>
<value compclass="ICE">6</value>
<value compclass="ROF">6</value>
<value compclass="LND">6</value>
<value compclass="ESP">1</value>
</values>
<desc>number of tasks for each component</desc>
</entry>
NOTE: by case-creating in STEP 1, the model run on CADES for maximal 30 minutes (Wallclocktime). For longer simulation, we need to modify that. SO vi env_batch.xml as following for 'group id="case.run" ', for an example, to run the model up to 48 hours (note: this is the wallclock time allowable currently on CADES):
<entry id="JOB_WALLCLOCK_TIME" value="48:00:00">
<type>char</type>
<valid_values/>
<desc>The machine wallclock setting. Default determined in config_machines.xml can be overwritten by testing</desc>
</entry>
<entry id="JOB_QUEUE" value="batch">
<type>char</type>
<valid_values/>
<desc>The machine queue in which to submit the job. Default determined in config_machines.xml can be overwritten by testing</desc>
</entry>
<entry id="USER_REQUESTED_WALLTIME" value="48:00:00">
<type>char</type>
<desc>Store user override for walltime</desc>
</entry>
- STEP 4. setup the case configured in STEP 2-3
7) ./case.setup
(./case.setup --clean FOR cleaning-up setting-up)
Firstly, The model setup will not be successful, with error relevant to missing 'fsurdat' file and its associated PFT parameter file. So, For User-defined PFT in this configuration, edit file user_nl_clm by adding the following:
fsurdat = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cesminput_ngee/lnd/clm2/surfdata_map/surfdata_51x63pt_kougarok-NGEE_TransA_simyr1850_c181115-sub12.nc'
paramfile = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cesminput_ngee/lnd/clm2/paramdata/clm_params_c180524-sub12.nc'
nyears_ad_carbon_only = 25
spinup_mortality_factor = 10
NOTE: the last 2 options are optional, but which can accelerate spinup much better.
ONE model edit For User-defined PFT is the 'maxpft' (which default is 17), by vi env_run.xml, modifying CLM_BLDNML_OPTS, like following
<entry id="CLM_BLDNML_OPTS" value="-maxpft 12 -bgc bgc -nutrient cnp -nutrient_comp_pathway rd -soil_decomp ctc -methane -nitrif_denitrif ">
<type>char</type>
<desc>CLM build-namelist options</desc>
</entry>
Secondly, it's also missing of metdata and setting for CPL_BYPASS. So, modifying user_nl_clm by adding the following:
metdata_type = 'gswp3v1_daymet'
metdata_bypass = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cesminput_ngee/atm/datm7/atm_forcing.datm7.GSWP3_daymet.1x1pt_kougarok-NGEE/cpl_bypass_full'
aero_file = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/ACME_inputdata/atm/cam/chem/trop_mozart_aero/aero/aerosoldep_monthly_1850_mean_1.9x2.5_c090803.nc'
CO2_file = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/ACME_inputdata/atm/datm7/CO2/fco2_datm_1765-2007_c100614.nc'
NOTE: Those options are to instruct model read datasets from indicated directory or files. In this case, the 'metdata' is located in directory metdata_bypass with 7 files of GSWP3_Daymet3_xxx_1980-2010_z16.nc and 1 file called zone_mapping.txt.
- STEP 5. Build the case
FOR CPL_BYPASS, ONE more editing IS required before code compiling. Edit Macro.make under case directory as following (appending -DCPL_BYPASS into CPPDEFS)
CPPDEFS := $(CPPDEFS) -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DCPL_BYPASS
THEN, build the model:
8) ./case.build
(./case.build --clean #cleaning-up ALL E3SM model components
./case.build --clean lnd #cleaning-up LND component
./case.build --clean-all #cleaning-up ALL model components and External components)
- STEP 6. Run the case
9) ./case.submit
NOTES:
*(a) the run direcory is:$CCSI_DIR/scratch/$USER/ELMuserpft_Kougarok_ICB1850CNPRDCTCBC/run
*(b) The run length is 5 days, with monthly output, i.e. NO real data write out EXCEPT for restart files at the end.
*(c) If you'd like what-ever time-length of the simulation, editing env_run.xml, for an example, like the following:
--id STOP_OPTION nyear
--id STOP_N 600
(These editing will let model run 600 years)
NOTE: you probably need to modify JOB_WALLCLOCK_TIME in env_batch.xml mentioned above in STEP 3.
*(d) If you'd like what-ever time-interval of produce RESTART files, again editing env_run.xml, for an example, like the following:
--id REST_OPTION nyear
--id REST_N 100
(These editing will let model output restart files every 100 years)
*(e) If your run like that of 600 years may not finish in the wallclock time, you need to editing env_run.xml to resume the model from the LAST saved restart point, like the following:
--id CONTINUE_RUN TRUE
TIPS: Since we configured to save 'restart' files every 100 years, the model will resume from THE LAST save point. It's critical for user, therefore, to configure 'REST_N', 'STOP_N', and 'JOB_WALLCLOCK_TIME' all properly, so that the run would save both IO and run-time. Normally those restart files are large and its frequent writing cost time.
*(f) AFTER you're done with ACCELERATED_SPINUP (aka ad_spinup), you need to run normal spinup, by vi env_run.xml as following:
--id CLM_ACCELERATED_SPINUP --val off
--id CLM_FORCE_COLDSTART --val off
AND, adding initial state into user_nl_clm, as following:
finidat = './ELMuserpft_Kougarok_ICB1850CNPRDCTCBC.clm2.r.0601-01-01-00000.nc'
Here, we assume the file ELMuserpft_Kougarok_ICB1850CNPRDCTCBC.clm2.r.0601-01-01-00000.nc is in the directory: $CCSI_DIR/scratch/$USER/ELMuserpft_Kougarok_ICB1850CNPRDCTCBC/run.
THEN, issue command: ./case.submit as before. This will run model for another 600 years as normal spinup stage. The newly generated ELMuserpft_Kougarok_ICB1850CNPRDCTCBC.clm2.r.0601-01-01-00000.nc will be used as ```finidat`` for TRANSIENT run (1850 - 2010 in our case here).
- STEP 6. Run the case for TRANSIENT stage (1850-2010)
THIS TIME, you need to: (1) create a fresh case with --compset ICB20TRCNPRDCTCBC; (2) configure case with -id STOP_N -val 160 years, in addition to WHAT have configured for spinup.
AND, adding initial state into user_nl_clm, as following:
finidat = '$CCSI_DIR/scratch/$USER/ELMuserpft_Kougarok_ICB1850CNPRDCTCBC/run/ELMuserpft_Kougarok_I1850CNPRDCTCBC.clm2.r.0601-01-01-00000.nc'
FINALLY, adding user-defined LULC into user_nl_clm, as following:
flanduse_timeseries = '/lustre/or-hydra/cades-ccsi/proj-shared/project_acme/cesminput_ngee/lnd/clm2/surfdata_map/landuse.timeseries_51x63pt_kougarok-NGEE_TransA_simyr1850_c181115-sub12.nc'
(NOTE: you MUST explicitly expand the $CCSI_DIR and $USER into real directory name; OR, copy that finitdat file into your transient run directory).