Skip to content

ELM Example Case V. elm ats coupled

Jump to bottom Edit New page
Fengming Yuan edited this page Mar 3, 2023 · 5 revisions

ELM Examples of Setting-up, Configuring, Building, and Running V.

**ELM-ATS case at Portage River outlet nearby-town, Oakharbor, OHIO **

image

NOTE:

  • In this example, there are 3 user-defined ENVs variables of directories: $E3SM_INPUTDATA, $E3SM_ROOT, $PROJECT_E3SM, for inputdata, e3sm model source codes, and root directory for cases and runs, respectively. USERS must have created these, and cloned both data and e3sm codes.

FOR this ELM-ATS example, the source ELM codes are branch EMI-ats as ELM-ATS branch

  • For ATS installed inc/libs, there are at least 2 user-defined ENVs variables of directories: $AMANZI_TPLS_DIR, $ATS_DIR, for TPLs and ATS with elm-ats interface API installed, respectively. USERS must have created these, and BOTH are built successfully.

  • $E3SM_INPUTDATA is cloned repository of a few site-level full datasets to run offline ELM, from pt-e3sm-inputdata.

Under this directory, there is an ats input subdirectory as well $E3SM_INPUTDATA/lnd/clm2/ats/

The 'ELM_USR_NAME' for this example is called 1x1pt_Oakharbor-GRID

  • Upon user's machine, $MACH_NAME is needed for E3SM. In these examples, 'mymac' for MacOS/linux, 'wsl' for Windows Subsystem for Linux, OR, 'cades' for a cluster in ORNL, may be directly used.

HERE we manually customize E3SM Land Model to run with option of 'CPL_BYPASS' (coupler_bypass), developed by Dr. Daniel Ricciuto at ORNL. This configuration will run offline ELM about 3 times faster and vey-slightly different results, although not bit-for-bit

++++++++++++++++++++++++++++++++++++++++++++

There are 4-5 forms of this configuration, upon what meteorological dataset to drive the land model. We will demonstrate 1 of them as following: (2) 'GSWP3' - point metdata extracted from half-degree global datasets , which is CPL_BYPASS version of ELM Example Case II. Point Mode with CPL_BYPASS, but with different forcing.

++++++++++++++++++++++++++++++++++++++++++++

CASE V: single-point user-defined satm-elm (CPL_BYPASS metdata_type of 'GSWP3).

  • STEP 1. Create a PTCLM case, with --compset ICBxxxx
  1) cd $E3SM_ROOT/cime/scripts

  2) ./create_newcase --case $PROJECT_E3SM/cases/elm_ats-ICB1850CNPRDCTCBC --res ELM_USRDAT --mach $MACH_NAME --compiler gnu --compset ICB1850CNPRDCTCBC --walltime 06:00:00
  • STEP 2. Configure the case created in STEP 1 (extracted 1 grid dataset from half-degree global one, namely '1x1pt_Oakharbor' from 1901-2014)
  3) cd $PROJECT_E3SM/cases/elm_ats-ICB1850CNPRDCTCBC

  4) ./xmlchange, (OR, vi env_run.xml)
             --id ELM_USRDAT_NAME --val 1x1pt_Oakharbor-GRID

 ADDITIONALLY, 
 if 'accelerated spinup', you need to edit 'env_run.xml' as following:
             --id CLM_FORCE_COLDSTART --val on
             --id CLM_ACCELERATED_SPINUP --val on
             --id CLM_BLDNML_OPTS --val '-bgc bgc -nutrient cn -nutrient_comp_pathway rd  -soil_decomp ctc -methane -nitrif_denitrif' 
     (i.e. remove P limitation for ad_spinup)

By default, the model run on machine is upon some pre-configured PE setting. For an example, the case setup by now MAY or MAY NOT run model on 1 node with 8 MPI Task or '-1'. If so, that's not good for 1 PT model, so vi env_mach_pes.xml as following so that model runs on 1 node with 1 MPI Tasks

  6) vi env_mach_pes.xml

    <entry id="NTASKS">
      <type>integer</type>
      <values>
        <value compclass="ATM">1</value>
        <value compclass="CPL">1</value>
        <value compclass="OCN">1</value>
        <value compclass="WAV">1</value>
        <value compclass="GLC">1</value>
        <value compclass="ICE">1</value>
        <value compclass="ROF">1</value>
        <value compclass="LND">1</value>
        <value compclass="ESP">1</value>
        <value compclass="IAC">1</value>
      </values>
      <desc>number of tasks for each component</desc>
    </entry> 

    <entry id="NTASKS_PER_INST">
      <type>integer</type>
      <values>
        <value compclass="ATM">1</value>
        <value compclass="CPL">1</value>
        <value compclass="OCN">1</value>
        <value compclass="WAV">1</value>
        <value compclass="GLC">1</value>
        <value compclass="ICE">1</value>
        <value compclass="ROF">1</value>
        <value compclass="LND">1</value>
        <value compclass="ESP">1</value>
        <value compclass="IAC">1</value>
      </values>
      <desc>number of tasks for each component</desc>
    </entry> 

OR, 
./xmlchange NTASKS=1
./xmlchange NTASKS_PER_INST=1

IF PNETCDF is not available,
./xmlchange PIO_TYPENAME=netcdf

THEN, issue the following command:

  7) ./case.setup 
(./case.setup --clean FOR cleaning-up)

The above command may or may not be successful (mostly with errors relevant to missing input files). AND it's also missing of metdata and setting for CPL_BYPASS. So, modifying user_nl_elm as following - please replace $E3SM_INPUTDATA with actually absoluted path:

 metdata_type = 'gswp3'
 metdata_bypass = '$E3SM_INPUTDATA/atm/datm7/atm_forcing.datm7.GSWP3.0.5d.v2.c180716_Oakharbor-Grid/cpl_bypass_full'
 fsurdat = '$E3SM_INPUTDATA/lnd/clm2/surfdata_map/surfdata_1x1pt_Oakharbor-GRID_simyr1850_c360x720_c20220201.nc'
 flanduse_timeseries = '$E3SM_INPUTDATA/lnd/clm2/surfdata_map/landuse.timeseries_1x1pt_Oakharbor-GRID_simyr1850-2015_c360x720_c20220201.nc'
 aero_file = '$E3SM_INPUTDATA//atm/cam/chem/trop_mozart_aero/aero/aerosoldep_monthly_1850_mean_1.9x2.5_c090803.nc'
 CO2_file = '$E3SM_INPUTDATA//atm/datm7/CO2/fco2_datm_1765-2007_c100614.nc'
 nyears_ad_carbon_only = 25
 spinup_mortality_factor = 10

 use_ats = .true.
 ats_inputdir = '$E3SM_INPUTDATA/lnd/clm2/ats'
 ats_inputfile = 'column_elm4ats.xml'

NOTE: (1)the last 3 options are for ATS. If use_ats = .false., ELM-ATS model will run as ELM only.

(2) Those options are to instruct model read datasets from indicated directory or files.

In this case, the 'metdata' is located in directory metdata_bypass with 7 files GSWP3_*_1901-2014_z19.nc, and an ascii file zone_mappings.txt. So, make sure those files are existed in those inputdata directories.

TWO (2) surface data, 'fsurdat=...', 'flanduse_timeseries=...', are used for spinups and transient runs. Make sure they are in the specified directories and file names are correct. IF NOT, edit as NEEDED.

(3) an example user_nl_elm file is under $E3SM_INPUTDATA/lnd/clm2/ats

THEN, re-issue command ./case.setup

  • STEP 3. Build the case from STEP 1-2

FOR ELM-ATS coupled, ONE more editing or checking IS required before code compiling.

(for newer version ELM since 2021-Oct), checking CAREFULLY cmake_macros/userdefined_cmake, which IS EXACTLY for ELM-ATS interface:

string(APPEND CONFIG_ARGS " ")
string(APPEND CPPDEFS " -DCPL_BYPASS ")
set(AMANZI_TPLS_DIR "$ENV{AMANZI_TPLS_DIR}")
set(ATS_DIR "$ENV{ATS_DIR}")
if (COMP_CLASS STREQUAL lnd)
  if (NOT ${AMANZI_TPLS_DIR} STREQUAL "")
    string(APPEND FFLAGS " -I${AMANZI_TPLS_DIR}/trilinos-13-0-afc4e525/include")
    string(APPEND FFLAGS " -I${AMANZI_TPLS_DIR}/SEACAS/include ")
    string(APPEND FFLAGS " -I${AMANZI_TPLS_DIR}/petsc-3.16/include -I${AMANZI_TPLS_DIR}/pflotran/src ")
    if (NOT ${ATS_DIR} STREQUAL "")
      string(APPEND CPPDEFS " -DUSE_ATS_LIB ")
      string(APPEND FFLAGS " -I${ATS_DIR}/include ")
    endif()
  endif()
endif()
if (COMP_CLASS STREQUAL cpl)
  string(APPEND LDFLAGS " -lstdc++")
  if (NOT ${AMANZI_TPLS_DIR} STREQUAL "")
    string(APPEND LDFLAGS " -L${AMANZI_TPLS_DIR}/lib")
    string(APPEND LDFLAGS " -L${AMANZI_TPLS_DIR}/trilinos-13-0-afc4e525/lib")
    string(APPEND LDFLAGS " -L${AMANZI_TPLS_DIR}/SEACAS/lib ")
    string(APPEND LDFLAGS " -L${AMANZI_TPLS_DIR}/petsc-3.16/lib -L${AMANZI_TPLS_DIR}/pflotran/src ")
    if (NOT ${ATS_DIR} STREQUAL "")
      string(APPEND LDFLAGS " -L${ATS_DIR}/lib -lerror_handling -latk -lfunctions -lgeometry -lgeochemutil -lgeochemsolvers -lgeochembase -lgeochemrxns -lgeochemistry -lmesh -lmesh_audit -lmesh_simple -lmesh_mstk -lmesh_extracted -lmesh_logical -lmesh_factory -ldbg -lwhetstone -ldata_structures -lmesh_functions -loutput -lstate -lsolvers -ltime_integration -loperators -lpks -lchemistry_pk -ltransport -lshallow_water -lats_operators -lats_eos -lats_surf_subsurf -lats_generic_evals -lats_column_integrator -lats_pks -lats_energy_relations -lats_energy -lats_flow_relations -lats_flow -lats_transport -lats_sed_transport -lats_deform -lats_surface_balance -lats_bgc -lats_mpc_relations -lats_mpc -lelm_ats")
    endif()
  endif()
endif()
  • PAY attention to 3 critical hard-wired path/file names: trilinos-13-0-afc4e525, petsc-3.16, -L${ATS_DIR}/lib -l....... Especially, the link library list (long and varied upon ATS building options). YOU may check a txt file, link_line , under ${ATS_DIR}/lib. (As mentioned before, ${ATS_DIR} is your ATS installation directory)

THEN, build the model:

  8) ./case.build
(./case.build --clean      #cleaning-up ALL E3SM model components
 ./case.build --clean lnd  #cleaning-up LND component
 ./case.build --clean-all  #cleaning-up ALL model components and External components)
  • STEP 4. Run the case
  ./case.submit

NOTES: *(a) the column_elm4ats.xml example under $E3SM_INPUTDATA/lnd/clm2/ats has a few hard-wired PATH/FILE names, So must edit it as what really copied or changed in your case.

*(b) the run direcory is:$PROJECT_E3SM/cases/elm-ats_ICB1850CNPRDCTCBC/run

*(c) The run length is 5 days, with monthly output, i.e. NO real data write out EXCEPT for restart files at the end.

*(d) If you'd like what-ever time-length of the simulation, editing env_run.xml, for an example, like the following:

  --id STOP_OPTION nyears
  --id STOP_N 200
 (These editing will let model run 200 years, with monthly outputs of default variables, AND will produce 2 restart files)

*(d) If simulation may not be finished in WALLCLOCK time, it will crash. Then have to editing env_run.xml to resume run:

   --id CONTINUE_RUN TRUE

NOTE: The unfinished run MUST have produced 2 restart files, .elm.r..nc, .cpl.r..nc. IF NOT, have to redo run from beginning, with smaller interval for producing restart files, like (retart file writing every 10 years):

  --id REST_N 10
  --id REST_OPTION nyears

Updated @ 2024-04-24

Add a custom sidebar
Clone this wiki locally