add python 3.7.0 module load for izumi (CESM only)

[peverwhee]

Add module load for python 3.7 on izumi for CESM in config_machines.xml.
Without this update, the python version is set in cime regardless of
whether a user does a module load before building. This addition ensures
that CAM specifically is being built & run using python 3.7.

Test suite: scripts_regression_tests.py, clean builds of CAM, CESM
Test baseline:
Test namelist changes:
Test status: bit for bit

Fixes #4116

User interface changes?: N

Update gh-pages html (Y/N)?: N

[billsacks]

I voiced my support for something like this in #4116, but I'd still like to understand this need a little more. I have two questions:

(1) On izumi, it seems that the default python3, with no modules loaded, is python3.6.8. I have had no problems running I compsets with that python version. Is the issue that CAM's python scripts require python 3.7+? If so, we should document that (e.g., in the software requirements listed in the README file in https://github.com/escomp/cesm), and possibly have a minimum version check somewhere appropriate to avoid cryptic issues if someone tries to run with 3.6 on a different machine (since CIME's overall minimum version check is currently for 3.6)

cime/scripts/Tools/standard_script_setup.py

Line 17 in 90b4e37

CIME.utils.check_minimum_python_version(3, 6)

(2) I'm thinking that, for things to work consistently, izumi users should do a module load of python3.7 in their environment before running the cime scripts; do you agree?

[jedwards4b]

@billsacks Users need to module load python before invoking any cesm scripts, but it's also needed here as the module purge line above that one will remove the users module environment.

[billsacks]

Users need to module load python before invoking any cesm scripts, but it's also needed here as the module purge line above that one will remove the users module environment.

Let me clarify my question: Up until now I have not needed to do a module load of python on izumi, because the default system-level python (3.6) has been sufficient. Is this changing now, at least for some configurations?

[peverwhee]

@billsacks

The issue we're running into is in our CAMDEN (ccpp-ized CAM) development. The subtle differences in Python 3.6 and 3.7 are causing failures related to inheritance and the use of super(). You are right in that CAM is unaffected by these differences, so I'm not sure it would make sense to update the documentation (yet).

[jedwards4b]

@peverwhee are the ccpp developers aware of this issue and have they considered supporting python3.6? This kind of requirement should really be justified.

[billsacks]

Okay, thanks for explaining @peverwhee . I'll go ahead and approve, though I agree with @jedwards4b 's last comment.

[peverwhee]

@jedwards4b yes, we're all aware of the issue.

Perhaps @gold2718 can explain this better?

[gold2718]

@peverwhee are the ccpp developers aware of this issue and have they considered supporting python3.6

@jedwards4b, the CCPP Framework does support python 3.6 (i.e., its standalone tests pass using the default python on Izumi). The issue is in running our build scripts called from buildlib.

Now, I am not sure the issue is python 3.6 but there is a problem.

Using our current version of cime5.8.38, I get:

Generating component namelists as part of build
  2021-10-28 12:35:27 atm 
Create namelist for component cam
ERROR: Python 3, minor version 6 is required, you have 2.7

All our executable scripts have the python3 shebang so I have no idea why that happens but when I do module load lang/python/3.7.0, then the python command becomes 3.7.0.

I checked out cime6.0.10 and made the change to buildlib (MODEL => COMP_NAME). Now I get:

Did not find an alias or longname compset match for FPHYStest 
Automatically adding SATM to compset
Automatically adding SLND to compset
Automatically adding SICE to compset
Automatically adding SOCN to compset
Automatically adding SROF to compset
Automatically adding SGLC to compset
Automatically adding SWAV to compset
Automatically adding SESP to compset
ERROR: Invalid compset name, FPHYStest, all stub components generated

Thoughts? Help?

[jedwards4b]

@gold2718 Are you using a cam branch? I can't find a compset call FPHYStest in cams config_compsets.xml

[gold2718]

@gold2718 Are you using a cam branch?

As @peverwhee mentioned, this is the new infrastructure. The relevant branch is:
https://github.com/peverwhee/CAMDEN/tree/updaters

[jedwards4b]

Did you check to see if compset FPHYStest is defined therein? It might be helpful if you pointed me to your sandbox on izumi.

[gold2718]

I have a sandbox at /home/goldy/Projects/CAMDEN

[jedwards4b]

I figured out the issue - the cime that you added to CAMDEN was a standalone checkout of cime, that is someone ran manage_externals from the cime subdirectory. There is an open issue #3923 But you also need to update the externals for the new broken up cime directory structure. I have made these changes in my sandbox in /scratch/cluster/jedwards/tmp/CAMDEN but it doesn't seem that ***@***.***:NCAR/CAMDEN exists so I can't give you a PR. Don't tell me you are using private repos.

…

On Thu, Oct 28, 2021 at 4:09 PM goldy ***@***.***> wrote: I have a sandbox at /home/goldy/Projects/CAMDEN — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4117 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOXUGFT736ZNNVA3G6QQDTUJHCZVANCNFSM5G5LHFNA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Jim Edwards CESM Software Engineer National Center for Atmospheric Research Boulder, CO

[gold2718]

But you also need to update the externals for the new broken up cime
directory structure.

Thanks!

[billsacks]

@peverwhee @gold2718 can you summarize where this stands? Do you still want this change or did you decide that this is unnecessary for now? I'm still okay with it if it's needed, and my impression is that @jedwards4b and @fischer-ncar are, too, though my feeling is that if it isn't actually needed, it seems best to skip it for now.

[gold2718]

can you summarize where this stands?

We would like to proceed for the reasons I recently added to #4116 because that is (what we believe to be) code improvement we cannot make in 3.6. An important consideration here is whether it is okay to require python 3.7 in CIME build scripts.

[gold2718]

@billsacks, Thanks!