Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python module load command for izumi #4116

Closed
peverwhee opened this issue Oct 27, 2021 · 4 comments · Fixed by #4117
Closed

Add python module load command for izumi #4116

peverwhee opened this issue Oct 27, 2021 · 4 comments · Fixed by #4117
Labels
Responsibility: CESM Responsibility to manage and accomplish this issue is through CESM

Comments

@peverwhee
Copy link
Collaborator

peverwhee commented Oct 27, 2021

To ensure CAM runs with python 3.7.0, add a module load command to the izumi section in config_machines.xml.

< command name="load" >lang/python/3.7.0

@rljacob
Copy link
Member

rljacob commented Oct 27, 2021

You're supposed to load your python environment prior to running any cime commands. Doing it within cime leads to chaos.

@billsacks
Copy link
Member

Under some circumstances, you need to do a module load python within cime in addition to having the right python module loaded in your own environment. e.g., I needed to add one for cheyenne for scripts_regression_tests to work correctly (#4033). I think the problem in that situation was that the overall python process (scripts_regression_tests) was spawning other CIME python processes (e.g., create_newcase), some of which were resetting the module environment according to config_machines, and this included a module reset that cleared any personally-loaded python module.

I can envision other possible issues that might occur even within a single create_newcase/case.setup/case.build/case.submit workflow: CIME calls things like buildlib as a python subprocess, so these scripts might be executed with the wrong python environment under some circumstances.

To deal with issues like this and other possibly unforeseen issues, I would support having an explicit module load of the correct python module. I agree with @rljacob that this mechanism can lead to chaos, though, and long-term we really need to think about #4059 .

@gold2718
Copy link

This is a CESM-only change (where there are already 11 explicit python module loads including 10 which appear to be loading a python 2.7 version).

@gold2718 gold2718 added the Responsibility: CESM Responsibility to manage and accomplish this issue is through CESM label Oct 28, 2021
@gold2718
Copy link

There has been a discussion over in #4117 about whether and why python 3.7 might be needed in CAM.
The new answer (after much testing) is that we do not need python 3.7 to build and run CAM.
However, we would like to use at least one python3.7 feature to clean up our code and for running tests.
An example of the change is in the "type" of a compiler regular expression.

In python 3.6.8 (default on Izumi):

>>> import re
>>> foo = re.compile(r"hi mom")
>>> type(foo)
<class '_sre.SRE_Pattern'>
>>> isinstance(foo, _sre.SRE_Pattern)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name '_sre' is not defined
>>> bar = foo.match("hi mom, how are you?")
>>> type(bar)
<class '_sre.SRE_Match'>

In python 3.7.0:

>>> import re
>>> foo = re.compile(r"hi mom")
>>> type(foo)
<class 're.Pattern'>
>>> isinstance(foo, re.Pattern)
True
>>> type(bar)
<class 're.Match'>
>>> isinstance(bar, re.Match)
True

So both in code (wanting to use duck typing to catch an input that may be a regular expression pattern) and in doctests (e.g., did a routine correctly return a regular expression match), python 3.7 offers a cleaner approach.

billsacks added a commit that referenced this issue Nov 8, 2021
add python 3.7.0 module load for izumi (CESM only)

Add module load for python 3.7 on izumi for CESM in config_machines.xml. 
Without this update, the python version is set in cime regardless of 
whether a user does a module load before building. This addition ensures 
that CAM specifically is being built & run using python 3.7.

Test suite: scripts_regression_tests.py, clean builds of CAM, CESM
Test baseline: 
Test namelist changes: 
Test status: bit for bit

Fixes #4116 

User interface changes?: N

Update gh-pages html (Y/N)?: N
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Responsibility: CESM Responsibility to manage and accomplish this issue is through CESM
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants