Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need to restrict CAMDEN python version to 3.7 or later #156

Closed
nusbaume opened this issue Sep 24, 2021 · 6 comments
Closed

Need to restrict CAMDEN python version to 3.7 or later #156

nusbaume opened this issue Sep 24, 2021 · 6 comments
Labels
code clean-up Made code simpler, better, and/or easier to read. enhancement New feature or request Github action Related to Github Actions for this repo.

Comments

@nusbaume
Copy link
Collaborator

There are several python features and code-cleanup requests that would be beneficial to apply in CAMDEN, but which require python 3.7 or later. Thus it would be helpful to explicitly require that CAMDEN use a python version that is newer or equal to version 3.7, which can now be done given that CIME has now moved over entirely to python 3.

In order to make this shift and extract the maximum set of benefits, several modifications should be made in the existing CAMDEN code and repo. These are:

  1. Need to update the CIME tag to at least cime5.8.44 or later. Otherwise python 2 can mistakenly be called during runtime which will cause CAMDEN to fail if the code changes below are implemented.

  2. Need to remove python 3.5 and 3.6 checks from Github Actions workflow, because they will fail if these listed changes are implemented.

  3. Add a python 3.7 check in ConfigCAM, and remove all other python version checks (e.g. the allow_abrev option should now always be present and set to False).

  4. Remove the (object) line from ConfigGen.

  5. Remove REGEX_TYPE = type(re.compile(r" ")) and replace it with re.Pattern in the instance-checking code.

  6. Replace current string.format(X) code with f-strings, and re-enable the consider-using-f-string check in the .pylintrc file.

Any other possible version-dependent changes that are found during development should also be added to this issue, at least until the version switch has been made and this issue is closed.

@nusbaume nusbaume added enhancement New feature or request Github action Related to Github Actions for this repo. code clean-up Made code simpler, better, and/or easier to read. labels Sep 24, 2021
@nusbaume
Copy link
Collaborator Author

Making a note here that there are multiple python version checks in cam_config_unit_tests.py and write_init_unit_tests.py that can be removed once the 3.7 version restriction has been implemented.

@nusbaume
Copy link
Collaborator Author

nusbaume commented Oct 4, 2021

Adding another note here that unless one is doing multi-class inheritance, the super() function does not require any (class, self) arguments in python 3 (apparently some versions of pylint/black complain about it). Might be worth removing?

@nusbaume
Copy link
Collaborator Author

nusbaume commented Oct 5, 2021

Making yet another note to update the Github Action python scripts as well, as they too are linted, and so should conform to python 3.7 or later.

Also noting that the error message starting on line 223 in pr_mod_file_tests.py is missing the format() statement, which can be added/replaced during the python version switch as well.

@gold2718
Copy link
Collaborator

Replace current string.format(X) code with f-strings, and re-enable the consider-using-f-string check in the .pylintrc file.

f strings seem to work in 3.6 (at least 3.6.8 on Izumi).

@nusbaume
Copy link
Collaborator Author

nusbaume commented Nov 1, 2021

I think the only new feature we would miss with python 3.6 is the use of re.Pattern. Given that we have a work-around for it already, I am ok with dropping the minimum version down to 3.6.

If that is ok with everyone else I'll go ahead and change the issue title.

@gold2718
Copy link
Collaborator

gold2718 commented Nov 1, 2021

I think the only new feature we would miss with python 3.6 is the use of re.Pattern.

Right. I mentioned this over in ESMCI/cime#4116
I think we should bump the CAMDEN version to 3.7 once that is resolved.
In the meantime, since this only affects testing at the moment, I think we are stable with 3.6, just make sure you use >= 3.7 when running tests.

nusbaume added a commit to nusbaume/CAM-SIMA that referenced this issue Oct 9, 2023
7b6d92e Merge pull request ESCOMP#198 from johnpaulalex/gitdir
927ce3a Merge pull request ESCOMP#197 from johnpaulalex/testpath
a04f114 Merge pull request ESCOMP#196 from johnpaulalex/readmod
d9c14bf Change the rest of the methods to use -C. Still some usage of getcwd in test_unit_repository_git.
332b106 Fix incorrect logged path of checkout_externals in test_sys_checkout: it was basically the parent of the current directory, which varies throughout the test. (it called abspath with '{0}/../../', which adds arbitrary and not-interpolated subdir '{0}' to the path, then removes it and removes one more level).
932a749 Remove printlog from read_gitmodules_file since read_externals_description_file() already has a nearly-the-same printlog (but add it to the other caller).
5d13719 Merge pull request ESCOMP#195 from johnpaulalex/check_repo
4233954 Update utest to mock _git_remote_verbose in a new way, since it is now called via the GitRepository class rather than on the specific GitRepository instance.
d7a42ae Check that desired repo was actually checked out.
71596bb Merge pull request ESCOMP#194 from johnpaulalex/manic2
4c96e82 Make the MANIC_TEST_BARE_REPO_ROOT env var special - give it a constant for easy tracking, and automatically tear it down after each test.
259bfc0 test_sys_checkout: use actual paths in on-the-fly configs rather than MANIC_TEST_BARE_REPO_ROOT env var. This will make it easier to test (in the near future) that checkout_externals actually checked out the desired repo dir.
557bbd6 Merge pull request ESCOMP#193 from johnpaulalex/manic
5314eed Remove MANIC_TEST_TMP_REPO_ROOT environment variable in favor of module-level variable.
345fc1e Merge pull request ESCOMP#191 from johnpaulalex/test_doc12
2117b84 test_sys_checkout: verify that basic by-tag/branch/hash tests actually take us to the correct git tag/branch/hash.
94d6e5f Merge pull request ESCOMP#190 from johnpaulalex/test_doc11
3ff33a6 Inline local-path-creation methods
47dea7f Merge pull request ESCOMP#189 from johnpaulalex/test_doc10
9ea75cb Grab-bag of renamings: Remove redundant _NAME from repo constants, and consistently add _REPO suffix (This causes the majority of diffs).
c0c847e Merge pull request ESCOMP#188 from johnpaulalex/test_doc9
2dd5ce0 test_sys_checkout.py: only check for correct 'required' or 'optional' state in the test that exercises required vs optional behavior. Removed a lot of boilerplate.
eb30859 Merge pull request ESCOMP#187 from johnpaulalex/test_doc8
1832e1f test_sys_checkout: Simplify many tests to only use a single external.
8689d61 Merge pull request ESCOMP#186 from johnpaulalex/test_doc7
fbee425 Grab bag of test_sys_checkout cleanups:    Doc inside of each test more clearly/consistently.    TestSysCheckoutSVN didn’t get the inlining-of-helper-methods treatment, now it has that.    Move various standalone repo helper methods (like create_branch) into a RepoUtils class.    README.md was missing newlines when rendered as markdown.    Doc the return value of checkout.main    Fix test_container_exclude_component - it was looking for the wrong key (which is never present); now it looks for the correct key.
f0ed44a Merge pull request ESCOMP#185 from johnpaulalex/test_doc6
a3d59f5 Merge pull request ESCOMP#184 from johnpaulalex/test_doc5
5329c8b test_sys_checkout: Inline config generation functions that are only called once.
464f2c7 test_sys_checkout: Inline another layer (per-config-file checks). Rename the 4 methods that are used multiple times, to reflect what they do rather than what they're called.
8872c0d Merge pull request ESCOMP#183 from johnpaulalex/doc_test4
c045335 Merge pull request ESCOMP#182 from johnpaulalex/doc_test3
c583b95 Merge pull request ESCOMP#181 from johnpaulalex/doc_test2
e01cfe2 test_sys_checkout: less confusing handling of return values from checkout_externals. Specifically, when doing a checkout, don't return tree_status from _before_ the checkout. Make a new wrapper to call checkout_externals a second time, to calculate the new status after a checkout (very frequent pattern).
2328681 test_sys_checkout: Remove another layer (which generates test component names)
c3717b6 Merge pull request ESCOMP#180 from johnpaulalex/doc_test
36d7a44 test_sys_checkout.py: remove one layer of functions (that check for local status enums). No-op.
2c4584b More documentation about tests: * contents of test repositories (n a new README.md) * various constants in test_sys_checkout.py that point to those contents, and terminology like container/simple/mixed. * in each test method, the scenarios being tested. * The coupling between test methods.
55e74bd Merge pull request ESCOMP#179 from johnpaulalex/circ
66be842 Remove circular dependency by making _External stop doing tricky things with sourcetrees.
82d3b24 Merge pull request ESCOMP#178 from johnpaulalex/test_doc
3223f49 Additional documentation of system tests - global variables, method descriptions.
45b7c01 Merge pull request ESCOMP#177 from jedwards4b/git_workflow
ace90b2 try setting credentials this way
f4d6aa9 try setting credentials this way
1d61a69 use this to set git credentials
7f9d330 use this to set git credentials
5ac731b add tmate code
836847b get git workflow working
dcd462d Merge pull request ESCOMP#176 from jedwards4b/add_github_testing
2d2479e Merge pull request ESCOMP#175 from johnpaulalex/fix
711a53f add github testing of prs and automatic tagging of main
cfe0f88 fix typos
5665d61 Fix broken checkout behavior introduced by PR ESCOMP#172.
27909e2 Merge pull request ESCOMP#173 from johnpaulalex/readall
00ad044 Further tiny refactorings and docs of checkout API (no-op).    Remove unused load_all param in _External.checkout().    Rename _External.checkout_externals() to checkout_subexternals(), to remove the ambiguity about whether the main external pointed to by the _External is itelf checked out (it is not)    Clarify load_all documentation - it’s always recursive, but applies different criteria at each level.    Rename variables in checkout.py (e.g. ext_description)  to match the equivalent code in sourcetree.py.
2ea3d1a Merge pull request ESCOMP#172 from johnpaulalex/fixit
43bf809 Merge pull request ESCOMP#171 from johnpaulalex/docstatus
e6aa7d2 Merge pull request ESCOMP#170 from johnpaulalex/printdir
adbd715 On checkout, refresh locally installed optional packages regardless of whether -o is passed in.
add0745 Comment tweaks, and fix 'ppath' typo
696527c Document the format of various status dictionaries, and the various paths and path components within an _External.
c677b94 When processing an external, print out its path in addition to the base filename (to disambiguate all the externals.cfg's)
975d7fd Merge pull request ESCOMP#169 from johnpaulalex/docfix_branch
09709e3 Document _Externals.status().  The original comment was apparently copy-pasted from checkout().
1d880e0 Merge pull request ESCOMP#167 from billsacks/fix_svn_on_windows
3510da8 Tweak a unit test to improve coverage
eb7fc13 Handle the possibility that the URL already ends with '/'
02ea87e Fix svn URLs on Windows
b1c02ab Merge pull request ESCOMP#165 from gold2718/doc_fix
9f4be8c Add documentation about externals = None feature
a3b3a03 Merge pull request ESCOMP#162 from ESMCI/fischer/python3
d4f1b1e Change shebang lines to python3
2fd941a Merge pull request ESCOMP#158 from billsacks/modified_solution
de08dc2 Add another option for when an external is in a modified state
e954582 Merge pull request ESCOMP#156 from billsacks/onbranch_show_hash
952e44d Change output: put tag/hash before branch name
1028843 Fix pre-existing pylint issues
01b13f7 When on a branch, show tag/hash, too
39ad532 Merge pull request ESCOMP#150 from gold2718/fix_combo_config
75f8f02 Merge pull request ESCOMP#152 from jedwards4b/sort_by_local_path
42687bd remove commented code
29e26af fix pylint issues
7c9f3c6 add a test for nested repo checkout
75c5353 fix spacing
24a3726 improve sorting, checkout externals with each comp
29f45b0 remove py2 test and fix super call
880a4e7 remove decode
1c53be8 no need for set call
36c56db simplier fix for issue
dc67cc6 simpler solution
b32c6fc fix to allow submodule name different from path
5b5e1c2 Merge pull request ESCOMP#144 from billsacks/improve_errmsg
c983863 Add another option for dealing with modified externals
59ce252 Add some details to the error message when externals are modified
be5a1a4 Merge pull request ESCOMP#143 from jedwards4b/add_exclude
2aa014a fix lint issue
49cd5e8 fix lint issues
418173f Added tests for ExternalsDescriptionDict
afab352 fix lint issue
be85b7d fix the test
a580a57 push test
d437108 add a test
21affe3 fix formatting issue
72e6b64 add an exclude option
c33a3bd Merge pull request ESCOMP#139 from jedwards4b/ignore_branch_prop
b124a9a ignore this silently, we use a hash so it does not matter

git-subtree-dir: manage_externals
git-subtree-split: 7b6d92ef689e2f65733e27f8635ab91fb341356b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code clean-up Made code simpler, better, and/or easier to read. enhancement New feature or request Github action Related to Github Actions for this repo.
Projects
None yet
Development

No branches or pull requests

2 participants