Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove RUNDIRS before running CI cases to cover re-run events #3076

Conversation

TerrenceMcGuinness-NOAA
Copy link
Collaborator

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA commented Nov 7, 2024

Description

This Pr removes pslot dir (with hash) from local archive folder on CI case completion and also adds cleanup_experiment function in the BASH utilities to clean up after a CI case runs. This change consolidates cleanup actions and adds functionality to clean the local archive folders.

Key changes:

  • Fixes bug for cleaning up the RUNDIRS directory before new experiments are run in the case the pipeline is being ran for a second time
  • Updated $HOMEgfs/ci/scripts/check_ci.sh to use the new cleanup_experiment function under the BASH CI system.
  • Added cleanup_experiment function in $HOMEgfs/ci/scripts/utils/ci_utils.sh to handle various cleanups including:
    • COMROOT/EXPDIR per case
    • ARCDIR and ATADIR directories for archive
    • STMP/RUNDIRS/${PSLOT}
  • Added get config var from EXPDIR utility get_config_var.py for getting config values in BASH

Resolves #2954, #3066

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

This PR still needs to be tested with the CI system which would effectively be a self-test.
image# Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • I have made corresponding changes to the system documentation if necessary

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI/CD Issue related to CI/CD label Nov 7, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Nov 7, 2024
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Nov 7, 2024
@emcbot
Copy link

emcbot commented Nov 7, 2024

Checkout Failed on Hera in Build# 2: Error cloning remote repo 'origin'

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Nov 7, 2024
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 7, 2024
@TerrenceMcGuinness-NOAA
Copy link
Collaborator Author

TerrenceMcGuinness-NOAA commented Nov 7, 2024

Checkout failed for gfs but not for gefs on a basic git command on Hera. Simply restarting CI in attempts to repeat the error.
Error was not repeatable for the gfs checkout and proceeded normally. scm also did the checkout in normal git fashion with updates only on previously gefs side as well.

@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Nov 7, 2024
@emcbot
Copy link

emcbot commented Nov 7, 2024

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3076


Experiment C48_ATM_6d349326 Completed 2 Cycles: *SUCCESS* at Thu Nov  7 13:41:03 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_6d349326 Completed 1 Cycles: *SUCCESS* at Thu Nov  7 14:05:23 CST 2024
Experiment C48_S2SW_6d349326 Completed 2 Cycles: *SUCCESS* at Thu Nov  7 15:06:18 CST 2024
Experiment C96_atm3DVar_6d349326 Completed 3 Cycles: *SUCCESS* at Thu Nov  7 15:18:15 CST 2024
Experiment C48_S2SWA_gefs_6d349326 Completed 1 Cycles: *SUCCESS* at Thu Nov  7 15:31:08 CST 2024
Experiment C96C48_hybatmDA_6d349326 Completed 3 Cycles: *SUCCESS* at Thu Nov  7 15:42:48 CST 2024

@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Nov 9, 2024
@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Nov 9, 2024
@emcbot
Copy link

emcbot commented Nov 9, 2024

CI Passed on Hercules in Build# 17
Built and ran in directory /work2/noaa/global/CI/HERCULES/3076


Experiment C48_ATM_1e17407f Completed 2 Cycles: *SUCCESS* at Fri Nov  8 22:31:17 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_1e17407f Completed 1 Cycles: *SUCCESS* at Fri Nov  8 22:43:24 CST 2024
Experiment C48_S2SWA_gefs_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 00:14:57 CST 2024
Experiment C96_atm3DVar_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 00:26:20 CST 2024
Experiment C48_S2SW_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 00:38:24 CST 2024
Experiment C96C48_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 00:50:51 CST 2024

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Nov 9, 2024
@emcbot
Copy link

emcbot commented Nov 9, 2024

CI Passed on Hera in Build# 18
Built and ran in directory /scratch1/NCEPDEV/global/CI/3076


Experiment C48_ATM_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 07:18:41 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 07:49:07 UTC 2024
Experiment C48mx500_3DVarAOWCDA_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 08:19:42 UTC 2024
Experiment C48_S2SWA_gefs_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 09:40:22 UTC 2024
Experiment C48_S2SW_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 09:51:05 UTC 2024
Experiment C96C48_hybatmaerosnowDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 10:45:46 UTC 2024
Experiment C96_atm3DVar_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 10:51:50 UTC 2024
Experiment C96C48_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 11:04:37 UTC 2024
Experiment C96C48_ufs_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 12:11:02 UTC 2024

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Nov 9, 2024
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Nov 9, 2024
@emcbot
Copy link

emcbot commented Nov 9, 2024

Checkout Failed on Orion in Build# 19: Error fetching remote repo 'origin'

1 similar comment
@emcbot
Copy link

emcbot commented Nov 9, 2024

Checkout Failed on Orion in Build# 19: Error fetching remote repo 'origin'

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Nov 9, 2024
@TerrenceMcGuinness-NOAA
Copy link
Collaborator Author

Massive failure of scm checkout of global-workflow on Orion.
Looking into a lead with this part of the error log.

fatal: write error: Disk quota exceeded
fatal: fetch-pack: invalid index-pack output

@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Hotfix: Removing RUNDIRS before running CI cases to cover re-run events Remove RUNDIRS before running CI cases to cover re-run events Nov 12, 2024
@WalterKolczynski-NOAA
Copy link
Contributor

PR description needs to be updated before merging.

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 1872499 into NOAA-EMC:develop Nov 13, 2024
5 checks passed
WalterKolczynski-NOAA pushed a commit to WalterKolczynski-NOAA/global-workflow that referenced this pull request Nov 22, 2024
…MC#3076)

This Pr removes **pslot** dir (with hash) from local archive folder on
CI case completion and also adds `cleanup_experiment` function in the
BASH utilities to clean up after a CI case runs. This change
consolidates cleanup actions and adds functionality to clean the local
archive folders.

Key changes:

- Fixes bug for cleaning up the `RUNDIRS` directory before new
experiments are run in the case the pipeline is being ran for a second
time
- Updated `$HOMEgfs/ci/scripts/check_ci.sh` to use the new
`cleanup_experiment` function under the BASH CI system.
- Added `cleanup_experiment `function in
`$HOMEgfs/ci/scripts/utils/ci_utils.sh` to handle various cleanups
including:
  - COMROOT/EXPDIR per case
  - ARCDIR and ATADIR directories for archive
  - STMP/RUNDIRS/${PSLOT}
- Added get config var from EXPDIR utility
[get_config_var.py](https://github.com/NOAA-EMC/global-workflow/pull/2961/files#diff-b780733ae1d45917730364e09a6c510e79d4cc8cffad6d9020c7961c53b987bc)
for getting config values in BASH

Resolves NOAA-EMC#2954
Resolves NOAA-EMC#3066
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA deleted the ci_develop_selftest branch December 6, 2024 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI/CD Issue related to CI/CD CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove on disk archive files in CI pipeline after running a test
4 participants