Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove .nc files found in OUTPUT_DIR depending on config file #710

Merged
merged 1 commit into from
Nov 18, 2024

Conversation

jtmims
Copy link
Collaborator

@jtmims jtmims commented Nov 18, 2024

Description
Currently, the MDTF holds onto all of the preprocessed data in the WORK_DIR, and it only removes the data in each POD's dir. This PR goes in an removes the rest of the .nc files if "save_pp_data" is set to false in the config file. It is done by using the "case_info.yml" file found in each POD dir. This will help conserve space for runs of the MDTF that involve many PODs.

Associated issue #702

How Has This Been Tested?
The framework was ran on multiple GFDL datasets and with various PODs on my workstation.

Checklist:

  • My branch is up-to-date with the NOAA-GFDL main branch, and all merge conflicts are resolved
  • The scripts are written in Python 3.12 or above (preferred; required if funded by a CPO grant), NCL, or R
  • All of my scripts are in the diagnostics/[POD short name] subdirectory, and include a main_driver script, template html, and settings.jsonc file
  • I have made corresponding changes to the documentation in the POD's doc/ subdirectory
  • I have requested that the framework developers add packages required by my POD to the python3, NCL, or R environment yaml file if necessary, and my environment builds with conda_env_setup.sh
  • I have added any necessary data to input_data/obs_data/[pod short name] and/or input_data/model/[pod short name]
  • My code is portable; it uses MDTF environment variables, and does not contain hard-coded file or directory paths
  • I have provided the code to generate digested data files from raw data files
  • Each digested data file generated by the script contains numerical data (no figures), and is 3 GB or less in size
  • I have included copies of the figures generated by the POD in the pull request
  • The repository contains no extra test scripts or data files

@jtmims jtmims added the framework Issue pertains to the framework code label Nov 18, 2024
src/output_manager.py Dismissed Show dismissed Hide dismissed
@wrongkindofdoctor wrongkindofdoctor merged commit eb4ed79 into NOAA-GFDL:main Nov 18, 2024
5 checks passed
@jtmims jtmims deleted the output branch November 18, 2024 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
framework Issue pertains to the framework code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants