Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CTest: split checksum analysis from test analysis, expose arguments #5456

Open
wants to merge 8 commits into
base: development
Choose a base branch
from

Conversation

EZoni
Copy link
Member

@EZoni EZoni commented Nov 13, 2024

Prototype of implementation to see if this can achieve goals such as:

  1. run test and analysis locally (possibly without CTest) without worrying about checksums
  2. avoid duplicate code for default regression analysis with custom parameters (e.g., tolerance, output format)
  3. minimize work needed to implement checksum regression analysis for new tests

This PR replaces #5447, see #5447 (comment).

Old usage

  1. Add this to the test analysis script:
import os
import sys
...
...
sys.path.insert(1, "../../../../warpx/Regression/Checksum/")
from checksumAPI import evaluate_checksum
...
...
# compare checksums	
evaluate_checksum(	
    test_name=os.path.split(os.getcwd())[1],	
    output_file=sys.argv[1],	
)	
  1. Add this to the CMakeLists.txt file:
add_warpx_test(
    test_1d_laser_acceleration_fluid_boosted  # name
    1  # dims
    2  # nprocs
    inputs_test_1d_laser_acceleration_fluid_boosted  # inputs
    analysis_1d_fluid_boosted.py  # analysis
    diags/diag1000001  # output
    OFF  # dependency
)

New usage

  1. Add this to the CMakeLists.txt file:
add_warpx_test(
    test_1d_laser_acceleration_fluid_boosted  # name
    1  # dims
    2  # nprocs
    inputs_test_1d_laser_acceleration_fluid_boosted  # inputs
    "analysis_1d_fluid_boosted.py diags/diag1000001"  # analysis
    "analysis_default_regression.py --path diags/diag1000001 --plotfile"  # checksum
    OFF  # dependency
)

The updated default regression analysis script has the following usage:

usage: analysis_default_regression.py [-h] [--path PATH] (--plotfile | --openpmd) [--rtol RTOL] [--skip-fields]
                                      [--skip-particles]
options:
  -h, --help        show this help message and exit
  --path PATH       path to output file(s)
  --plotfile        output format is plotfile
  --openpmd         output format is openPMD
  --rtol RTOL       relative tolerance to compare checksums
  --skip-fields     skip fields when comparing checksums
  --skip-particles  skip particles when comparing checksums

To-do

  • Add missing checksum files or update existing checksum files that were not used
  • Update documentation

Follow-up

  • Improve documentation even more:
    • do we need all user-facing features of checksumAPI or are some obsolete?
    • can we merge documentation for testing and checksums into one section?
  • Check/fix custom tolerances (git grep "# checksum" Examples/ | grep "rtol")
  • Add logic to reset tolerances based on environment variables (e.g., to run all tests in single-precision)

@EZoni EZoni added the component: tests Tests and CI label Nov 13, 2024
@EZoni EZoni force-pushed the ctest_checksums_args branch 8 times, most recently from e341b73 to 9df2a19 Compare November 15, 2024 22:21
@EZoni EZoni changed the title [WIP] CTest: split checksum analysis from test analysis, expose arguments CTest: split checksum analysis from test analysis, expose arguments Nov 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: tests Tests and CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants