Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add snaphot tests of canned forecasters #307

Merged
merged 1 commit into from
Apr 10, 2024
Merged

Conversation

dshemetov
Copy link
Contributor

Checklist

Please:

  • Make sure this PR is against "dev", not "main".
  • Request a review from one of the current epipredict main reviewers:
    dajmcdon.
  • Makes sure to bump the version number in DESCRIPTION and NEWS.md.
    Always increment the patch version number (the third number), unless you are
    making a release PR from dev to main, in which case increment the minor
    version number (the second number).
  • Describe changes made in NEWS.md, making sure breaking changes
    (backwards-incompatible changes to the documented interface) are noted.
    Collect the changes under the next release number (e.g. if you are on
    0.7.2, then write your changes under the 0.8 heading).

Change explanations for reviewer

Adds snapshot tests. Could add more snapshots with different parameter settings, but I went with a few for now.

Magic GitHub syntax to mark associated Issue(s) as resolved when this is merged into the default branch

@dshemetov dshemetov requested a review from dajmcdon as a code owner April 3, 2024 23:08
@dshemetov dshemetov force-pushed the ds/snapshot-tests branch from 82f4744 to 6ac50bf Compare April 3, 2024 23:15
@dshemetov dshemetov force-pushed the ds/snapshot-tests branch from 6ac50bf to a6894d0 Compare April 3, 2024 23:38
@dshemetov
Copy link
Contributor Author

dshemetov commented Apr 3, 2024

Ran into an issue where the CI and my local snapshots disagreed. Ran renv::update() locally, updated packages, and it seemed to fix it by updating the snapshots. A little concerning that our canned forecasts can change in the 3rd decimal place based on updated dependencies. But at least these snapshots will help us identify when such changes come from our work or from an upstream dependency.

Copy link
Contributor

@dajmcdon dajmcdon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely sure what impact this has, but maybe you can explain before merging?

@dshemetov
Copy link
Contributor Author

dshemetov commented Apr 10, 2024

So for each of our canned forecasters, I included the forecast outputs of a few (~default) forecaster (using included JHU data) as snapshots in the tests. Now these tests will compare the output of those forecasts on developer branches and will give us (at least some) confidence we have not altered these forecaster outputs with our code changes. And if the forecasts have changed, you can run testthat::snapshot_review() to get a nice diff interface, where you can decide whether to accept or reject the new snapshots.

@dajmcdon
Copy link
Contributor

Cool! And it runs on R CMD Check?

@dshemetov
Copy link
Contributor Author

Yup, it's part of the tests, so it runs with devtools::test() too.

@dshemetov dshemetov merged commit 4dde3c4 into dev Apr 10, 2024
3 checks passed
@dshemetov dshemetov deleted the ds/snapshot-tests branch April 10, 2024 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add canned forecaster snapshot tests
2 participants