Skip to content

Releases: epiforecasts/scoringutils

scoringutils 2.0.0

31 Oct 14:46
@nikosbosse nikosbosse
v2.0.0
efb41af
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.
Compare
Choose a tag to compare
scoringutils 2.0.0 Latest
Latest

This update represents a major rewrite of the package and introduces breaking changes. If you want to keep using the older version, you can download it using remotes::install_github("epiforecasts/[email protected]").

The update aims to make the package more modular and customisable and overall cleaner and easier to work with. In particular, we aimed to make the suggested workflows for evaluating forecasts more explicit and easier to follow (see visualisation below). To do that, we clarified input formats and made them consistent across all functions. We refactored many functions to S3-methods and introduced forecast objects with separate classes for different types of forecasts. A new set of as_forecast_<type>() functions was introduced to validate the data and convert inputs into a forecast object (a data.table with a forecast class and an additional class corresponding to the forecast type (see below)). Another major update is the possibility for users to pass in their own scoring functions into score(). We updated and improved all function documentation and added new vignettes to guide users through the package. Internally, we refactored the code, improved input checks, updated notifications (which now use the cli package) and increased test coverage.

The most comprehensive documentation for the new package after the rewrite is the revised version
of our original scoringutils paper.

See the NEWS file for a detailed overview of the changes.

What's Changed

Read more

Contributors

sbfnk, MichaelChirico, and 7 other contributors
Assets 2
Loading

scoringutils 1.2.2

29 Nov 17:22
@nikosbosse nikosbosse
v1.2.2
413d427
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
scoringutils 1.2.2

This minor release includes two bug fixes, as well as some small changes to the package infrastructure. It reflects the current version on CRAN, 1.2.2.

Package updates

  • Added a startup message to inform users of an upcoming major update and asking for feedback (see #333)
  • Added some fixes and updates to the package infrastructure on github
  • Updates the minimum required R version to 3.6

Bug fixes

  • Fixed a bug in set_forecast_unit() that made the function fail if a data.frame instead of a data.table was provided (thank you, @elray1)
  • Fixed a bug in the metrics table in the "Scoring forecasts directly vignette" where some rows were duplicated (thank you, @elray1)

What's Changed

Full Changelog: v1.2.0...v1.2.2

Contributors

sbfnk, elray1, and nikosbosse
Assets 2
Loading
0 Join discussion

scoringutils 1.2.0

26 Jul 07:40
@nikosbosse nikosbosse
v1.2.0
f74e08f
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
scoringutils 1.2.0

This major release contains a range of new features and bug fixes that have been introduced in minor releases since 1.1.0. The most important changes are:

  • Documentation updated to reflect changes since version 1.1.0, including new transform and workflow functions.
  • New set_forecast_unit() function allows manual setting of forecast unit.
  • summarise_scores() gains new across argument for summarizing across variables.
  • New transform_forecasts() and log_shift() functions allow forecast transformations. See the documentation for transform_forecasts() for more details and an example use case.
  • Input checks and test coverage improved for bias functions.
  • Bug fix in get_prediction_type() for integer matrix input.
  • Links to scoringutils paper and citation updates.
  • Warning added in interval_score() for small interval ranges.
  • Linting updates and improvements.

Thanks to @nikosbosse, @seabbs, and @sbfnk for code and review contributions. Thanks to @Bisaloo for the suggestion to use a linting GitHub Action that only triggers on changes, and @adrian-lison for the suggestion to add a warning to interval_score() if the interval range is between 0 and 1.

Package updates

  • The documentation was updated to reflect the recent changes since scoringutils 1.1.0. In particular, usage of the functions set_forecast_unit(), check_forecasts() and transform_forecasts() are now documented in the Vignettes. The introduction of these functions enhances the overall workflow and help to make the code more readable. All functions are designed to be used together with the pipe operator. For example, one can now use something like the following:
example_quantile |> 
  set_forecast_unit(c("model", "location", "forecast_date", "horizon", "target_type")) |> 
  check_forecasts() |> 
  score()

Documentation for the transform_forecasts() has also been extended. This functions allows the user to easily add transformations of forecasts, as suggested in the paper "Scoring epidemiological forecasts on transformed scales". In an epidemiological context, for example, it may make sense to apply the natural logarithm first before scoring forecasts, in order to obtain scores that reflect how well models are able to predict exponential growth rates, rather than absolute values. Users can now do something like the following to score a transformed version of the data in addition to the original one:

data <- example_quantile[true_value > 0, ]
data |>
  transform_forecasts(fun = log_shift, offset = 1) |> 
  score() |> 
  summarise_scores(by = c("model", "scale"))

Here we use the log_shift() function to apply a logarithmic transformation to the forecasts. This function was introduced in scoringutils 1.1.2 as a helper function that acts just like log(), but has an additional argument offset that can add a number to every prediction and observed value before applying the log transformation.

Feature updates

  • Made check_forecasts() and score() pipeable (see issue #290). This means that
    users can now directly use the output of check_forecasts() as input for
    score(). As score() otherwise runs check_forecasts() internally anyway
    this simply makes the step explicit and helps writing clearer code.

What's Changed

Full Changelog: v1.1.0...v1.2.0

Contributors

sbfnk, Bisaloo, and 3 other contributors
Assets 2
Loading

scoringutils version 1.1.0

30 Jan 18:34
@nikosbosse nikosbosse
v1.1.0
692b8bb
Compare
Choose a tag to compare
scoringutils version 1.1.0

A minor update to the package with some bug fixes and minor changes.

Feature updates

Package updates

  • Removed the on attach message which warned of breaking changes in 1.0.0.
  • Renamed the metric argument of summarise_scores() to relative_skill_metric. This argument is now deprecated and will be removed in a future version of the package. Please use the new argument instead.
  • Updated the documentation for score() and related functions to make the soft requirement for a model column in the input data more explicit.
  • Updated the documentation for score(), pairwise_comparison() and summaris_scores() to make it clearer what the unit of a single forecast is that is required for computations
  • Simplified the function plot_pairwise_comparison() which now only supports plotting mean score ratios or p-values and removed the hybrid option to print both at the same time.

Bug fixes

  • Missing baseline forecasts in pairwise_comparison() now trigger an explicit and informative error message.
  • The requirements table in the getting started vignette is now correct.
  • Added support for an optional sample column when using a quantile forecast format. Previously this resulted in an error.

List of Pull Requests

New Contributors

Full Changelog: v1.0.0...v1.1.0

Contributors

sbfnk, damonbayer, and 3 other contributors
Assets 2
Loading

1.0.0

17 May 17:38
@nikosbosse nikosbosse
v1.0.0
aefc221
Compare
Choose a tag to compare
1.0.0

Major update to the package and most package functions with lots of breaking changes.

Feature updates

  • new and updated Readme and vignette
  • the proposed scoring workflow was reworked. Functions were changed so they
    can easily be piped and have simplified arguments and outputs.

new functions and function changes

  • the function eval_forecasts() was replaced by a function score() with a
    much reduced set of function arguments.
  • Functionality to summarise scores and to add relative skill scores was moved
    to a function summarise_scores()
  • new function check_forecasts() to analyse input data before scoring
  • new function correlation() to compute correlations between different metrics
  • new function add_coverage() to add coverage for specific central prediction
    intervals
  • new function avail_forecasts() allows to visualise the number of available
    forecasts
  • new function find_duplicates() to find duplicate forecasts which cause an
    error
  • all plotting functions were renamed to begin with plot_. Arguments were
    simplified
  • the function pit() now works based on data.frames. The old pit function
    was renamed to pit_sample(). PIT p-values were removed entirely.
  • the function plot_pit() now works directly with input as produced by pit()
  • many data-handling functions were removed and input types for score() were
    restricted to sample-based, quantile-based or binary forecasts.
  • the function brier_score() now returns all brier scores, rather than taking
    the mean before returning an output.
  • crps, dss and logs were renamed to crps_sample(), dss_sample(), and
    logs_sample()

Bug fixes

  • Testing was expanded
  • minor bugs were fixed, for example a bug in the sample_to_quantile function
    (#223)

package data updated

  • package data is now based on forecasts submitted to the European Forecast Hub
    (https://covid19forecasthub.eu/).
  • all example data files were renamed to begin with example_
  • a new data set, summary_metrics was included that contains a summary of the
    metrics implemented in scoringutils

Other breaking changes

  • The 'sharpness' component of the weighted interval score was renamed to
    dispersion. This was done to make it more clear what the component represents
    and to maintain consistency with what is used in other places.
Assets 2
Loading

0.1.8

18 Mar 15:20
@Bisaloo Bisaloo
v0.1.8
23c68f6
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
0.1.8

Full Changelog: v0.1.7...v0.1.8

Contributors

sbfnk, Bisaloo, and 2 other contributors
Assets 2
Loading

0.1.7

18 Mar 14:56
@nikosbosse nikosbosse
v0.1.7
ec3fdf5
Compare
Choose a tag to compare
0.1.7 
v0.1.7

make criteria to classify a forecast as binary stricter
Assets 2
Loading