-
Notifications
You must be signed in to change notification settings - Fork 2
/
README.Rmd
192 lines (142 loc) · 9.16 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
set.seed(42)
```
```{r srr, eval = FALSE, echo = FALSE}
#' @srrstats {G1.2}
#' @srrstats {EA1.0} Model developers and decision support implementers
#' @srrstats {EA1.1}
#' @srrstats {EA1.2}
```
# predictNMB <a href='https://docs.ropensci.org/predictNMB/'><img src='man/figures/logo.png' align="right" height="139" /></a>
<!-- badges: start -->
[](https://github.com/ropensci/predictNMB/actions)
[](https://www.gnu.org/licenses/gpl-3.0)
[](https://app.codecov.io/gh/ropensci/predictNMB?branch=master)
[](https://www.repostatus.org/#active)
[](https://github.com/ropensci/software-review/issues/566)
[](https://doi.org/10.21105/joss.05328)
[](https://CRAN.R-project.org/package=predictNMB)
[](https://www.r-pkg.org/pkg/predictNMB)
<!-- badges: end -->
## Overview
predictNMB is a tool to evaluate (hypothetical) clinical prediction models based on their estimated Net Monetary Benefit (NMB). It may be used by prediction model developers who intend to find out how performant their model needs to be clinically useful or by those in health services deciding whether or not to implement an existing model.
`{predictNMB}` has two main functions:
- `do_nmb_sim()`: takes user-defined inputs for a given prediction model and population, then evaluates the NMB by performing simulations.
- `screen_simulation_inputs()`: calls `do_nmb_sim()` many times, using a range of values for any of its inputs. This is useful for sensitivity analysis.
## Installation
You can install the stable version from CRAN with:
``` r
install.packages("predictNMB")
```
You can install the development version from [GitHub](https://github.com/) with:
``` r
# install.packages("remotes")
remotes::install_github("ropensci/predictNMB")
# or, equivalently:
install.packages("predictNMB", repos = "https://ropensci.r-universe.dev")
```
## Estimating model performance
We must first define a hypothetical NMB associated with each square of a confusion matrix (2$\times$2 table). To do this, we create an R function which, when run, returns a named vector representing the four NMB values that we need. Since we are using Quality-adjusted life years lost (`qalys_lost`) due to the event and a willingness to pay (`wtp`) value of \$28,000, this will function will also store some other details so that we can create a cost-effectiveness plot later.
`get_nmb_sampler()` is a `predictNMB` function that makes creating the function easier! If you create these functions without `get_nmb_sampler()`, you can't make the cost-effectiveness plot but can still evaluate in terms of NMB.
Here, we make `NMBsampler` functions for cutpoint selection and evaluation separately. These incorporate uncertainty by passing in functions that sample values from their underlying distributions. The healthcare event leads to about 1 QALY lost, the cost of the treatment is about \$800 and the treatment reduces the rate of the event by about 18%.
```{r}
library(predictNMB)
fx_nmb_training <- get_nmb_sampler(
wtp = 28033,
qalys_lost = function() rnorm(n = 1, 1, 0.05), # about 1
high_risk_group_treatment_effect = function() rbeta(n = 1, 200, 900), # about 0.11
high_risk_group_treatment_cost = function() rgamma(n = 1, 2400, 3), # about $800
use_expected_values = TRUE
)
fx_nmb_evaluation <- get_nmb_sampler(
wtp = 28033,
qalys_lost = function() rnorm(n = 1, 1, 0.05),
high_risk_group_treatment_effect = function() rbeta(n = 1, 200, 900),
high_risk_group_treatment_cost = function() rgamma(n = 1, 2400, 3)
)
```
We can then pass this to the simulation function.
Required arguments:
- `n_sims`: number of simulations to run. More simulations take longer but are more stable.
- `event_rate`: event incidence rate, or the proportion of patients experiencing the event.
- `sim_auc`: vector of hypothetical AUCs; e.g. `seq(0.7, 0.95, 0.05)` or `c(0.75, 0.80, 0.85)`.
- `n_valid`: number of samples the validation set draws within each simulation (evaluating the NMB under each cutpoint).
- `fx_nmb_training`: function that returns a named vector used to get cutpoints on the training set. Recommended to use constant values.
- `fx_nmb_evaluation`: function that returns a named vector used to get cutpoints on the evaluation set. Recommended to use sampled values.
- `cl`: (Optional) users can pass a cluster made using `parallel::makeCluster()`. If it is given, the simulations are run in parallel (faster).
```{r, eval=FALSE}
library(parallel)
cl <- makeCluster(detectCores())
```
```{r, include=FALSE}
library(parallel)
chk <- Sys.getenv("_R_CHECK_LIMIT_CORES_", "")
if (nzchar(chk) && chk == TRUE) {
ncores <- 2
} else {
ncores <- detectCores()
}
cl <- makeCluster(ncores)
```
```{r warning=FALSE}
sim_screen_obj <- screen_simulation_inputs(
n_sims = 500, n_valid = 10000, sim_auc = seq(0.7, 0.95, 0.05), event_rate = 0.05,
fx_nmb_training = fx_nmb_training, fx_nmb_evaluation = fx_nmb_evaluation,
cutpoint_methods = c("all", "none", "youden"), cl = cl
)
```
These simulations can be interpreted as a range of hypothetical situations under different levels of model performance within our specific healthcare setting. We can visualise how this change may affect preferences between the model-guided strategy versus a treat-all or treat-none strategy — in other words, using the model to determine who should get treatment, rather than everyone or no one.
`autoplot()` can be used on the object returned from this function to quickly inspect these trends:
```{r, message=FALSE}
autoplot(sim_screen_obj)
```
Here, we are visualising many simulations under different inputs. If we just want to inspect a single set of inputs, say when the model AUC is 0.9, we can run that simulation alone using `do_nmb_sim()`, or access it from our existing screen.
```{r, eval=F}
single_sim_obj <- do_nmb_sim(
n_sims = 1000, n_valid = 10000, sim_auc = 0.9, event_rate = 0.1,
fx_nmb_training = fx_nmb, fx_nmb_evaluation = fx_nmb,
cutpoint_methods = c("all", "none", "youden"), cl = cl
)
```
```{r}
single_sim_obj <- sim_screen_obj$simulations[[5]]
```
When plotting a single set of simulation inputs, we see the distributions of the NMB across all simulations under each strategy. Note that we have added some tasteful themes using `theme_sim()` that help reduce clutter on these types of plots.
```{r}
autoplot(single_sim_obj) + theme_sim()
```
For this scenario, when the model has an AUC of 0.9, we can see that the NMB is best when using the model-guided decisions. However, decision-makers may want to visualise these on a cost-effectiveness plot which shows the health benefit versus the costs. We can do this using the `ce_plot()` function, specifying the treat-none option as the reference strategy.
```{r}
ce_plot(single_sim_obj, ref_col = "none")
```
From this plot, we can see that both the treat-all and model-guided decision lead to more QALYs than treating nobody (because more patients are receiving an effective treatment) but that the treat-all strategy is not cost-effective according to our cost-effectiveness plane (based on a willingness to pay of \$28,033). Based on this, we would conclude that the hypothetical model-guided decision support system would be likely to be cost-effective should we be able to develop a model with this level of discrimination for this specific healthcare problem/context.
## Citation
```{r}
citation("predictNMB")
```
## Further reading
The [`predictNMB` website](https://docs.ropensci.org/predictNMB/) and its vignettes:
- [Getting started with `predictNMB`](https://docs.ropensci.org/predictNMB/articles/predictNMB.html)
- [Summarising results from `predictNMB`](https://docs.ropensci.org/predictNMB/articles/summarising-results-with-predictNMB.html)
- [Detailed example: pressure injury model](https://docs.ropensci.org/predictNMB/articles/detailed-example.html)
- [Creating NMB functions](https://docs.ropensci.org/predictNMB/articles/creating-nmb-functions.html)
## Contributing
Please note that this package is released with a [Contributor Code of
Conduct](https://ropensci.org/code-of-conduct/). By contributing to this
project, you agree to abide by its terms.
- If you think you have encountered a bug, please [submit an
issue](https://github.com/ropensci/predictNMB).
- Please include a
[reprex](https://reprex.tidyverse.org/articles/articles/learn-reprex.html)
(a minimal, reproducible example) to clearly communicate about your
code.