Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Final manuscript analyses and edits in targets pipeline #202

Merged
merged 112 commits into from
Dec 12, 2024
Merged

Final manuscript analyses and edits in targets pipeline #202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
112 commits
Select commit Hold shift + click to select a range
a628525
fix targets pipeline for new input fxns
kaitejohnson Oct 31, 2024
d5a935c
keep eval_fit from failing when ww is missing
kaitejohnson Oct 31, 2024
6a0cd96
tweak fit to not error and work if model runs
kaitejohnson Oct 31, 2024
990912a
save the things that were saved even when model fails, for consistenc…
kaitejohnson Oct 31, 2024
8539735
fix handling of model coloring
kaitejohnson Oct 31, 2024
8763cf8
for now use scoring utils v 1.2.0
kaitejohnson Nov 2, 2024
437fd05
fix version of scoringutils"
kaitejohnson Nov 5, 2024
4fe7325
other bug fixes from fix scoringutils branch
kaitejohnson Nov 5, 2024
6cf0c41
fix desc, remove from imports?
kaitejohnson Nov 5, 2024
1608353
get a scoringutils version that works in container
kaitejohnson Nov 5, 2024
2a70b2b
attempt to fix get draws from ww data and ww eval data
kaitejohnson Nov 5, 2024
ddd1371
fix to put extra cols after scoring just in case
kaitejohnson Nov 5, 2024
e9bf125
add benchmarks
kaitejohnson Nov 6, 2024
c6cfb15
make legend two rows using guides
kaitejohnson Nov 12, 2024
a4771bf
add sigma_rt plots to post process
kaitejohnson Nov 12, 2024
22b139d
tweak handling of rerunning failed jobs
kaitejohnson Nov 12, 2024
68e03b4
replace MA with CA for example, add option to specify sites for the f…
kaitejohnson Nov 13, 2024
f82287e
update docs
kaitejohnson Nov 13, 2024
d56e879
add specific sites for CA
kaitejohnson Nov 13, 2024
3185723
fix typo in scores
kaitejohnson Nov 13, 2024
3fb4ed8
fix bug in collection rfequency
kaitejohnson Nov 15, 2024
214572d
remove the flag around low ww values
kaitejohnson Nov 15, 2024
481197c
add heatmaps of performance by forecast date and location
kaitejohnson Nov 15, 2024
6354bcd
add metadata and heatmaps of metadata
kaitejohnson Nov 15, 2024
3974573
change fig size, add docs
kaitejohnson Nov 15, 2024
ec10a14
add docs
kaitejohnson Nov 15, 2024
cd8f8e8
update to rerun for 3 forecast date locs that fail wwinference checks
kaitejohnson Nov 15, 2024
ad82468
fix docs
kaitejohnson Nov 15, 2024
d3308ef
add workaround for wwinference failure
kaitejohnson Nov 16, 2024
f04cadd
add geom text to heatmap with performance of each forecast
kaitejohnson Nov 16, 2024
c991f0e
add a bunch of summary tables/figures
kaitejohnson Nov 16, 2024
c136699
write function to get ind forecast score
kaitejohnson Nov 16, 2024
0576be4
add function to get rel crps of ind location
kaitejohnson Nov 16, 2024
8927d33
add rel_crps heatmap to supplement
kaitejohnson Nov 16, 2024
804eb38
make rel_crps heatmap log scale and mid point = 1
kaitejohnson Nov 17, 2024
0a28c09
don't exclude the manually excluded ones in retro head to head compar…
kaitejohnson Nov 17, 2024
1ec6022
make metadata heatmap for retro comparison
kaitejohnson Nov 17, 2024
c09407a
add hub model inclusion figure
kaitejohnson Nov 17, 2024
e7f2e59
fix docs
kaitejohnson Nov 17, 2024
e9857d2
add description of time period
kaitejohnson Nov 17, 2024
e50f3eb
swap model to be visualized
kaitejohnson Nov 17, 2024
1f7c623
update description
kaitejohnson Nov 20, 2024
1ff9638
Issue 209: Compute the relative scores in real-time (#210)
kaitejohnson Nov 26, 2024
87c0a26
add functionality to either score draws with crps or quantiled foreca…
kaitejohnson Nov 26, 2024
b597b34
add a function to format the real time hosp only WIS as the ones from…
kaitejohnson Nov 26, 2024
b0ffef8
add a color for the real time hosp only model
kaitejohnson Nov 26, 2024
de173c9
add the real time hosp only wis to pipeline
kaitejohnson Nov 26, 2024
b314a12
make the rel real-time heatmap WIS for consistency
kaitejohnson Nov 27, 2024
9819466
move rel real time heatmap to fig 5
kaitejohnson Nov 27, 2024
00fa697
add requirements for top row of hub comparison figure
kaitejohnson Nov 27, 2024
2c13cd5
get relative wis retro for both models, change colors
kaitejohnson Nov 27, 2024
649c834
create new fig5
kaitejohnson Nov 27, 2024
3eb1eaa
exclude NAs
kaitejohnson Nov 27, 2024
d791d6d
draft fig 5
kaitejohnson Nov 27, 2024
d3d273e
forgot to remove retro from std ranking fig
kaitejohnson Nov 27, 2024
1d29a8a
separately compute mean rel crps and add to density plots
kaitejohnson Nov 29, 2024
ff5bdc1
fix fig 5 to properly compute mena rel crps
kaitejohnson Nov 29, 2024
a53c5c4
add the two different ways of computing the mean rel crps
kaitejohnson Nov 29, 2024
cc74ecb
fix bug
kaitejohnson Nov 29, 2024
51974a2
add a plot of the distribution of relative crps scores
kaitejohnson Nov 29, 2024
974a1c6
add log scale x axis of density plot
kaitejohnson Nov 29, 2024
c7f59cd
add forecasts better to output summary stats
kaitejohnson Nov 29, 2024
8458be5
add functionality to exclude the manual exclusions from metadata summ…
kaitejohnson Nov 29, 2024
037b72e
update targets pipeline with new metadata and convergence exclusions
kaitejohnson Nov 29, 2024
a4d119b
fix bug in summary tables
kaitejohnson Nov 29, 2024
26855d5
tweaks to Fig 5
kaitejohnson Dec 3, 2024
847208e
tweaks to fig3
kaitejohnson Dec 3, 2024
7387f3c
remove limits for now on fig3
kaitejohnson Dec 3, 2024
63e39d2
changes to fig3 to reflect new way of aggregating and computing relat…
kaitejohnson Dec 3, 2024
0c6059c
temporary changes to fig4
kaitejohnson Dec 3, 2024
bcce20c
breakup hub comparison fig into two figs and remove relative heatmap
kaitejohnson Dec 3, 2024
5fb39a4
add new fig4 all time changes
kaitejohnson Dec 3, 2024
6a48e11
make coverage plot horizontal
kaitejohnson Dec 3, 2024
5036380
fix size of supp fig
kaitejohnson Dec 3, 2024
2dac20d
add real time relative figure
kaitejohnson Dec 3, 2024
65c8b37
fix documentation
kaitejohnson Dec 4, 2024
c3b9659
fix docs
kaitejohnson Dec 4, 2024
ffdba08
more doc fixes
kaitejohnson Dec 4, 2024
4e242df
doc fix take 17
kaitejohnson Dec 4, 2024
e9bf2a2
add horizontal line at 1
kaitejohnson Dec 4, 2024
75fe273
replace ww wis with github scores, tweak plots
kaitejohnson Dec 4, 2024
e2083a9
add loc summaries rt relative
kaitejohnson Dec 4, 2024
162c9e6
tweaks to figs and summaries
kaitejohnson Dec 5, 2024
a51c023
add qq plot and coverage range plot for real time
kaitejohnson Dec 5, 2024
7b6db67
fix metadata heatmaps
kaitejohnson Dec 5, 2024
532676d
remove the retro models from the bar chart
kaitejohnson Dec 5, 2024
5a3ef42
save summary metadata ww files
kaitejohnson Dec 9, 2024
40cd8a9
add docs
kaitejohnson Dec 9, 2024
109fdb4
add a formatted figure of C(t)
kaitejohnson Dec 10, 2024
d3d9c13
Merge pull request #212 from CDCgov/211-add-real-time-comp-fig5
kaitejohnson Dec 10, 2024
a813977
reorg to find figures more easily
kaitejohnson Dec 10, 2024
b65d96d
Update wweval/R/combine_and_summarize_ww_data.R
kaitejohnson Dec 12, 2024
077f407
Update wweval/R/score_real_time_outputs.R
kaitejohnson Dec 12, 2024
be3c647
Update wweval/R/score_real_time_outputs.R
kaitejohnson Dec 12, 2024
ba5d4b5
Update wweval/R/score_real_time_outputs.R
kaitejohnson Dec 12, 2024
7fcf6ec
Update wweval/R/supplement_analyses_and_figs.R
kaitejohnson Dec 12, 2024
a63ba57
Update wweval/R/ms_fig4.R
kaitejohnson Dec 12, 2024
3ab2020
Update wweval/R/ms_fig4.R
kaitejohnson Dec 12, 2024
7331a91
Update wweval/R/ms_fig4.R
kaitejohnson Dec 12, 2024
6d530ba
Update wweval/R/ms_fig4.R
kaitejohnson Dec 12, 2024
cffa8fe
Update wweval/R/order_elements.R
kaitejohnson Dec 12, 2024
42ccfa3
Update wweval/R/order_elements.R
kaitejohnson Dec 12, 2024
ecd5697
fix heatmap to relative mean, other tweaks to remove ref to mean rel
kaitejohnson Dec 12, 2024
46db0a7
Merge branch 'hot-fix-full-targets-eval' of https://github.com/CDCgov…
kaitejohnson Dec 12, 2024
a8e1717
update docs
kaitejohnson Dec 12, 2024
8897da8
add bias summary to real time relative
kaitejohnson Dec 12, 2024
10213fd
change way of identifying better and worse forecasts
kaitejohnson Dec 12, 2024
f1b323d
get a summary table of standard ranks
kaitejohnson Dec 12, 2024
767c208
change stats on improved forecasts to fold change
kaitejohnson Dec 12, 2024
ff68f0e
reflect fold change of 4x not 3
kaitejohnson Dec 12, 2024
814b8dc
replace mean_rel w rel_mean for clarity
kaitejohnson Dec 12, 2024
d201133
mean_rel wis to real_mean
dylanhmorris Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
551 changes: 468 additions & 83 deletions _targets_eval_postprocessing.R
View file
Open in desktop

Large diffs are not rendered by default.

42 changes: 32 additions & 10 deletions output/benchmarking/all_forecasts_by_forecast_date.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,45 @@
forecast_date crps_hosp crps_ww bias_hosp bias_ww ae_hosp ae_ww wweval_commit_hash wwinference_version time_stamp
2023-10-16 4.256933535461482 4.677370441094627 -0.020467990890688254 -0.10726857585139318 5.895361948275802 6.40167764415396 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-10-16 5.059832198025414 4.501664311215313 -0.0763335020242915 -0.056744279176201375 7.196846342932589 6.417293680713388 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-10-23 4.18242235006235 4.413074319173952 -0.11766093117408907 -0.11811315789473684 5.872168190751324 5.918167022644099 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-10-30 3.9219506958608275 3.5210321563246176 -0.1053626012145749 -0.08041593567251462 5.393236641297567 4.556872909525013 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-11-06 5.255322819720833 4.242950709531141 -0.2441136133603239 -0.20688696172248805 7.414134578598956 6.0135603392186425 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-11-13 7.012799991126642 4.998084614217982 -0.5118594382591093 -0.46080248205741625 9.64599332435846 6.824715861342708 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-11-20 6.690375498626213 4.569349653386474 -0.41770723684210526 -0.30216682330827066 9.214825658027166 6.337797225183365 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-11-27 5.367627142174053 5.031212394265873 -0.26671799089068826 -0.24200260097919216 7.5482110987083395 7.068444939700455 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-12-04 6.764647750371329 7.420530433101996 -0.31960437753036436 -0.3100663875598086 9.440455766083506 10.504710217364053 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-12-11 6.8594860507920785 5.4574081640829295 -0.16721419534412957 -0.17022117794486216 9.55203634898675 7.785618001707871 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-12-18 8.358376515160765 6.455104681756234 -0.12688347672064776 -0.17103555764411027 11.198803534577065 8.875209211526409 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-12-25 6.894327278046133 7.758221090506614 -0.10081376518218624 -0.013425675675675677 9.34975699525672 10.585508784012148 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-01-01 10.31738321239008 7.901786950597124 0.2344056174089069 0.1605827067669173 14.418033531994073 10.919645060776363 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-01-08 28.7932922854294 23.31310760445013 0.6610759109311741 0.7008788088642659 37.046751992777224 30.793527506859903 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-01-15 19.445812351084225 16.387511402770986 0.7812803643724696 0.7319326923076923 25.995676615059736 22.008691829376154 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-01-22 8.951908471135088 6.350654896121348 0.600787955465587 0.4347277327935223 12.903170731444915 8.784297320600754 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-01-29 5.099562638508678 6.354915264254163 0.08971103238866397 0.09217830634278003 7.255509012912055 8.999088732619368 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-02-05 5.1731132957947406 6.678920578971974 -0.14593003542510122 -0.029273355263157892 7.087254525265564 9.716560898845575 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-02-12 11.222311326853164 11.951289868227024 -0.070354504048583 0.02381831983805668 14.188932611966608 14.992837778794046 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-02-19 7.663909447407364 7.6769418153113 0.3177726467611336 0.3024981203007519 10.733195134321834 10.273287233821865 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-02-26 4.921266785280591 5.603520805103531 0.29640599696356273 0.3353181390977444 7.022260884531381 8.071790643313324 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-03-04 4.233753799775528 4.364753819915278 0.369924721659919 0.34656012851897183 5.919483809471711 6.114155205887852 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2024-03-11 3.357096806226528 3.3860785699505214 0.2889935475708502 0.14207775119617225 4.575087973170657 4.692928674471748 2a70b2b 227-inf-feedback-posterior-mod-eta 2024-11-05T21:00:22Z
2023-10-16 4.256933535461482 4.677370441094627 -0.02046799089068825 -0.10726857585139318 5.895361948275802 6.40167764415396 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-10-23 4.047315226528276 4.631789914889093 -0.09872773279352226 -0.16621130030959752 5.610628735112493 6.210695845766179 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-10-30 4.100770029594852 4.379587942220779 -0.14525657894736843 -0.19440208978328175 5.544465252465204 5.941233543544665 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-10-30 4.100770029594852 4.379587942220779 -0.14525657894736843 -0.19440208978328177 5.544465252465204 5.941233543544665 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-11-06 5.255407521988535 5.932679027358801 -0.29768509615384614 -0.29097226522187825 6.975464949424841 7.864106878051551 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-11-13 6.946577503431648 6.936479202826244 -0.4860530111336032 -0.49226186790505677 8.943972972771006 9.221247117807152 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-11-20 7.759903401522265 5.902333740557295 -0.49563879048583 -0.419578044375645 9.916515197242543 8.0139594342945 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-11-27 7.640750277248922 7.255636035039714 -0.5068352732793522 -0.42566498968008254 10.198548996354523 9.726490371356666 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-12-04 9.021347319079029 9.072298721634764 -0.5661029858299595 -0.5088498452012384 11.773734392455939 11.600043600429995 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-11-13 6.946577503431648 6.936479202826244 -0.4860530111336032 -0.4922618679050568 8.943972972771006 9.221247117807152 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-11-20 7.759903401522265 5.902333740557295 -0.49563879048583 -0.419578044375645 9.916515197242544 8.0139594342945 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-11-27 7.640750277248922 7.255636035039714 -0.5068352732793522 -0.42566498968008254 10.198548996354525 9.726490371356666 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-12-04 9.021347319079029 9.072298721634764 -0.5661029858299595 -0.5088498452012384 11.77373439245594 11.600043600429997 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-12-11 8.417316895980093 8.542035707872587 -0.45781313259109313 -0.42599513157894736 10.822920070978173 11.027416678451043 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-12-18 8.815867884856738 8.349023201732756 -0.36597988360323885 -0.37057855263157896 11.373181687982749 10.851875301875848 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-12-25 7.789253128499376 7.969011598093365 -0.27195976720647774 -0.20117733253588518 10.43430440859026 10.966165282778 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-01-01 7.592317000509569 6.536994643195594 0.03862095141700405 -0.025541353383458646 10.37069031741096 9.032050896441008 d23b28a v0.1.0 2024-10-24T17:48:45Z
2023-12-25 7.789253128499376 7.969011598093365 -0.27195976720647774 -0.2011773325358852 10.43430440859026 10.966165282778 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-01-01 7.592317000509569 6.536994643195594 0.03862095141700405 -0.02554135338345864 10.37069031741096 9.032050896441008 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-01-08 15.72873971884139 17.34649853791809 0.5114973431174089 0.5977302631578947 20.23980158633673 22.760906397690366 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-01-15 16.8368764606006 16.16988800060564 0.7335118927125506 0.7420599581339713 21.470723001323357 21.324997246737848 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-01-22 12.578120503174347 8.591867733854587 0.752081983805668 0.5869108187134503 16.745523991865298 11.962299845196963 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-01-22 12.578120503174349 8.591867733854587 0.752081983805668 0.5869108187134503 16.745523991865298 11.962299845196965 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-01-29 6.250229873276504 5.597408510283156 0.46447431680161944 0.3517020035885168 8.897102295171909 7.708389135822814 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-02-05 4.573424286089247 5.8253084042799985 0.12928175607287448 0.12520877192982455 6.391327797333847 7.939784670173955 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-02-12 4.104713553309723 6.313650246216045 0.11586000504032258 0.16542651255707763 5.819277416265341 8.622847250097367 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-02-19 6.860996913410704 7.869580159555713 0.3967626518218623 0.41254633258678614 9.385290651710555 10.540015575926462 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-02-26 5.701146824066562 7.33035510471486 0.4148002277327935 0.44525109649122807 7.9176729122140515 9.904555497601788 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-03-04 5.112789339042601 4.854152444626718 0.5130581983805668 0.4640028195488722 7.123287289921584 6.752411039873908 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-03-11 4.312835875935859 4.263065993475463 0.48293927125506075 0.3586044575725027 5.945585926634464 5.904156153397102 d23b28a v0.1.0 2024-10-24T17:48:45Z
2024-03-11 4.312835875935859 4.263065993475463 0.4829392712550607 0.3586044575725027 5.945585926634464 5.904156153397102 d23b28a v0.1.0 2024-10-24T17:48:45Z
Loading
Loading