Use national data for nchs-mortality signals

[rzats]

Description

Currently, only state-specific data is used for any of the data with source=nchs-mortality. As the Covidcast dashboard uses the US as a whole by default, this leads to "N/A" values showing up in plots and numeric text.

This PR makes use of the previously discarded national data for all signals in the NCHS family, pulling it from the dataset rather than throwing it away.

Changelog

• Alters the available geo resolutions for NCHS data from state to ['state', 'nation'].
  • Excludes the percent_of_expected_deaths - despite the name, it contains proportions of expected deaths (e.g. 1.1 instead of 110%), which are nontrivial to aggregate.
• Adds logic to process nation data from the dataset.
• Adds tests for whether data with the new resolutions is correctly created and exported to CSVs.

Fixes

• Fixes Synthesize "national" data for nchs-mortality:deaths_covid_incidence_prop #1906

[melange396]

It turns out that this dataset already has national-level data in it, we have just been throwing it away!

covidcast-indicators/nchs_mortality/delphi_nchs_mortality/pull.py

Line 100 in 66b8ab2

df = df[df["state"] != "United States"]

You can also see samples of this by browsing the dataset at:
https://data.cdc.gov/NCHS/Provisional-COVID-19-Death-Counts-by-Week-Ending-D/r8kw-7aab/data_preview

It is better to use this as-is from the source data than to rebuild it as an aggregation.

[rzats]

@melange396 I've reworked the PR to use the data within the dataset, rather than synthesize new data. Of note:

• Removed one of the new tests, since the sum of state counts no longer adds up to the national count. This is due to some small cells being excluded from the data, as described in the test dataset:

One or more data cells have counts between 1–9 and have been suppressed in accordance with NCHS confidentiality standards.

• Re-included the percent_of_expected_deaths metric, as the data for it no longer needs to be nontrivially synthesized.

[melange396]

nice! we should do a little cleanup on the inner loops of run_module(), and then i think it will be good to go.

[melange396]

you should be able to pull this up two levels (outside of for metric in METRICS:) to reduce repetition

you could even do the filtering on geo_id ==/!= "us" there too, if you want to make another [sub]copy of df_pull

[melange396]

looks good!

[melange396]

nchs-mortality national-level data went in last thursday for the first time...

• Good news: it added old dates (though not old issues, but thats probbly fine), and most signals look good to me.
• Less good news: values for "*_prop" national signals are all null.
  • Unfortunately, the one we were interested in for in the dashboard is a prop signal (from Synthesize "national" data for nchs-mortality:deaths_covid_incidence_prop #1906), and it looks like zeroes in the plots.

Did this work for you in your local testing? Since these values are coming out as "NULL", it makes me think the denominator here is 0 or null, which then makes me think that the national population is not getting set properly here.

Also, to make this message go away, we need to mark the nchs-mortality signals as having national-level data in the spreadsheet and then get it transferred to the csv. We can take care of that after the signals are properly acquired.