Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nssp patching code #2000

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open
Changes from 31 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
c78ae21
nssp patching code
minhkhul Jul 23, 2024
7694c0a
lint
minhkhul Jul 23, 2024
a3ed4c2
add test
minhkhul Jul 24, 2024
1628d34
add test
minhkhul Jul 24, 2024
2536b94
Add patching how-to to readme
minhkhul Jul 24, 2024
e4d45e5
adjust current_issue_dir name for weekly data instead of daily.
minhkhul Jul 25, 2024
db906fc
lint
minhkhul Jul 25, 2024
8f0bb32
adjust test for more cases
minhkhul Jul 25, 2024
7f151f5
add custom_run flag
minhkhul Aug 6, 2024
c093349
handle custom flag on but bad config
minhkhul Aug 6, 2024
a967416
make patch config check readable
minhkhul Aug 6, 2024
c020da6
make good_patch_config check comprehensive
minhkhul Aug 6, 2024
9a6130b
rewrite good_patch_config for clarity
minhkhul Aug 7, 2024
b8a2177
add unit tests for good_patch_config check
minhkhul Aug 7, 2024
a7d9443
add test_pull unit test for patching case + cleanup format
minhkhul Aug 7, 2024
e29e07e
split test cases + move to pytest
minhkhul Aug 8, 2024
0a4bfb6
add test for multi-week patching
minhkhul Aug 9, 2024
6c0abad
rename tests for clarity + restructure test_patch tests to clarify pu…
minhkhul Aug 15, 2024
7078dd0
make expected issue dates explicit in test_patch_confirm_dir_structur…
minhkhul Aug 15, 2024
d435bf0
add log to distinguish grabbing records from socrata vs locally store…
minhkhul Aug 15, 2024
8734daa
Update nssp/README.md
minhkhul Aug 28, 2024
5a6f8b6
Update nssp/README.md
minhkhul Aug 28, 2024
4356494
Add auto-download source backup data + update docs + test
minhkhul Aug 29, 2024
ca427a4
adjust custom_run flag to leave room for non-patch custom runs
minhkhul Aug 30, 2024
f58b068
move pull logic from run.py into pull.py
minhkhul Aug 30, 2024
5e93175
logger to static
minhkhul Aug 30, 2024
2d8670d
adjust unit tests
minhkhul Aug 30, 2024
f0335f6
more unit test adjustment
minhkhul Aug 31, 2024
7e06f94
move get_source_data to pull.py + make get_source_data run when sourc…
minhkhul Sep 3, 2024
e678ce6
auto-remove source_dir content after finish patch run
minhkhul Sep 3, 2024
bc1d7a7
lint happy
minhkhul Sep 3, 2024
84cba84
Update pull.py
minhkhul Sep 10, 2024
742737b
Update pull.py - remove stat debug
minhkhul Sep 10, 2024
e13d3db
add progress log for source file download
minhkhul Sep 10, 2024
9cec6ff
lint
minhkhul Sep 10, 2024
5450d8b
lint
minhkhul Sep 10, 2024
0a8da1e
Merge remote-tracking branch 'origin/main' into nssp_patching
minhkhul Dec 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions nssp/README.md
Original file line number Diff line number Diff line change
@@ -73,3 +73,10 @@ with the percentage of code covered by the tests.

None of the linting or unit tests should fail, and the code lines that are not covered by unit tests should be small and
should not include critical sub-routines.

## Running Patches:
A daily backup of source in the form of csv files can be found on `bigchunk-dev-02.delphi.cmu.edu` under `/common/source_backup/nssp`. This data is needed to create patches. Talk to your sysadmin for access.
When your credentials to the server are working, to create patching data for a specific date range in batch issue format, adjust `params.json` in accordance with instructions in `patch.py`, then run
Comment on lines +83 to +84
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A daily backup of source in the form of csv files can be found on `bigchunk-dev-02.delphi.cmu.edu` under `/common/source_backup/nssp`. This data is needed to create patches. Talk to your sysadmin for access.
When your credentials to the server are working, to create patching data for a specific date range in batch issue format, adjust `params.json` in accordance with instructions in `patch.py`, then run
A daily backup of source data, stored as csv files, can be found on `bigchunk-dev-02.delphi.cmu.edu` under `/common/source_backup/nssp`. This data is needed to create patches since the source does not store issue dates. Talk to your sysadmin for access.
When your credentials to the server are working, to create patching data for a specific date range in batch issue format, adjust `params.json` in accordance with instructions in `patch.py`, then run

```
env/bin/python -m delphi_nssp.patch
```
155 changes: 155 additions & 0 deletions nssp/delphi_nssp/patch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
"""
This module is used for patching data in the delphi_nssp package.

To use this module, you need to turn on the custom_run flag
and specify the range of issue dates in params.json, like so:

{
"common": {
"custom_run": true,
...
},
"validation": {
...
},
"patch": {
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
"source_backup_credentials": {
"host": "bigchunk-dev-02.delphi.cmu.edu",
"user": "user",
"path": "/common/source_backup/nssp"
},
"patch_dir": "/Users/minhkhuele/Desktop/delphi/covidcast-indicators/nssp/AprilPatch",
"start_issue": "2024-04-20",
"end_issue": "2024-04-21",
"source_dir": "/Users/minhkhuele/Desktop/delphi/covidcast-indicators/nssp/source_data"
nmdefries marked this conversation as resolved.
Show resolved Hide resolved
}
}

It will generate data for that range of issue dates, and store them in batch issue format:
[name-of-patch]/issue_[issue-date]/nssp/actual_data_file.csv
"""

import sys
from datetime import datetime, timedelta
from os import listdir, makedirs, path
from shutil import rmtree

from delphi_utils import get_structured_logger, read_params
from epiweeks import Week

from .pull import get_source_data
from .run import run_module


def good_patch_config(params, logger):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can bake this in the read_params() in delphi_utils.... @nmdefries thoughts?
pydantic doesn't a full pledged support for json schema validation, but seeing how the read_params has no validation what so ever, we could look into it sometime in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I wrote this method and half way through was like hmm this should probably be generalized since it may be applicable to other indicators too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeahh I'm like oh I also would like that for mine 👀 and also saw that the current read params isn't doing any validations at all

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, could make sense to add this or a similar fn to delphi_utils. Probably we'd want it and other params validation to be separate fns from read_params, since we don't always want params validation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned it the #2002 and I think for right now it's better to remove this and deal with this in another issue

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@minhkhul it's up to you to keep or remove the good_patch_config function. While we do intend to move it to utils at some later point, we don't know when that will be. Pros of leaving it here are that it is performing the check, and we can use it as a starting point for the future fn. Cons are that we have to remember it's here and use it, and then also replace it with the utils version once that's available.

"""
Check if the params.json file is correctly configured for patching.

params: Dict[str, Any]
Nested dictionary of parameters, typically loaded from params.json file.
logger: Logger object
Logger object to log messages.
"""
valid_config = True
custom_run = params["common"].get("custom_run", False)
if not custom_run:
logger.error("Calling patch.py without custom_run flag set true.")
valid_config = False

patch_config = params.get("patch", {})
if patch_config == {}:
logger.error("Custom flag is on, but patch section is missing.")
valid_config = False
else:
required_patch_keys = ["start_issue", "end_issue", "patch_dir", "source_dir"]
missing_keys = [key for key in required_patch_keys if key not in patch_config]
if missing_keys:
logger.error("Patch section is missing required key(s)", missing_keys=missing_keys)
valid_config = False
else:
try: # issue dates validity check
start_issue = datetime.strptime(patch_config["start_issue"], "%Y-%m-%d")
end_issue = datetime.strptime(patch_config["end_issue"], "%Y-%m-%d")
if start_issue > end_issue:
logger.error("Start issue date is after end issue date.")
valid_config = False
except ValueError:
logger.error("Issue dates must be in YYYY-MM-DD format.")
valid_config = False

if valid_config:
logger.info("Good patch configuration.")
return True
logger.info("Bad patch configuration.")
return False


def patch():
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
"""
Run the nssp indicator for a range of issue dates.

The range of issue dates is specified in params.json using the following keys:
- "patch": Only used for patching data
- "source_backup_credentials": (optional) dict, credentials to log in to
server where historical source data is backed up.
if "source_dir" doesn't exist or has no files in it, we download source data to source_dir before running patch.
else, we assume all needed source files are already in source_dir.
- "host": str, hostname of the server where source data is backed up
- "user": str, username to log in to the server
- "path": str, path to the directory containing backup csv files
- "start_date": str, YYYY-MM-DD format, first issue date
- "end_date": str, YYYY-MM-DD format, last issue date
- "patch_dir": str, directory to write all issues output
- "source_dir": str, directory to read source data from.
"""
params = read_params()
logger = get_structured_logger("delphi_nssp.patch", filename=params["common"]["log_filename"])
if not good_patch_config(params, logger):
sys.exit(1)

source_dir = params["patch"]["source_dir"]
downloaded_source = False
if not path.isdir(source_dir) or not listdir(source_dir):
get_source_data(params, logger)
downloaded_source = True
Comment on lines +112 to +114
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: so we download source data if the source_data dir doesn't exist or is entirely empty. But the downloaded data will be marked as belonging to "today"'s issue, so it may not get used at all in the patch run (if the requested range of issue dates doesn't include "today"). Is that right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, this actually downloads the full source_dir from the server? I saw the definition of get_source_data below.


start_issue = datetime.strptime(params["patch"]["start_issue"], "%Y-%m-%d")
end_issue = datetime.strptime(params["patch"]["end_issue"], "%Y-%m-%d")

logger.info(start_issue=start_issue.strftime("%Y-%m-%d"))
logger.info(end_issue=end_issue.strftime("%Y-%m-%d"))
logger.info(source_dir=source_dir)
logger.info(patch_dir=params["patch"]["patch_dir"])

makedirs(params["patch"]["patch_dir"], exist_ok=True)

current_issue = start_issue
while current_issue <= end_issue:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (optional): I have a slight preference to loop over a date range like dates = pd.date_range(start=params["patch"]["start_issue"], end=params["patch"]["end_issue"]) rather than using a while loop.

logger.info("patching issue", issue_date=current_issue.strftime("%Y-%m-%d"))

current_issue_source_csv = f"""{source_dir}/{current_issue.strftime("%Y-%m-%d")}.csv"""
if not path.isfile(current_issue_source_csv):
logger.info("No source data at this path", current_issue_source_csv=current_issue_source_csv)
current_issue += timedelta(days=1)
continue

params["patch"]["current_issue"] = current_issue.strftime("%Y-%m-%d")

# current_issue_date can be different from params["patch"]["current_issue"]
# due to weekly cadence of nssp data. For weekly sources, issue dates in our
# db matches with first date of epiweek that the reporting date falls in,
# rather than reporting date itself.
current_issue_date = Week.fromdate(current_issue).startdate()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: Since our current_issue while loop goes day-by-day, we will get the same current_issue_date for 7 current_issues in a row (Sunday-Saturday of the week starting with a given current_issue_date). That means that the run_module call below will generate the same data.

This can fixed by keeping track of which current_issue_dates we've seen before (e.g. in a set) and skipping loop iterations/current_issues where the corresponding current_issue_date is in the set.

current_issue_dir = f"""{params["patch"]["patch_dir"]}/issue_{current_issue_date.strftime("%Y%m%d")}/nssp"""
makedirs(f"{current_issue_dir}", exist_ok=True)
params["common"]["export_dir"] = f"""{current_issue_dir}"""

run_module(params, logger)
current_issue += timedelta(days=1)

if downloaded_source:
rmtree(source_dir)
minhkhul marked this conversation as resolved.
Show resolved Hide resolved


if __name__ == "__main__":
patch()
90 changes: 75 additions & 15 deletions nssp/delphi_nssp/pull.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,60 @@
# -*- coding: utf-8 -*-
"""Functions for pulling NSSP ER data."""

import sys
import textwrap
from os import makedirs, path

import pandas as pd
import paramiko
from sodapy import Socrata

from .constants import NEWLINE, SIGNALS, SIGNALS_MAP, TYPE_DICT


def get_source_data(params, logger):
"""
Download historical source data from a backup server.

This function uses 'source_backup_credentials' configuration in params to connect
to a server where backup nssp source data is stored.
It then searches for CSV files that match the inclusive range of issue dates
and location specified by 'path', 'start_issue', and 'end_issue'.
These CSV files are then downloaded and stored in the 'source_dir' directory.
Note: This function is typically used in patching only. Normal runs grab latest data from SODA API.
"""
makedirs(params["patch"]["source_dir"], exist_ok=True)
ssh = paramiko.SSHClient()
Copy link
Contributor

@aysim319 aysim319 Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was wondering how this would work out since we're now saving raw data in /home/indicators/runtime/nssp/raw_backup_files....

I think for next release maybe also change where the raw files are saved; maybe /common/indicator/<data_source>?

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
host = params["patch"]["source_backup_credentials"]["host"]
user = params["patch"]["source_backup_credentials"]["user"]
ssh.connect(host, username=user)

# Generate file names of source files to download
dates = pd.date_range(start=params["patch"]["start_issue"], end=params["patch"]["end_issue"])
csv_file_names = [date.strftime("%Y-%m-%d") + ".csv" for date in dates]

# Download source files
sftp = ssh.open_sftp()
sftp.chdir(params["patch"]["source_backup_credentials"]["path"])
num_files_transferred = 0
for remote_file_name in csv_file_names:
try:
local_file_path = path.join(params["patch"]["source_dir"], remote_file_name)
sftp.stat(remote_file_name)
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
sftp.get(remote_file_name, local_file_path)
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
num_files_transferred += 1
except IOError:
logger.warning(
"Source backup for this date does not exist on the remote server.", missing_filename=remote_file_name
)
sftp.close()
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
ssh.close()

if num_files_transferred == 0:
logger.error("No source data was transferred. Check the source backup server for potential issues.")
sys.exit(1)

def warn_string(df, type_dict):
"""Format the warning string."""
warn = textwrap.dedent(
@@ -27,7 +73,7 @@ def warn_string(df, type_dict):
return warn


def pull_nssp_data(socrata_token: str):
def pull_nssp_data(socrata_token: str, params: dict, logger) -> pd.DataFrame:
"""Pull the latest NSSP ER visits data, and conforms it into a dataset.

The output dataset has:
@@ -39,26 +85,40 @@ def pull_nssp_data(socrata_token: str):
----------
socrata_token: str
My App Token for pulling the NWSS data (could be the same as the nchs data)
test_file: Optional[str]
When not null, name of file from which to read test data
params: dict
Nested dictionary of parameters, should contain info on run type.
logger:
Logger object

Returns
-------
pd.DataFrame
Dataframe as described above.
"""
# Pull data from Socrata API
client = Socrata("data.cdc.gov", socrata_token)
results = []
offset = 0
limit = 50000 # maximum limit allowed by SODA 2.0
while True:
page = client.get("rdmq-nq56", limit=limit, offset=offset)
if not page:
break # exit the loop if no more results
results.extend(page)
offset += limit
df_ervisits = pd.DataFrame.from_records(results)
custom_run = params["common"].get("custom_run", False)
if not custom_run:
# Pull data from Socrata API
client = Socrata("data.cdc.gov", socrata_token)
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
results = []
offset = 0
limit = 50000 # maximum limit allowed by SODA 2.0
while True:
page = client.get("rdmq-nq56", limit=limit, offset=offset)
if not page:
break # exit the loop if no more results
results.extend(page)
offset += limit
df_ervisits = pd.DataFrame.from_records(results)
logger.info("Number of records grabbed from Socrata API", num_records=len(df_ervisits), source="Socrata API")
elif custom_run and logger.name == "delphi_nssp.patch":
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
issue_date = params.get("patch", {}).get("current_issue", None)
source_dir = params.get("patch", {}).get("source_dir", None)
df_ervisits = pd.read_csv(f"{source_dir}/{issue_date}.csv")
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
logger.info(
"Number of records grabbed from source_dir/issue_date.csv",
num_records=len(df_ervisits),
source=f"{source_dir}/{issue_date}.csv",
)
df_ervisits = df_ervisits.rename(columns={"week_end": "timestamp"})
df_ervisits = df_ervisits.rename(columns=SIGNALS_MAP)

23 changes: 13 additions & 10 deletions nssp/delphi_nssp/run.py
Original file line number Diff line number Diff line change
@@ -34,7 +34,6 @@
from .constants import AUXILIARY_COLS, CSV_COLS, GEOS, SIGNALS
from .pull import pull_nssp_data


def add_needed_columns(df, col_names=None):
"""Short util to add expected columns not found in the dataset."""
if col_names is None:
@@ -45,7 +44,6 @@ def add_needed_columns(df, col_names=None):
df = add_default_nancodes(df)
return df


def logging(start_time, run_stats, logger):
"""Boilerplate making logs."""
elapsed_time_in_seconds = round(time.time() - start_time, 2)
@@ -62,7 +60,7 @@ def logging(start_time, run_stats, logger):
)


def run_module(params):
def run_module(params, logger=None):
"""
Run the indicator.

@@ -72,18 +70,23 @@ def run_module(params):
Nested dictionary of parameters.
"""
start_time = time.time()
logger = get_structured_logger(
__name__,
filename=params["common"].get("log_filename"),
log_exceptions=params["common"].get("log_exceptions", True),
)
custom_run = params["common"].get("custom_run", False)
# logger doesn't exist yet means run_module is called from normal indicator run (instead of any custom run)
if not logger:
logger = get_structured_logger(
__name__,
filename=params["common"].get("log_filename"),
log_exceptions=params["common"].get("log_exceptions", True),
)
if custom_run:
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
logger.warning("custom_run flag is on despite direct indicator run call. Normal indicator run continues.")
minhkhul marked this conversation as resolved.
Show resolved Hide resolved
custom_run = False
export_dir = params["common"]["export_dir"]
socrata_token = params["indicator"]["socrata_token"]

run_stats = []
## build the base version of the signal at the most detailed geo level you can get.
## compute stuff here or farm out to another function or file
df_pull = pull_nssp_data(socrata_token)
df_pull = pull_nssp_data(socrata_token, params, logger)
## aggregate
geo_mapper = GeoMapper()
for signal in SIGNALS:
1 change: 1 addition & 0 deletions nssp/setup.py
Original file line number Diff line number Diff line change
@@ -13,6 +13,7 @@
"epiweeks",
"freezegun",
"us",
"paramiko",
]

setup(
4 changes: 4 additions & 0 deletions nssp/tests/source_dir/2021-01-02.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
week_end,geography,county,percent_visits_combined,percent_visits_covid,percent_visits_influenza,percent_visits_rsv,percent_visits_smoothed,percent_visits_smoothed_covid,percent_visits_smoothed_1,percent_visits_smoothed_rsv,ed_trends_covid,ed_trends_influenza,ed_trends_rsv,hsa,hsa_counties,hsa_nci_id,fips,trend_source
2020-10-01T00:00:00.000,United States,All,2.84,1.84,0.48,0.55,2.83,2.07,0.34,0.44,Decreasing,Increasing,Increasing,All,All,All,0,United States
2020-06-29T00:00:00.000,Alabama,All,1.01,0.85,0.17,0.0,0.89,0.66,0.22,0.03,Increasing,Decreasing,No Change,All,All,All,1000,State
2020-02-25T00:00:00.000,Alabama,Blount,,,,,,,,,Data Unavailable,Data Unavailable,Data Unavailable,"Jefferson (Birmingham), AL - Shelby, AL","Bibb, Blount, Chilton, Cullman, Jefferson, Shelby, St. Clair, Walker",150,1009,HSA
4 changes: 4 additions & 0 deletions nssp/tests/source_dir/2021-01-08.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
week_end,geography,county,percent_visits_combined,percent_visits_covid,percent_visits_influenza,percent_visits_rsv,percent_visits_smoothed,percent_visits_smoothed_covid,percent_visits_smoothed_1,percent_visits_smoothed_rsv,ed_trends_covid,ed_trends_influenza,ed_trends_rsv,hsa,hsa_counties,hsa_nci_id,fips,trend_source
2020-10-01T00:00:00.000,United States,All,2.84,1.84,0.48,0.55,2.83,2.07,0.34,0.44,Decreasing,Increasing,Increasing,All,All,All,0,United States
2020-06-29T00:00:00.000,Alabama,All,1.01,0.85,0.17,0.0,0.89,0.66,0.22,0.03,Increasing,Decreasing,No Change,All,All,All,1000,State
2020-02-25T00:00:00.000,Alabama,Blount,,,,,,,,,Data Unavailable,Data Unavailable,Data Unavailable,"Jefferson (Birmingham), AL - Shelby, AL","Bibb, Blount, Chilton, Cullman, Jefferson, Shelby, St. Clair, Walker",150,1009,HSA
4 changes: 4 additions & 0 deletions nssp/tests/source_dir/2021-01-09.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
week_end,geography,county,percent_visits_combined,percent_visits_covid,percent_visits_influenza,percent_visits_rsv,percent_visits_smoothed,percent_visits_smoothed_covid,percent_visits_smoothed_1,percent_visits_smoothed_rsv,ed_trends_covid,ed_trends_influenza,ed_trends_rsv,hsa,hsa_counties,hsa_nci_id,fips,trend_source
2020-10-01T00:00:00.000,United States,All,1,1,1,1,1,1,1,1,Decreasing,Decreasing,Decreasing,All,All,All,0,United States
2020-06-29T00:00:00.000,Oklahoma,All,1.01,0.85,0.17,0.0,0.89,0.66,0.22,0.03,Increasing,Decreasing,No Change,All,All,All,1000,State
2020-02-25T00:00:00.000,Alabama,Blount,,,,,,,,,Data Unavailable,Data Unavailable,Data Unavailable,"Jefferson (Birmingham), AL - Shelby, AL","Bibb, Blount, Chilton, Cullman, Jefferson, Shelby, St. Clair, Walker",150,1009,HSA
4 changes: 4 additions & 0 deletions nssp/tests/source_dir/2021-01-12.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
week_end,geography,county,percent_visits_combined,percent_visits_covid,percent_visits_influenza,percent_visits_rsv,percent_visits_smoothed,percent_visits_smoothed_covid,percent_visits_smoothed_1,percent_visits_smoothed_rsv,ed_trends_covid,ed_trends_influenza,ed_trends_rsv,hsa,hsa_counties,hsa_nci_id,fips,trend_source
2020-10-01T00:00:00.000,United States,All,1,1,1,1,1,1,1,1,Decreasing,Decreasing,Decreasing,All,All,All,0,United States
2020-06-29T00:00:00.000,Oklahoma,All,1.01,0.85,0.17,0.0,0.89,0.66,0.22,0.03,Increasing,Decreasing,No Change,All,All,All,1000,State
2020-02-25T00:00:00.000,Alabama,Blount,,,,,,,,,Data Unavailable,Data Unavailable,Data Unavailable,"Jefferson (Birmingham), AL - Shelby, AL","Bibb, Blount, Chilton, Cullman, Jefferson, Shelby, St. Clair, Walker",150,1009,HSA
Loading
Loading