Regularize data fetching methods

[sgsmob]

There are many different ways the indicators fetch remote data (email, AWS S3, cURL, sftp, etc.) and each one does so in a different spot. It would be helpful to have some regularized interface for data fetching (say, an AbstractDataFetcher class from which EmailDataFetcher, etc. inherit) as part of all indicators to consistently and clearly handle data fetching and any associated failures.

[chinandrew]

Was curious and did an inventory of data fetching methods for reference.

• cdc_covidnet: makes API call with requests
• changehc: FTP download with paramiko
• claims_hosp: not 100% sure, but there's pandas calls to filepath variable (EDIT: FTP download with paramiko)
• combo_cases_and_deaths: uses covidcast.signal
• google_symptoms: passes download URL to pandas
• jhu: passes download URL to pandas
• nchs_mortality: uses sodapy to access Socrata API
• quidel and quidel_covidtest: accesses an email address with imap_tools
• safegraph: downloads from s3 with an aws s3 sync+subprocess command
• usafacts: passes download URL to pandas

[krivard]

@mariajahja do you have details on how claims_hosp fetches source data? Is it a separate process that pulls from a magic email address? It might be good to normalize that with how quidel does it.

[mariajahja]

@krivard It’s a separate python script (using paramiko) that pulls from a private Delphi ftp server. HSP deposits them there directly, and a bigchunk machine downloads it locally, runs the sensor update, then deletes the copy.

If it’s helpful, back when HSP used email @korlaxxalrok created a script to pull that data.