ArchiveDiffer vs covidcast API blog #822
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for cmu-delphi-main ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
krivard
reviewed
Jun 26, 2023
| heroImage: ensemble-hero.jpg | ||
| heroImageThumb: ensemble-thumb.jpg | ||
| summary: | | ||
| Recently, the Delphi team discovered some items in USAFacts where the ArchiveDiffer cache (which determines which rows of each day's output actually get added to the database) has lost sync with the API. |
There was a problem hiding this comment.
Suggested change
| Recently, the Delphi team discovered some items in USAFacts where the ArchiveDiffer cache (which determines which rows of each day's output actually get added to the database) has lost sync with the API. | |
| Recently, the Delphi team discovered some items in USAFacts where the ArchiveDiffer cache (which ensures only new rows of data get added to the database) had lost sync with the API. |
| </blockquote> | ||
|
|
||
| ## The Widget | ||
| The widget works as following: |
There was a problem hiding this comment.
could also be "in the following way"
Suggested change
| The widget works as following: | |
| The widget works as follows: |
| ## The Widget | ||
| The widget works as following: | ||
|
|
||
| - Pull each .csv file from ArchiveDiffer cache currently stored in an AWS S3 bucket. |
There was a problem hiding this comment.
Suggested change
| - Pull each .csv file from ArchiveDiffer cache currently stored in an AWS S3 bucket. | |
| - Pull each .csv file from the ArchiveDiffer cache currently stored in an AWS S3 bucket. |
|
|
||
| - Pull each .csv file from ArchiveDiffer cache currently stored in an AWS S3 bucket. | ||
|
|
||
| - Construct the parameters for an API call based on the labels of our .csv on ArchiveDiffer. |
There was a problem hiding this comment.
Suggested change
| - Construct the parameters for an API call based on the labels of our .csv on ArchiveDiffer. | |
| - Construct the parameters for an API call based on the labels of the .csv from ArchiveDiffer. |
|
|
||
|
|
||
| ## The Result | ||
| There exists significant mismatches between data obtained from the ArchiveDiffer cache and covidcast API. |
There was a problem hiding this comment.
Suggested change
| There exists significant mismatches between data obtained from the ArchiveDiffer cache and covidcast API. | |
| There exist significant mismatches between data obtained from the ArchiveDiffer cache and covidcast API. |
|
|
||
| Full-file mismatch happens when the API returns no row at all when looking for data from a file on ArchiveDiffer. Fortunately, USAFacts is the only source with significant number of files where the full file is missing. | ||
|
|
||
| | Sources | Number of full data missing from API | |
There was a problem hiding this comment.
Suggested change
| | Sources | Number of full data missing from API | | |
| | Sources | Number of full files missing from API | |
| | dsew-cpr | 2 | | ||
| | chng | 1 | | ||
|
|
||
| Analysis by date shows that time in mismatched files are pretty evenly distributed. Overall, there is no particular period of time with less/more mismatch than others. |
There was a problem hiding this comment.
Suggested change
| Analysis by date shows that time in mismatched files are pretty evenly distributed. Overall, there is no particular period of time with less/more mismatch than others. | |
| Analysis by date shows that mismatched files are pretty evenly distributed in time. Overall, there is no particular period with less/more mismatches than others. |
|
|
||
| Analysis by date shows that time in mismatched files are pretty evenly distributed. Overall, there is no particular period of time with less/more mismatch than others. | ||
|
|
||
|  |
There was a problem hiding this comment.
Suggested change
|  | |
|  |
|  | ||
|
|
||
| ## Widget Limitations | ||
| This widget cannot tell if a mismatched row exists in S3 but not API or vice versa. It can only show the number of rows that should exists in both storage places but in reality, only exists in one of them. |
There was a problem hiding this comment.
This will make more sense if we explicitly define what a mismatched row means in the above analysis. Maybe as part of the The Widget section?
There was a problem hiding this comment.
Other limitations:
For simplicity and expediency, this widget checks only the most recent version of the data, and ignores all data versioning information. Data versioning support in the Epidata API exists because it matters for modeling purposes _when_ information became available (this is the `issue` parameter in the covidcast endpoint). When we make repairs, Epidata permits us to correct old issues, but S3 does not permit us to correct old uploads. This means that patching typically only touches the API, and leaves ArchiveDiffer data alone even if the data being patched is the most recent version available. This widget does not check the timestamp of S3 uploads for ArchiveDiffer data or the `issue` column of API data. This means that it includes differences due to patches, which we might wish to ignore.
| This widget cannot tell if a mismatched row exists in S3 but not API or vice versa. It can only show the number of rows that should exists in both storage places but in reality, only exists in one of them. | ||
|
|
||
| ## Next steps | ||
| Assuming that the ArchiveDiffer is always correct in comparison to the API content, we should start patching data in the API from the least accurate indicators, indicator-combination and chng. |
There was a problem hiding this comment.
Suggested change
| Assuming that the ArchiveDiffer is always correct in comparison to the API content, we should start patching data in the API from the least accurate indicators, indicator-combination and chng. | |
| We should construct one or more patches which bring the API data and ArchiveDiffer data back into sync. Most changes will modify the API data, since we expect the ArchiveDiffer data to generally be more accurate than the API data, which lies downstream of ArchiveDiffer. We should however identify any differences which are due to pre-existing patches, and ensure that ArchiveDiffer really does represent the most recent figures on all data. | |
| Priority should be given to the least-accurate and most-used indicators: jhu, chng, and indicator-combination. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.