Properly decode UTF-8 from gsheet csv

[melange396]

What i was referring to in #1546 (comment) is not actually a problem with extended ascii or unicode characters -- unicode seems like it should actually be fine to use as long as it is decoded properly...

• In Update Google Docs Meta Data #1546, i saw the replacement of weird ascii characters (that i thought might have been escaped to comply with some CSV format) with what i thought were proper/plain quotation marks (they were actually unicode's stylized quotes).

• db_signals.csv in the current main branch has the weird extended ascii chars (Search for "Children cannot get COVID-19" to see an example:

  delphi-epidata/src/server/endpoints/covidcast_utils/db_signals.csv

  Line 139 in 9e21bfb

  fb-survey,smoothed_wbelief_children_immune,FALSE,smoothed_wbelief_children_immune,FALSE,Belief Children Can't Get COVID-19,FALSE,Estimated percentage of people who believe that the statement �Children cannot get COVID-19� is definitely or probably true.,"Estimated percentage of people who believe that the statement �Children cannot get COVID-19� is definitely or probably true.

  ). In my exploring today, i realized these funky characters also show up on the website ( see https://delphi.cmu.edu/covidcast/indicator-signal/?sensor=fb-survey-smoothed_wbelief_children_immune , screenshot for posterity:
  ).

• Somehow, that broken encoding was fixed in Update Google Docs Meta Data #1546. You will have to take my word for it or carefully examine that version of the file for yourself, the diff from that PR is too big for github to display it nicely.

• You can see that the broken decoding came back in https://github.com/cmu-delphi/delphi-epidata/pull/1547/files#diff-5acd8942e330087af27801ddefd15a2ece4fef9aedcd37ea2e5bcbdb42808bf1R22 after i did a find/replace for some of the quotes. I dont know why we have a mix of low-ascii and higher unicode characters for quotes in the sheet, but maybe google does some "smart" autoformatting for us.

• I still cant explain how they were "fixed" in Update Google Docs Meta Data #1546 but reverted in Update Google Docs Meta Data #1547 ¯\_(ツ)_/¯

• This code demonstrates that it is a bad decoding:

>>> chr(8220) # unicode quote (left double quote, to be specific)
'“'
>>> chr(8220).encode("utf8") # how its represented in utf8 (3 bytes in the extended ascii range)
b'\xe2\x80\x9c'
>>> chr(8220).encode("utf8").decode("8859") # what it looks like when its improperly decoded (an "a-hat" and two effectively unprintable chars)
'â\x80\x9c'

• This S.O. answer essentially explains why it happens with the .text accessor in requests: https://stackoverflow.com/questions/44203397/python-requests-get-returns-improperly-decoded-text-instead-of-utf-8/52615216#52615216

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[melange396]

I ran the CSV file sync GH action using the branch from this PR, so it should be using the associated changes: https://github.com/cmu-delphi/delphi-epidata/actions/runs/11247472116

It produced its own PR that appears to preserve the unicode quotes from #1546: #1549 . This is the desired result, but since i cant currently explain the difference in behavior between #1546 and #1547, i cant rule out that this run could also be a fluke.

The code at the core of the GH action is pretty hokey. For example, it uses a strange procedure to omit lines at the end of the csv that are ~empty, but doesnt explain itself. If it were any more complicated than it is, i would suggest redoing it with csv-specific methods/packages/libraries.

[nmdefries]

This is a reasonable fix. We may eventually want to change the hokey GH action, but that will be a bigger project.

Great job on the investigation and writeup for this!