Census 2020 example

[Azaya89]

Created a new example using the 2020 US census dataset. The file exists locally as a large .parq file that will be uploaded to S3 at a later time.

• minor updates on the 2010 census example - Moved to Update census 2010 example #459

NOTES:

1. The url added in the downloads section of the anaconda-project.yml files is not a real link and that is what is causing the CI build failure.

[maximlt]

I suspect it is due to #429 but I'm not sure how to resolve it.

You need to re-create the conda environment locally following the contributing guide.

The test file added is a 0.1% sample of the full dataset but it is still about 8MB in size. I don't know if that is too large and should be reduced further.

It's still way too large. You should aim for the minimum dataset size possible, it's fine if it's just a few KB as long as it contains data that is representative of the whole dataset. For instance, if the code expects some data category, then it should be in the sample dataset to let the notebook run entirely.

[maximlt]

Is there an absolute need to rename the original census project census_one? Without doing anything else, this is going to break all the links to its web page and deployment.

I would also not call the new one census_two but census2020.

[Azaya89]

Is there an absolute need to rename the original census project census_one? Without doing anything else, this is going to break all the links to its web page and deployment.

I would also not call the new one census_two but census2020.

I imagine renaming the original from census to something else makes sense seeing as there are now more than one census notebooks in the examples gallery (and possibly more in the future). However, I tried renaming both to census2010 and census2020 but the doit validate step emits a warning that only lower case characters and underscore allowed in the naming. I wasn't sure ignoring that warning was ideal that is why I now renamed both to the current names.

[maximlt]

However, I tried renaming both to census2010 and census2020 but the doit validate step emits a warning that only lower case characters and underscore allowed in the naming

Sounds like a bug in the validation code, something like census2020 should be allowed.

[Azaya89]

You need to re-create the conda environment locally following the contributing guide.

Done. Thanks

It's still way too large. You should aim for the minimum dataset size possible, it's fine if it's just a few KB as long as it contains data that is representative of the whole dataset. For instance, if the code expects some data category, then it should be in the sample dataset to let the notebook run entirely.

Reduced it to <1MB now.

[maximlt]

Replying to your comment elsewhere:

Thank you. I'm still in favor of renaming the first one to census2010 though.

If you intend to rename it, then redirect links have to be set up:

• Full link: https://examples.holoviz.org/gallery/census/census.html to https://examples.holoviz.org/gallery/census2010/census2010.html
• Shortcut link: https://examples.holoviz.org/census to https://examples.holoviz.org/census2010
• Unfortunately, it's not super easy to set a redirect link for the deployment itself (https://census-notebook.holoviz-demo.anaconda.com/notebooks/census.ipynb), so renaming would break it. We have recently broken them all (new subdomain) and no one complained as far as I know so it seems it wouldn't be too bad.

Alternatively, we could just:

• Change the title property in the project YAML to Census 2010
• Change the notebook top-level heading to Census 2010

[Azaya89]

Alternatively, we could just:

• Change the title property in the project YAML to Census 2010
• Change the notebook top-level heading to Census 2010

I already did these in this PR. Would that be enough to differentiate both examples eventually?

[maximlt]

Would that be enough to differentiate both examples eventually?

I think so?

[Azaya89]

I think so?

OK. I will revert the other renaming then

[hoxbro]

My suggestion was that you use the processing script to save it to disk as new data and use that data in the notebook.

[Azaya89]

My suggestion was that you use the processing script to save it to disk as new data and use that data in the notebook.

Oh? Alright then. Will do...

[maximlt]

@Azaya89 you will need to re-lock the project as the solve is failing:

Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

PackagesNotFoundError: The following packages are not available from current channels:

  - libcurl==8.11.0=hbbe4b11_0

Not your fault, sometimes conda-forge marks some packages as broken (adding the broken label on conda-forge) which means these packages are no longer available on the conda-forge channel but on conda-forge/label/broken.

conda-forge/admin-requests#1147

[jbednar]

It was hard to follow the discussion above, but it looks like the original one is still called census rather than census2010, and if so, I agree -- let's preserve those links. We'll put a link to census2020 within census so that wherever someone lands they will find both.

Even apart from the file size, the test data seems more complex than necessary. I think you can provide an option to write_parquet to store the test data into a single flat .parq file rather than a directory full of separate part files. Looks like the old census didn't do that, but I don't think there was a good reason for that, as e.g. opensky uses a single parquet file.

[Azaya89]

It was hard to follow the discussion above, but it looks like the original one is still called census rather than census2010, and if so, I agree -- let's preserve those links.

Correct.

We'll put a link to census2020 within census so that wherever someone lands they will find both.

OK. That will require a separate PR then.

Even apart from the file size, the test data seems more complex than necessary. I think you can provide an option to write_parquet to store the test data into a single flat .parq file rather than a directory full of separate part files. Looks like the old census didn't do that, but I don't think there was a good reason for that, as e.g. opensky uses a single parquet file.

OK. I will do that.

[maximlt]

OK. That will require a separate PR then.

I'd make sense doing it in this PR.