Raw Census models

[gsa-suk]

Create models for historic raw census tables.

Tasks

Preview Give feedback

1. Create a new S3 bucket and PG database #2587
   +0
   
2. Create a zip file suitable for use in a local environment #2575
   +0
   
3. Create a data model to move the contents of the files into
4. Create and test code that populates the raw PG models from zip to S3 to PG #2588
   +0
   

   
5. Document initial preprocessing for raw census data #2646
   +0
   

[gsa-suk]

• Use the ELECAUDITHEADER AND ELECAUDIT models from https://github.com/GSA-TTS/FAC/blob/jmm/history_data_etl/backend/historic/models.py.
• Add the other models following the Census schema in https://github.com/GSA-TTS/FAC/blob/main/backend/dissemination/census/FACDissemSchema.sql

[JeanMarie-PM]

@gsa-suk I think we need to make a fresh start.
Here's what I propose for the creating the raw census models:

• Create a new S3 bucket to store CSV files from Census (could be Oracle dumpe later on)
• Create a new Postgres instance to house the Census data in the RDBMS with minimal changes (i.e. tables named ELECAUDITHEADER, etc).
• Create a Python package (non-djanago) at the same level as backend. Let's call it Census2FAC or something similar. This package should have all the code to movethe csv files to the new S3 bucket, the code to populate the new database from the csv files, the code to validate pdf checksums, etc. We should be able to drop this package from our codebase at a future point in time.
• Once this is done, we can have django use the new Posgres database to populate SF-SAC and the dissemination tables - either directly or clones of these.