Skip to content

Latest commit

 

History

History
174 lines (112 loc) · 5.52 KB

README.md

File metadata and controls

174 lines (112 loc) · 5.52 KB
Raw

👀 Chicago Councilmatic

Chicago Councilmatic tracks all things related to Chicago City Council: the legislation introduced and passed, its various committees and the meetings they hold, and the alders themselves.

You can search and browse legislation from 2011 onwards. Some interesting searches include:

  • changes in how properties are zoned
  • what the City is paying out in settlements
  • committee appointments
  • celebrating birthdays?!

Chicago Councilmatic is a free and easy way to access official Chicago City Council information.

Production data and SQLite archive

If you'd like to explore the data for Chicago Councilmatic yourself, we make a SQLite database using Datasette available here: https://puddle.datamade.us/chicago_council. This data is updated nightly.

The data updates for the production instance of this site is done with GitHub actions in the https://github.com/datamade/chicago-council-scrapers repository. Data is also updated nightly.

Setup

These days, we run apps in containers for local development. Prefer to run the app locally? See the legacy setup instructions.

Install OS level dependencies:

Run the application

docker-compose up

Load in the data

Chicago Councilmatic needs to data work properly and the database is quite large (several GB). You can either restore from another database (faster) or scrape the data yourself.

Option 1: Import data from production

If you have existing data, bring down your existing data & volumes

docker-compose down --volumes

spin up just the postgres container. this will initialize your database

docker-compose up postgres

in a new terminal window, shell into the postgres container

docker exec -it CONTAINER_ID /bin/bash
psql -d postgres -U postgres

in the PSQL interpreter, delete the old database

DROP DATABASE chi_councilmatic;

then, exit out of PSQL and the container. while the postgres container is still running, run the heroku pg:pull command to restore the production data to your local database. This will take about an hour.

PGUSER=postgres PGPASSWORD=postgres PGHOST=127.0.0.1 PGPORT=32001 heroku pg:pull DATABASE_URL chi_councilmatic --app chi-councilmatic-production

Option 2: Scrape directly from the source

The docker-compose-scrape.yml file contains a service to scrape Legistar web API and populate your database with standardized data on the council and its members, legislation, and events. The default command scrapes all committees and people, and any events and legislation updated in the last 30 days or scheduled for a future date.

To import / update everything:

docker-compose -f docker-compose-scrape.yml up

To import / update bills from the past 30 days (window=30):

docker-compose -f docker-compose-scrape.yml run scrapers pupa update --rpm=0 chicago bills window=30

To import / update people:

docker-compose -f docker-compose-scrape.yml run scrapers pupa update --rpm=0 chicago people

To import / update events:

docker-compose -f docker-compose-scrape.yml run scrapers pupa update --rpm=0 chicago events

This may take a few minutes to an hour, depending on the volume of recent updates! Once it's finished, head over to http://localhost:8000 to view your shiny new app!

Optional: Update the search index

If you wish to use search in your local install, add your shiny new data to your search index with the rebuild_index command from Haystack.

docker-compose run --rm app python manage.py update_index --batch-size=100

Set-up pre-commit

Pre-commit hooks are scripts that run on your local machine before every commit.

We use the pre-commit framework to run code linters and formatters that keep our codebase clean.

To set up Pre-Commit, install the Python package on your local machine using

python -m pip install pre-commit

If you'd rather not install pre-commit globally, create and activate a virtual environment in this repo before running the above command.

Then, run

pre-commit install

to set up the git hooks.

Since hooks are run locally, you can modify which scripts are run before each commit by modifying .pre-commit-config.yaml.

Testing

This app provides a basic set of tests that hit all endpoints for the bills, people, events, and organizations in your local database and ensure that nothing breaks. To run them, add some data to your database, as described in Load in the data, then run pytest via docker-compose.

docker-compose -f docker-compose.yml -f tests/docker-compose.yml run --rm app

Team

Original project team

  • David Moore - project manager
  • Forest Gregg - Open Civic Data (OCD) and Legistar scraping
  • Cathy Deng - data models and loading
  • Derek Eder - front end
  • Eric van Zanten - search and dev ops

Maintenance

Chicago Councilmatic is maintained by the DataMade team.

Errors / Bugs

If something is not behaving intuitively, it is a bug, and should be reported. Report it here: https://github.com/datamade/chi-councilmatic/issues

Copyright

Copyright (c) 2015-2023 Participatory Politics Foundation and DataMade. Released under the MIT License.