Newport News Police Incident Scraper

This project allows you to gather and save all the following reports from the City of Newport News Police Department:

Daily Offense Reports:

• Sunday
• Monday
• Tuesday
• Wednesday
• Thursday
• Friday
• Saturday

Daily Arrest Reports:

• Sunday
• Monday
• Tuesday
• Wednesday
• Thursday
• Friday
• Saturday

To configure the project to write to your Firebase database:

1. Download your Firebase JSON service account key, and put it into the same folder as this project.
2. Rename the firebase_config_sample.py file to firebase_config.py.
3. In the firebase_config.py file, change the 'serviceAccountKey.json' filename to match your service key's filename.
4. Change the firebase_database_url to your Firebase database's URL.

To configure the project to automatically geocode the addresses (i.e., get the formatted address, latitude, and longitude):

1. Rename the gcp_config_sample.py file to gcp_config.py.
2. In the gcp_config.py file, change the gcp_api_key API key to your Google Cloud Platform API key.

Running the script

Before running the script, you'll need to install the following as root. This is an example for CentOS:

% yum install python3 python3-devel gcc-c++
% pip3 install BeautifulSoup4
% pip3 install firebase_admin

To run the script:

% python3 police_scrape.py

Optionally, you can run the script only for today's reports. This is useful when automating this to run one or more times a day. There's no need to re-pull and re-process reports from previous days, especially considering geocoding is done for each incident, which uses metered Google Cloud resources.

% python3 police_scrape.py today

To schedule running the script, you can use cron on a Linux server. Here's an example crontab entry that runs the script each hour, on the hour, from 9am to 11pm:

[dev@localhost ~]$ crontab -l
00 09-23 * * * python3 /home/dev/police_scrape/police_scrape.py today >/dev/null 2>&1