FEAT: Publish BUS Performance files to Tableau

[rymarczy]

This PR is currently sitting on top of #451

I can re-base once #451 is merged into main.

This change reads daily bus_event files from the public S3 bucket, concatenates them together and sends them to Tableau to be used by OPMI. This process happens in two steps:

1. Join daily bus event files together and saved joined parquet file to tableau partition of public bucket.
2. Tableau publisher app picks up tableau partition files from public bucket, converts them to a .hyper file and uploads them to ITD Tableau sever.

Two datasets are created by this process an ALL dataset, consisting of all available bus event files. Along with a RECENT dataset containing only the last 7 days of bus event files. The RECENT dataset will be updated on tableau continuously throughout the day. The ALL dataset will only be updated once per day around 6AM.

Asana Task: https://app.asana.com/0/1189492770004753/1208021735441636

[github-actions]

Coverage of commit f50e52c

Summary coverage rate:
  lines......: 75.5% (2493 of 3304 lines)
  functions..: no data found
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/aws/ecs.py                                                             |27.1%     48|    -     0|    -      0
  src/lamp_py/aws/s3.py                                                              |48.4%    287|    -     0|    -      0
  src/lamp_py/bus_performance_manager/events_gtfs_rt.py                              |79.2%     48|    -     0|    -      0
  src/lamp_py/bus_performance_manager/events_gtfs_schedule.py                        |95.9%     49|    -     0|    -      0
  src/lamp_py/bus_performance_manager/events_tm.py                                   |66.7%     33|    -     0|    -      0
  src/lamp_py/bus_performance_manager/gtfs_utils.py                                  |80.8%     26|    -     0|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_to_parquet.py                             |71.4%     98|    -     0|    -      0
  src/lamp_py/ingestion/compress_gtfs/pq_to_sqlite.py                                |87.8%     49|    -     0|    -      0
  src/lamp_py/ingestion/compress_gtfs/schedule_details.py                            |81.2%     96|    -     0|    -      0
  src/lamp_py/ingestion/config_busloc_trip.py                                        |83.3%     18|    -     0|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |83.3%     18|    -     0|    -      0
  src/lamp_py/ingestion/convert_gtfs.py                                              |79.6%     49|    -     0|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt.py                                           |49.5%    214|    -     0|    -      0
  src/lamp_py/ingestion/converter.py                                                 |88.3%     60|    -     0|    -      0
  src/lamp_py/ingestion/light_rail_gps.py                                            |54.4%     79|    -     0|    -      0
  src/lamp_py/ingestion/utils.py                                                     |61.6%    112|    -     0|    -      0
  src/lamp_py/ingestion_tm/jobs/parition_table.py                                    |45.0%     80|    -     0|    -      0
  src/lamp_py/ingestion_tm/jobs/whole_table.py                                       |75.8%     91|    -     0|    -      0
  src/lamp_py/migrations/versions/performance_manager_prod/sql_strings/strings_001.py| 100%     12|    -     0|    -      0
  src/lamp_py/mssql/mssql_utils.py                                                   |27.5%     69|    -     0|    -      0
  src/lamp_py/performance_manager/alerts.py                                          |53.3%    199|    -     0|    -      0
  src/lamp_py/performance_manager/flat_file.py                                       |82.7%    104|    -     0|    -      0
  src/lamp_py/performance_manager/gtfs_utils.py                                      |91.4%     81|    -     0|    -      0
  src/lamp_py/performance_manager/l0_gtfs_rt_events.py                               |95.6%    135|    -     0|    -      0
  src/lamp_py/performance_manager/l0_gtfs_static_load.py                             |88.1%    135|    -     0|    -      0
  src/lamp_py/performance_manager/l0_gtfs_static_mod.py                              | 100%     33|    -     0|    -      0
  src/lamp_py/performance_manager/l0_rt_trip_updates.py                              |92.9%     70|    -     0|    -      0
  src/lamp_py/performance_manager/l0_rt_vehicle_positions.py                         |98.6%     71|    -     0|    -      0
  src/lamp_py/performance_manager/l1_cte_statements.py                               | 100%     14|    -     0|    -      0
  src/lamp_py/performance_manager/l1_rt_metrics.py                                   | 100%     28|    -     0|    -      0
  src/lamp_py/performance_manager/l1_rt_trips.py                                     | 100%    154|    -     0|    -      0
  src/lamp_py/postgres/postgres_utils.py                                             |76.7%    227|    -     0|    -      0
  src/lamp_py/postgres/rail_performance_manager_schema.py                            | 100%    202|    -     0|    -      0
  src/lamp_py/runtime_utils/alembic_migration.py                                     |94.7%     19|    -     0|    -      0
  src/lamp_py/runtime_utils/remote_files.py                                          | 100%     50|    -     0|    -      0

Download coverage report

[arkadyan]

👍

[github-actions]

Coverage of commit 352e11b

Summary coverage rate:
  lines......: 75.8% (2505 of 3304 lines)
  functions..: no data found
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/runtime_utils/remote_files.py                                          | 100%     50|    -     0|    -      0

Download coverage report