Harmony failure after database direct edits and log truncate

[Freelensia]

Our website freelensia.com is still using Harmony for calendar management combined with Sharetribe Go code (github.com/sharetribe/sharetribe)

Last weekend we faced a mass-signup spam attack (we did not have GG capcha for signup), which resulted in 60K new users added to the database. We tried to delete the fake users using a rake task but it did not work so well.

We manually scanned the dumped .sql file and confirmed that the fake users only showed up in:

• people
• community_memberships
• active_sessions
• emails
• user_custom_profile_values

So we used a DELETE method to delete them from each table using Heidi SQL. After a server restart, we had to fix Sidekiq which was still sending emails to those fake users, but all went well.

But Harmony does not work anymore. Saving a listing with a calendar shows "Failed to Connect to Booking Service" error. Here are our Puma error logs.

We tried to restart it from our Monit dashboard. The process shows RUNNING but on the website calendar is not available.

We tried a hard restart with:

sudo su
cd /home/ubuntu/app/harmony
sudo docker-compose up

We got this log: https://pastebin.com/DQ1TkXQN We had a bunch of errors like:)

api_1  |        at clojure.main.main(main.java:37)
api_1  | Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Unknown database 'harmony_db'

or

api_1  | 06:18:48.983 [main] INFO  com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Closed.
api_1  | 06:18:48.991 [main] ERROR harmony.errors - {"component":"error-reporter","event":"uncaught-exception","data":{"msg":"Failed to encode log event data as JSON!"}}
api_1  | clojure.lang.ExceptionInfo: Error in component :db-conn-pool in system com.stuartsierra.component.SystemMap calling #'com.stuartsierra.component/start
api_1  |        at clojure.core$ex_info.invokeStatic(core.clj:4617)

Just FYI we ran out of space on our server, so we emptied the content of some log files such as:

truncate -s 0 /home/ubuntu/app/freelensia/shared/log/sidekiq.log
truncate -s 0 /home/ubuntu/app/freelensia/shared/log/sidekiq.log.1
sudo su
truncate -s 0 /var/lib/docker/containers/c7a8c30d1c26caafc07b6204d2ad5e615a5cf5778bed5d66b784091d5465e5e3/c7a8c30d1c26caafc07b6204d2ad5e615a5cf5778bed5d66b784091d5465e5e3-json.log

These files had 16GB, 1.4GB and 6GB sizes, respectively. We also downsized our EBS by swapping with a smaller EBS volume following this guide. But the EC2 instance and the website works fine after that, so that is probably not the problem.

Anybody with ideas?