Maker does not process ConnectionReady resulting in takers constantly re-connecting

[da-kami]

Maker mainnet, version 0.4.12

Incident happened at: 2022-05-02T11:43:00 UTC and lasted for about 1.5 hours.
Since the maker was not processing messages in maker_inc_connection correctly this was basically an outage.

---

Analysis

• `maker_inc_connections is not blocked completely but only processes very few messages in the timeframe where the incident happened.
• The maker keeps processing new connections because we see Upgrade new connection in the logs
• We don't know why ConnectionReady message is not processed in maker_inc_connections.
• We don't know what triggered this behaviour in the first place

With the current logs it is hard to understand why the code shows the behaviour experienced.

---

Action plan:

• Add timeout to xtra fork where one can configure a handler timeout on Context, should default to no timeout, ideally prepare PR for upstreaming: Make sure error on timeout includes message and actor name via std::any::type_name: Allow handlers to timeout comit-network/xtra#4.
• Apply timeout to entire upgrade function (consider removing inner timeouts?): Configure timeouts for connection actor(s?) #2005.
• Uses bounded mail boxes for all actors: Use bounded mail boxes for xtra::Actors #2006.
• Steal connection on 2nd connection attempt: Steal connection on 2nd connection attempt #2007.
• Write ticket for upstreaming prometheus metrics to xtra: Upstream metrics to xtra comit-network/xtra#3.

[luckysori]

Closing because we have now split this up into separate issues/tasks, as indicated above.