Alternative policy for dealing with failed actions in process manager

[thomaseizinger]

With #1200, the process manager decouples failures in actions such as TryBroadcastTransaction from the actual processing of the event. The rationale for this behaviour is outlined in ADR011.

Right now, the action is attempted once and in case it fails, an error is logged. This is obviously not ideal. Many of these actions, if not all of them could be re-attempted and might succeed later if, for example, the error is due to IO failure which are often only temporary.

Re-attempting these actions also ties in well with ADR006. An actor tracking a resource such as the blockchain monitor might be temporarily down or not functioning properly and as a result restarted by a supervising actor. Re-attempting an action such as TryBroadcastTransaction would then result in a very resilient system.

A related point to this is one of code duplication. At the moment, individual actors such as monitor or oracle hydrate their own state independently on startup from the database. Conceptually though, this functionality is already present in the process manager. For example, after a contract-setup, the transaction needs to be broadcasted.

Expressed differently, our process manager is currently actually managing multiple processes. Let's take contract setup as an example again. The process that is modeled here could be - in a simplified way - expressed as:

1. Perform contract setup
2. Broadcast transaction
3. Monitor for transactions

If we were to model this process somehow then we should be able unify those concerns:

1. Upon contract setup, we would "start" this process. Throughout its lifetime, this process would try to complete by retrying broadcast and monitor if they failed the first time.
2. Upon startup, we would check if a CFD would have kicked off a process and start the process as well.

Some questions that come to mind:

• Is it safe to always start every process as we go through the events of an existing CFD? Like, should we start the contract-setup process although there might be more events coming for a CFD?
• Contrary to that, would be safe to only kick off the process that would have been started by the last event?
• What would such a process look like? Is it an actor? Is a struct that is owned by the process manager?

[rishflab]

Sorry I just read this after making a new discussion which might be an alternative solution to this issue: #1434

* Is it safe to always start every process as we go through the events of an existing CFD? Like, should we start the contract-setup process although there might be more events coming for a CFD?

Isn't contract setup done outside the process manager. The process manager only deals with setup/failed events.

[thomaseizinger]

You are sort of quoting out of context :)

That question assumes that we have multiple process managers such that for example the entirety of contract setup (start, finish, broadcast, monitor) is governed by a particular process manager.

Checkout this: https://blog.scooletz.com/2014/11/21/process-manager-in-event-sourcing/