Atomic commitment protocols (ACP) and idempotency

[bonomat]

Some of our protocols make use of a send/receive-ack approach, where each of the party sends over an Ack message saying they are done and waiting for the response Ack from the other party.
This is sometimes not enough. A good read on this problem: https://en.wikipedia.org/wiki/Two_Generals%27_Problem

We might want to look into more concrete protocols which allows distributed systems to reach consensus such as 2-phase commit or other protocol.

While some protocols (2-Phase commit) make use of the actors involved, we can sometimes make use of a third actor: the blockchain (e.g. for opening a position or collaboratively settling one) to find consensus.

With this in mind, we don't have to come up with one solution fits all our protocols and can make use of different approaches, e.g. follow the golden approach of idempotent APIs. For example, rollover is such a call where identical requests should not have a different result.

I opened this discussion so that we can collect ideas and propose how to solve them: e.g. when implementing #1622 we can follow the approach of idepmotency for rollover, for other (create cfd and collab settlement) we can look into other ideas.

Eventually I think we should create ADRs to record our decisions and reasoning.