[Docs] Document consensus orchestration

consensus/README.md

@@ -140,6 +140,222 @@ graph TB
   J --> note2
 ```
 
+## Synchronization between the consensus processes
+
+The consensus module currently depends on the `PaceMaker`, `StateSync`, `LeaderElection`, `Networking`.
+It has a bootstrapping state where it:
+  * Initializes connections to the network through a bootstrap node
+  * Gets the latest block then switches to one of the mutually exclusive modes: `sync` or `consensus`
+  * Keeps an updated current height (the greatest block height seen on the network)
+
+```mermaid
+sequenceDiagram
+participant Consensus
+participant StateSync
+Note over Node: This embeds Networking, FSM
+participant Node
+participant Network
+Node->>Node:Start
+Node->>Network:Join
+loop
+  Node->>Network:RequestCurrentHeight
+  Network-->>Node: 
+  Node->>Node:SwitchMode
+end
+alt SyncMode
+  par
+    Node->>Consensus:Pause
+  and
+    Node->>StateSync:Start
+    Note over Node,StateSync: Details omitted for another diagram
+    StateSync-->>Node:SyncDone
+  end
+else ConsensusMode
+  par
+    Node->>StateSync:Pause
+  and
+    Node->>Consensus:Start
+    Consensus-->>Node:OutOfSync
+  end
+end
+```
+
+### StateSync Mode
+
+In this mode the node is behind the latest known-to-network height and will try to catchup by downloading then applying the downloaded blocks (sequentially) to its local state.
+
+The `download` and `apply` routines may run in parallel, but `apply` may be blocked by the former if the needed block is missing, it will wait until the needed blocks are downloaded.
+
+#### Download routine
+
+* The `download` routine is alive as long as `sync` mode is on
+* It checks in its persistence for the latest downloaded block and tries to get and add to its persistence all the blocks; up to the network current height
+* After downloading, and before inserting the block, basic (stateless) verification is made to the block
+* A downloaded and inserted block is a structurally valid block but should by no mean considered valid w.r.t. its validators signatures or transactions within
+
+```mermaid
+sequenceDiagram
+participant Persistence
+Note over Node: (Download routine)
+participant Node
+participant Network
+par
+  loop
+  Note left of Network: Constantly asking for the highest<br /> known block from network
+  Node->>Network:UpdateNetWorkCurrentHeight
+  Network-->>Node: 
+  end
+and
+  Node->>Persistence:GetLastDownloadedBlock
+  Persistence-->>Node: 
+  loop LastDownloadedBlock < NetworkCurrentHeight
+    Node->>Network:GetBlock
+    Network-->>Node: 
+    Note left of Network: Currently the node asks all the network<br />for the block it wants to download
+    Node->>Persistence:Append
+    Persistence-->>Node: 
+    Note left of Node: append if new block
+    Node->>Node:Wait(until new block)
+    Note right of Node: If latest network block is reached<br />Wait until it increments to continue
+  end
+end
+```
+
+### Apply routine
+
+_Note: We do not detail how individual transactions are applied or how state is derived from them. Just assume that the state (specifically validator set) may be mutated after each block application._
+
+* The `apply` routine remains alive as long as `sync` mode is on
+* It needs a starting state (genesis, or loaded local state) to start building blocks from
+* Each block is validated and applied before moving to the next one
+* The block application is a sequential process starting from genesis, applies blocks in order, up till network current height
+* Since basic validation is done at download step and assume that the Pocket node trusts its persistence layer, it is safe to skip basic revalidation
+* The `sync` mechanism needs to maintain the chain of trust while applying blocks by performing the following:
+  * Before applying a block `n`, verify that is is signed by a quorum of `n-1`'s validator set (genesis validator set for block `1`)
+  * By applying each block, the validator set is updated (validators joining or leaving), starting from genesis validator set for a new node
+* With this chain of trust, a synching node systematically detects crafted blocks
+  * No malicious or faulty node could inject an alternative block without making at least `2t+1` sign it
+  * The persistence layer is mainly a cache for the block application, so a node won't restart application from genesis each time it's booted
+* When the routine applies `NetworkCurrentHeight`, it signals this so the node could switch to `consensus` mode. Meanwhile, it waits to apply a new downloaded block
+
+_Improvement: After applying a block, the server's signature (the one the block has been downloaded from) is replaced by the current node signature. Meaning that from now on, the current node endorse the block validity (when serving the block to other synchers). It could also serve as means to check the integrity of its own database._
+
+```mermaid
+sequenceDiagram
+participant Persistence
+participant SyncRoutine
+participant BlockApplier
+participant State
+loop AppliedBlock < NetworkCurrentHeight
+  SyncRoutine->>Persistence:GetDownloadedBlock
+  Persistence->>SyncRoutine: 
+  Note right of Persistence: Or genesis if starting from scratch
+  SyncRoutine->>State:GetValidatorSet
+  State->>SyncRoutine: 
+  SyncRoutine->>SyncRoutine:Verify(block.qc, ValidatorSet)
+  SyncRoutine->>BlockApplier:ApplyBlock
+  BlockApplier->>SyncRoutine: 
+  Note left of BlockApplier: Tells if it's valid block what are the  <br />changes to be made to the state
+  SyncRoutine->>State:Update
+  Note left of State: ValidatorSet should be updated here
+  SyncRoutine->>Persistence:MarkBlockAsApplied
+  SyncRoutine->>SyncRoutine:Wait(until block downloaded)
+  Note left of BlockApplier: Wait for the next block to apply<br /> if it's not downloaded yet
+end
+SyncRoutine->>SyncRoutine:SignalSyncEnd
+Note right of SyncRoutine: StateSync has finished<br />Switch to consensus mode
+```
+
+## Consensus Mode
+
+In this mode, the current node is up to date w.r.t. the latest block applied by the network and can start now participating to the consensus process.
+
+This process is driven by:
+* A pace maker, that alerts the consensus flow about key timings in the block production process
+  * `MinBlockTime`: The earliest time a leader node could reap the mempool and start proposing a new block
+  * `RoundTimeout`: How much time a round is allowed to take before calling for another
+  * Pacemaker uses a local clock and a node is only aware of the global pace through new round events
+* A random but deterministic process to elect the leader of each new round
+  * Given a unique random seed known to all validators and information about the current height and round being validated, any validator is able to know who is the leader of a round without the need to communicate with others
+  * The leader election strategy aims to give validators a chance of leading the round proportional to their stake
+  * Fallback to a round robin strategy if probabilistic election doesn't work
+* A consensus flow that aims to increment the height (thus block production) of the chain
+  * See [Block Generation](#block-generation)
+
+### PaceMaker block proposal delaying
+
+The pace maker ensures minimum block time production with the aim to have constant block production pace.
+
+* Adding a delay instead of directly proposing a block makes the the process concurrent.
+  * It ensures that the block proposal is done only once per new round step
+* When the leader gathers enough `NewRound` messages from replicas to propose a block, a first call to build a block is made
+  * The proposal attempt may happen before `MinBlockTime` which the `PaceMaker` will delay.
+  * While delayed, more `NewRound` messages may come-in and the node will use the higher QC obtained to propose the block (discards the previous QC).
+  * If the timer expires before having any block proposal attempt, any call will trigger the block proposal without delay
+  * If a late message is received after a block is proposed by another call, the late message is discarded
+
+#### Concurrent requests
+```mermaid
+sequenceDiagram
+  participant ReapMempool
+  participant BlockPrepare
+  participant Delay
+
+  ReapMempool->>BlockPrepare:RequestPrepare (id=1)
+  BlockPrepare->>+Delay:Start delay
+  Note right of Delay: Wait for minBlockTime
+  ReapMempool->>BlockPrepare:RequestPrepare (id=2)
+  BlockPrepare--xReapMempool:doNotPrepare (id=1)
+  Delay->>-BlockPrepare:End delay
+  BlockPrepare->>ReapMempool:doPrepare (id=2)
+  ReapMempool->>BlockPrepare: Prepare block
+```
+
+#### Late requests
+
+```mermaid
+sequenceDiagram
+  participant ReapMempool
+  participant BlockPrepare
+  participant Delay
+
+  BlockPrepare->>+Delay:Start delay
+  Note right of Delay: Wait for minBlockTime
+  Delay->>-BlockPrepare:End delay
+  ReapMempool->>BlockPrepare:doPrepare (id=1)
+  BlockPrepare->>ReapMempool:RequestPrepare (id=1)
+  ReapMempool->>BlockPrepare: Prepare block
+```
+
+### Example height,round,step increment
+
+| Height | Round | Step | Comment                                  |
+|--------|-------|------|------------------------------------------|
+| 1      | 1     | 0    | Initial round, initial block,            |
+| 1      | 1     | 1    | Enter Prepare step                       |
+| 1      | 1     | 2    | Enter Pre-Commit step                    |
+| 1      | 2     | 0    | Round interrupted, reset step            |
+| 1      | 2     | 1    | Enter Prepare step again                 |
+| 1      | 2     | 2    | Enter Pre-Commit step again              |
+| 1      | 2     | 3    | Enter Commit step for the first time     |
+| 2      | 1     | 0    | Incremented height, reset round and step |
+
+#### Example invalid increments
+
+| Height | Round | Step | Comment                                                        |
+|--------|-------|------|----------------------------------------------------------------|
+| 1      | 1     | 2    | Enter Pre-Commit step                                          |
+| 1      | 2     | 2    | **Invalid**: If `Round` increments, `Step` has to reset to `0` |
+
+| Height | Round | Step | Comment                                                            |
+|--------|-------|------|--------------------------------------------------------------------|
+| 1      | 1     | 2    | Enter Pre-Commit step                                              |
+| 2      | 0     | 0    | **Invalid**: `Height` only increments when previous Step is at `3` |
+
+| Height | Round | Step | Comment                                                                       |
+|--------|-------|------|-------------------------------------------------------------------------------|
+| 1      | 2     | 3    | Enter Commit step                                                             |
+| 2      | 2     | 3    | **Invalid**: When `Height` increments, `Round` to `1` and `Step` reset to `0` |
 
 ## Implementation