[Docs] upgrade/chain halt recovery

Dockerfile.release

@@ -8,7 +8,6 @@ RUN apt-get update && \
     apt-get install -y --no-install-recommends ca-certificates && \
     rm -rf /var/lib/apt/lists/*
 
-
 # Use `1025` G/UID so users can switch between this and `heighliner` image without a need to chown the files.
 RUN groupadd -g 1025 pocket && useradd -u 1025 -g pocket -m -s /sbin/nologin pocket
 

app/upgrades/historical.go

@@ -11,6 +11,7 @@ package upgrades
 
 import (
 	"context"
+	"fmt"
 
 	storetypes "cosmossdk.io/store/types"
 	upgradetypes "cosmossdk.io/x/upgrade/types"
@@ -29,6 +30,7 @@ func defaultUpgradeHandler(
 	configurator module.Configurator,
 ) upgradetypes.UpgradeHandler {
 	return func(ctx context.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
+		fmt.Println("Starting the migration in defaultUpgradeHandler.")
 		return mm.RunMigrations(ctx, configurator, vm)
 	}
 }
@@ -87,3 +89,10 @@ var Upgrade_0_0_4 = Upgrade{
 	// No changes to the KVStore in this upgrade.
 	StoreUpgrades: storetypes.StoreUpgrades{},
 }
+
+// Upgrade_0_0_9 is a small upgrade on TestNet.
+var Upgrade_0_0_9 = Upgrade{
+	PlanName:             "v0.0.9",
+	CreateUpgradeHandler: defaultUpgradeHandler,
+	StoreUpgrades:        storetypes.StoreUpgrades{},
+}

docusaurus/docs/develop/developer_guide/chain_halt_troubleshooting.md

@@ -8,13 +8,15 @@ title: Chain Halt Troubleshooting
 - [Understanding Chain Halts](#understanding-chain-halts)
   - [Definition and Causes](#definition-and-causes)
   - [Impact on Network](#impact-on-network)
-- [Troubleshooting Process](#troubleshooting-process)
+- [Troubleshooting `wrong Block.Header.AppHash`](#troubleshooting-wrong-blockheaderapphash)
   - [Step 1: Identifying the Issue](#step-1-identifying-the-issue)
   - [Step 2: Collecting Node Data](#step-2-collecting-node-data)
   - [Step 3: Analyzing Discrepancies](#step-3-analyzing-discrepancies)
   - [Step 4: Decoding and Interpreting Data](#step-4-decoding-and-interpreting-data)
   - [Step 5: Comparing Records](#step-5-comparing-records)
   - [Step 6: Investigation and Resolution](#step-6-investigation-and-resolution)
+- [Troubleshooting `wrong Block.Header.LastResultsHash`](#troubleshooting-wrong-blockheaderlastresultshash)
+- [Syncing from genesis](#syncing-from-genesis)
 
 ## Understanding Chain Halts
 
@@ -40,7 +42,7 @@ Chain halts can have severe consequences for the network:
 
 Given these impacts, swift and effective troubleshooting is crucial to maintain network health and user trust.
 
-## Troubleshooting Process
+## Troubleshooting `wrong Block.Header.AppHash`
 
 ### Step 1: Identifying the Issue
 
@@ -94,3 +96,20 @@ Based on the identified discrepancies:
 2. Develop a fix or patch to address the issue.
 3. If necessary, initiate discussions with the validator community to reach social consensus on how to proceed.
 4. Implement the agreed-upon solution and monitor the network closely during and after the fix.
+
+## Troubleshooting `wrong Block.Header.LastResultsHash`
+
+Errors like the following can occur from using the incorrect binary version at a certain height.
+
+```bash
+reactor validation error: wrong Block.Header.LastResultsHash.
+```
+
+The solution is to use the correct binary version to sync the full node at the correct height.
+
+Tools like [cosmosvisor](https://docs.cosmos.network/v0.45/run-node/cosmovisor.html) make it easier
+to sync a node from genesis, using the appropriate binary for each range of block heights.
+
+## Syncing from genesis
+
+If you're encountering any of the errors mentioned above while trying to sync the historical blocks - make sure you're running the correct version of the binary in accordance with this table [Upgrade List](../../protocol/upgrades/upgrade_list.md).

docusaurus/docs/develop/developer_guide/recovery_from_chain_halt.md

@@ -0,0 +1,173 @@
+---
+sidebar_position: 7
+title: Chain Halt Recovery
+---
+
+## Chain Halt Recovery <!-- omit in toc -->
+
+This document describes how to recover from a chain halt. It assumes the cause of
+the chain halt has been identified, the new release has been created, and verified
+function correctly.
+
+:::tip
+See [Chain Halt Troubleshooting](./chain_halt_troubleshooting.md) for more information on identifying the cause of a chain halt.
+:::
+
+- [Background](#background)
+- [Resolving halts during a network upgrade](#resolving-halts-during-a-network-upgrade)
+  - [Manual binary replacement (preferred)](#manual-binary-replacement-preferred)
+  - [Rollback, fork and upgrade](#rollback-fork-and-upgrade)
+    - [Step 5: Data rollback - retrieving snapshot at a specific height](#step-5-data-rollback---retrieving-snapshot-at-a-specific-height)
+    - [Step 6: Validator Isolation - risks](#step-6-validator-isolation---risks)
+
+## Background
+
+Pocket network is built on top of `cosmos-sdk`, which utilizes the CometBFT consensus engine.
+Byzantine Fault Tolerant (BFT) consensus algorithm requires that **at least** 2/3 of Validators
+are online and voting for the same block to reach a consensus. In order to maintain liveness
+and avoid a chain-halt, we need the majority (> 2/3) of Validators to participate
+and use the same version of the software.
+
+## Resolving halts during a network upgrade
+
+If the halt is caused by the network upgrade, it is possible the solution can be as simple as
+skipping an upgrade (i.e. `unsafe-skip-upgrade`) and creating a new (fixed) upgrade.
+
+Read more about [upgrade contingency plans](../../protocol/upgrades/contigency_plans.md).
+
+### Manual binary replacement (preferred)
+
+:::note
+
+This is the preferred way of resolving consensus-breaking issues.
+
+**Significant side effect**: this breaks an ability to sync from genesis **without manual interventions**.
+For example, when a consensus-breaking issue occurs on a node that is synching from the first block, node operators need
+to manually replace the binary with the new one. There are efforts underway to mitigate this issue, including
+configuration for `cosmovisor` that could automate the process.
+
+<!-- TODO_IMPROVE(@okdas): Add links to Cosmovisor documentation on how the new UX can be used to automate syncing from genesis without human input. -->
+
+:::
+
+Since the chain is not moving, **it is impossible** to issue an automatic upgrade with an upgrade plan. Instead,
+we need **social consensus** to manually replace the binary and get the chain moving. 
+
+1. Prepare and verify a new binary that addresses the consensus-breaking issue.
+2. Reach out to the community and validators so they can upgrade the binary manually. 
+   :::warning UNKNOWN, NEED TO INVESTIGATE
+   <!-- TODO_INVESTIGATE(@okdas): check if need to sync rounds/steps -->
+   We might need to coordinate the timing of when the nodes should be started. In Tendermint version of Pocket Network
+   (Morse), this was necessary to sync consensus rounds and steps, getting the chain moving. It might not be a
+   requirement anymore, but we need to double-check. [More information](https://docs.cometbft.com/v1.0/spec/consensus/consensus).
+3. Update [the documentation](../../protocol/upgrades/upgrade_list.md) to include a range a height when the binary needs
+   to be repleced. Consider a configuration change for `cosmovisor` so it would automatically replace the binary when
+   synching from genesis. <!-- TODO_IMPROVE(@okdas): Investigate and add Cosmovisor documentation. -->
+
+
+```mermaid
+sequenceDiagram
+    participant DevTeam
+    participant Community
+    participant Validators
+    participant Documentation
+    participant Network
+
+    DevTeam->>DevTeam: Prepare and verify new binary
+    DevTeam->>Community: Announce new binary and instructions
+    DevTeam->>Validators: Notify validators to upgrade manually
+    Validators->>Validators: Manually replace the binary
+    Validators->>Network: Restart nodes with new binary
+    DevTeam->>Documentation: Update upgrade documentation
+    Validators->>Network: Network resumes operation
+
+```
+
+### Rollback, fork and upgrade
+
+:::info
+
+These instructions are only relevant to Pocket Network's Shannon release.
+
+We do not currently use `x/gov` and on-chain voting for upgrades.
+
+Instead, our DAO votes on upgrades off-chain and the Foundation executes
+transactions on their behalf.
+
+:::
+
+**Performing a rollback is analogous to forking the network at the older height.**
+
+This should be avoided unless absolutely necessary.
+
+However, if necessary, the instructions to follow are:
+
+1. Prepare & verify a new binary that addresses the consensus-breaking issue.
+2. [Create a release](../../protocol/upgrades/release_process.md).
+3. [Prepare an upgrade transaction](../../protocol/upgrades/upgrade_procedure.md#writing-an-upgrade-transaction) to the new version.
+4. Get the Validator set off the network **3 blocks** prior to the height of the chain halt. For example:
+   - Assume an issue at height `103`
+   - Get the validator set at height `100`
+   - Submit an upgrade transaction at `101`
+   - Upgrade the chain at height `102`
+   - Avoid the issue at height `103`
+5. Ensure all validators rolled back to the same height and use the same snapshot - ([how to get the snapshot](#step-5-data-rollback---retrieving-snapshot-at-a-specific-height))
+   - The snapshot should be imported into each Validator's data directory
+   - This is necessary to ensure data continuity and prevent forks.
+6. Isolate the validator set from full nodes - ([why this is necessary](#step-6-validator-isolation---risks))
+   - This is necessary to avoid full nodes from gossiping blocks that have been rolled back.
+   - This may require using a firewall or a private network
+   - Validators should only be gossip blocks amongst themselves.
+7. Start the network and perform the upgrade. For example, reiterating the process above:
+   - Start all Validators at height `100`
+   - On block `101`, submit the `MsgSoftwareUpgrade` transaction with a `Plan.height` set to `102`.
+   - `x/upgrade` will perform the upgrade in the `EndBlocker` of block `102`
+   - If using `cosmosvisor`, the node will wait to replace the binary
+8. Wait for the network to reach the height of the previous ledger (`104`+)
+9. Allow validators to open their network to full nodes again.
+   - Note that full nodes will need to perform the rollback or use a snapshot as well.
+
+```mermaid
+sequenceDiagram
+    participant DevTeam
+    participant Foundation
+    participant Validators
+    participant FullNodes
+    participant Network
+
+    DevTeam->>DevTeam: Prepare & verify new binary
+    DevTeam->>DevTeam: Create a release
+    Validators->>Validators: Roll back to height before issue or import snapshot
+    Validators->>Validators: Isolate from Full Nodes
+    Foundation->>Validators: Distribute upgrade transaction
+    Validators->>Network: Start network and perform upgrade
+    Validators->>Network: Wait until over consensus-breaking height
+    Validators->>FullNodes: Open network connections
+    FullNodes->>Network: Sync with updated network
+    Validators->>Network: Network resumes operation
+
+```
+
+#### Step 5: Data rollback - retrieving snapshot at a specific height
+
+There are two ways to get a snapshot from a prior height:
+
+1. Execute
+   ```bash
+   poktrolld rollback --hard
+   ```
+   repeately until the command responds with the desired block number.
+2. Use a snapshot and start the node with `--halt-height=100` parameter so it only syncs up to certain height and then
+   gracefully shuts down. Add this argument to `poktrolld start` like this:
+   ```bash
+   poktrolld start --halt-height=100
+   ```
+
+
+#### Step 6: Validator Isolation - risks
+
+Having at least one node that has knowledge of the forking ledger can jeopardize the whole process. In particular, the
+following errors in logs are the sign of the nodes populating existing blocks:
+  - `found conflicting vote from ourselves; did you unsafe_reset a validator?`
+  - `conflicting votes from validator`
+

docusaurus/docs/protocol/upgrades/contigency_plans.md

@@ -0,0 +1,81 @@
+---
+title: Failed upgrade contingency plan
+sidebar_position: 5
+---
+
+:::tip
+
+This documentation covers failed upgrade contingency for `poktroll` - a `cosmos-sdk` based chain.
+
+While this can be helpful for other blockchain networks, it is not guaranteed to work for other chains.
+
+:::
+
+## Contingency plans <!-- omit in toc -->
+
+There's always a chance the upgrade will fail.
+
+This document is intended to help you recover without significant downtime.
+
+- [Option 0: The bug is discovered before the upgrade height is reached](#option-0-the-bug-is-discovered-before-the-upgrade-height-is-reached)
+- [Option 1: The upgrade height is reached and the migration didn't start](#option-1-the-upgrade-height-is-reached-and-the-migration-didnt-start)
+- [Option 2: The migration is stuck](#option-2-the-migration-is-stuck)
+- [Option 3: The network is stuck at the future height after the upgrade](#option-3-the-network-is-stuck-at-the-future-height-after-the-upgrade)
+- [Documentation and scripts to update](#documentation-and-scripts-to-update)
+
+### Option 0: The bug is discovered before the upgrade height is reached
+
+**Cancel the upgrade plan!**
+
+See the instructions of [how to do that here](./upgrade_procedure.md#cancelling-the-upgrade-plan).
+
+### Option 1: The upgrade height is reached and the migration didn't start
+
+If the nodes on the network stopped at the upgrade height and the migration did not
+start yet (i.e. there are no logs indicating the upgrade handler and store migrations are being executed),
+we mist gather social consensus to restart validators with the `--unsafe-skip-upgrade=$upgradeHeightNumber` flag.
+
+This will skip the upgrade process, allowing the chain to continue and the protocol team to plan another release.
+
+`--unsafe-skip-upgrade` simply skips the upgrade handler and store migrations.
+The chain continues as if the upgrade plan was never set.
+The upgrade needs to be fixed, and then a new plan needs to be submitted to the network.
+
+:::caution
+
+`--unsafe-skip-upgrade` needs to be documented in the list of upgrades and added to the scripts so the next time somebody tries to sync the network from genesis - they will automatically skip the failed upgrade. [Documentation and scripts to update](#documentation-and-scripts-to-update)
+
+<!-- TODO_IMPROVE(@okdas): new cosmovisor UX can simplify this -->
+
+:::
+
+### Option 2: The migration is stuck
+
+If the migration is stuck, there's always a chance the state has been mutated for
+the upgrade but the migration didn't complete.
+
+In such a case, we need to:
+
+- Roll back validators to the backup. A snapshot is taken by `cosmovisor` automatically prior to upgrade when`UNSAFE_SKIP_BACKUP` is set to `false` (which is a default and recommended value -
+  [more information](https://docs.cosmos.network/main/build/tooling/cosmovisor#command-line-arguments-and-environment-variables)).
+- All full nodes and validators on the network: skip the upgrade by adding `--unsafe-skip-upgrade=$upgradeHeightNumber`
+  argument to your `poktroll start` command. Like this:
+  ```bash
+  poktrolld start --unsafe-skip-upgrade=$upgradeHeightNumber # ... the rest of the arguments
+  ```
+- Protocol team: document and add `--unsafe-skip-upgrade=$upgradeHeightNumber` to the scripts (such as docker-compose and cosmovisor installer) so the next time somebody
+  tries to sync the network from genesis they will automatically skip the failed upgrade. [Documentation and scripts to update](#documentation-and-scripts-to-update)
+- Resolve the issue with an upgrade and schedule another plan.
+
+<!-- TODO_IMPROVE(@okdas): new cosmovisor UX can simplify this -->
+
+### Option 3: The network is stuck at the future height after the upgrade
+
+This should be treated as a consensus or non-determinism bug that is unrelated to the upgrade. See [Recovery From Chain Halt](../../develop/developer_guide/recovery_from_chain_halt.md) for more information on how to handle such issues.
+
+### Documentation and scripts to update
+
+- The [upgrade list](./upgrade_list.md) should reflect a failed upgrade and provide a range of heights that served by each version.
+- Systemd service should include`--unsafe-skip-upgrade=$upgradeHeightNumber` argument in its start command [here](https://github.com/pokt-network/poktroll/blob/main/tools/installer/full-node.sh).
+- [Helm chart](https://github.com/pokt-network/helm-charts/blob/main/charts/poktrolld/templates/StatefulSet.yaml) (consider exposing via a `values.yaml` file)
+- [docker-compose](https://github.com/pokt-network/poktroll-docker-compose-example/tree/main/scripts) example 

docusaurus/docs/protocol/upgrades/release_process.md

@@ -16,13 +16,6 @@ sidebar_position: 4
 This document is for the Pocket Network protocol team's internal use only.
 :::
 
-- [1. Determine if the Release is Consensus-Breaking](#1-determine-if-the-release-is-consensus-breaking)
-- [2. Create a GitHub Release](#2-create-a-github-release)
-  - [Legend](#legend)
-- [3. Write an Upgrade Plan](#3-write-an-upgrade-plan)
-- [4. Issue Upgrade on TestNet](#4-issue-upgrade-on-testnet)
-- [5. Issue Upgrade on MainNet](#5-issue-upgrade-on-mainnet)
-
 ### 1. Determine if the Release is Consensus-Breaking
 
 :::note
@@ -59,12 +52,18 @@ You can find an example [here](https://github.com/pokt-network/poktroll/releases
 ```text
 ## Protocol Upgrades
 
+<!--
+IMPORTANT:If this release will be used to issue upgrade on the network, add a link to the upgrade code
+such as https://github.com/pokt-network/poktroll/blob/main/app/upgrades/historical.go#L51.
+-->
+
 - **Planned Upgrade:** ❌ Not applicable for this release.
 - **Breaking Change:** ❌ Not applicable for this release.
 - **Manual Intervention Required:** ✅ Yes, but only for Alpha TestNet participants. If you are participating, please follow the [instructions provided here](https://dev.poktroll.com/operate/quickstart/docker_compose_walkthrough#restarting-a-full-node-after-re-genesis-) for restarting your full node after re-genesis.
 - **Upgrade Height:** ❌ Not applicable for this release.
 
 ## What's Changed
+
 <!-- GitHub Release Notes continue here -->
 ```