-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Docs][Misc] Add iavl tree diff (#776)
## Summary Adds a tool for troubleshooting divergent states between two data dirs. --------- Co-authored-by: Daniel Olshansky <[email protected]>
- Loading branch information
Showing
7 changed files
with
515 additions
and
0 deletions.
There are no files selected for viewing
96 changes: 96 additions & 0 deletions
96
docusaurus/docs/develop/developer_guide/chain_halt_troubleshooting.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| --- | ||
| sidebar_position: 6 | ||
| title: Chain Halt Troubleshooting | ||
| --- | ||
|
|
||
| ## Chain Halt Troubleshooting <!-- omit in toc --> | ||
|
|
||
| - [Understanding Chain Halts](#understanding-chain-halts) | ||
| - [Definition and Causes](#definition-and-causes) | ||
| - [Impact on Network](#impact-on-network) | ||
| - [Troubleshooting Process](#troubleshooting-process) | ||
| - [Step 1: Identifying the Issue](#step-1-identifying-the-issue) | ||
| - [Step 2: Collecting Node Data](#step-2-collecting-node-data) | ||
| - [Step 3: Analyzing Discrepancies](#step-3-analyzing-discrepancies) | ||
| - [Step 4: Decoding and Interpreting Data](#step-4-decoding-and-interpreting-data) | ||
| - [Step 5: Comparing Records](#step-5-comparing-records) | ||
| - [Step 6: Investigation and Resolution](#step-6-investigation-and-resolution) | ||
|
|
||
| ## Understanding Chain Halts | ||
|
|
||
| ### Definition and Causes | ||
|
|
||
| Chain halts are critical events in blockchain networks that occur when validators fail to reach consensus. In our Cosmos SDK-based chain, consensus requires agreement from at least 67% of the voting power to produce new blocks. | ||
|
|
||
| These halts typically arise from non-deterministic scenarios, where different validators have conflicting views of the blockchain state. Common causes include: | ||
|
|
||
| 1. Inconsistent application of state transition rules | ||
| 2. Divergent interpretations of transaction data | ||
| 3. Synchronization issues between nodes | ||
| 4. Software bugs leading to inconsistent state calculations | ||
|
|
||
| ### Impact on Network | ||
|
|
||
| Chain halts can have severe consequences for the network: | ||
|
|
||
| - Transaction processing comes to a standstill | ||
| - Users cannot interact with the blockchain | ||
| - Potential loss of confidence in the network's reliability | ||
| - Financial implications for stakeholders and users | ||
|
|
||
| Given these impacts, swift and effective troubleshooting is crucial to maintain network health and user trust. | ||
|
|
||
| ## Troubleshooting Process | ||
|
|
||
| ### Step 1: Identifying the Issue | ||
|
|
||
| When a chain halt occurs, the first step is to identify the specific error message or symptom. A common indicator is the `wrong Block.Header.AppHash` error, which suggests a mismatch in the calculated state between nodes. | ||
|
|
||
| ### Step 2: Collecting Node Data | ||
|
|
||
| To investigate the discrepancy: | ||
|
|
||
| 1. Locate the database directory on the affected node (typically `$HOME/.poktroll/data`). | ||
| 2. Obtain the same directory from a healthy full node or one with a different `Block.Header.AppHash`. | ||
| 3. Ensure you have appropriate permissions to access and copy this data. | ||
|
|
||
| ### Step 3: Analyzing Discrepancies | ||
|
|
||
| Utilize the [iavl-tree-diff](https://github.com/pokt-network/poktroll/tree/main/tools/iavl-tree-diff) tool to compare the two database snapshots: | ||
|
|
||
| 1. Clone the repository containing the tool. | ||
| 2. Follow the tool's documentation to set up and run the comparison. | ||
| 3. The tool will highlight specific areas where the two nodes' states diverge. | ||
|
|
||
| ### Step 4: Decoding and Interpreting Data | ||
|
|
||
| The blockchain data is stored in `protobuf` format, encoded as hexadecimal. | ||
|
|
||
| To interpret this data: | ||
|
|
||
| 1. Use [CyberChef](<https://gchq.github.io/CyberChef/#recipe=From_Hex('Auto')Protobuf_Decode('',false,false)>): a versatile data manipulation tool. | ||
| 2. Input the hexadecimal data into CyberChef. | ||
| 3. Apply the "From Hex" operation followed by "Protobuf Decode" to reveal the human-readable content. | ||
|
|
||
|  | ||
|
|
||
| ### Step 5: Comparing Records | ||
|
|
||
| After decoding, compare the data from both nodes: | ||
|
|
||
| 1. Use a diff tool (CyberChef can also be used for this purpose). | ||
| 2. Identify specific fields or values that differ between the two records. | ||
| 3. Pay close attention to timestamps, numerical values, and complex data structures. | ||
|
|
||
|  | ||
|
|
||
| The image above illustrates a difference in the JSON representation of an object, which is likely the root cause of the non-deterministic state breaking consensus between nodes. | ||
|
|
||
| ### Step 6: Investigation and Resolution | ||
|
|
||
| Based on the identified discrepancies: | ||
|
|
||
| 1. Investigate the underlying cause of the difference (e.g., race condition, improper state management). | ||
| 2. Develop a fix or patch to address the issue. | ||
| 3. If necessary, initiate discussions with the validator community to reach social consensus on how to proceed. | ||
| 4. Implement the agreed-upon solution and monitor the network closely during and after the fix. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # iavl-tree-diff <!-- omit in toc --> | ||
|
|
||
| - [Features](#features) | ||
| - [How It Works](#how-it-works) | ||
| - [Usage](#usage) | ||
| - [Example Output](#example-output) | ||
|
|
||
| **iavl-tree-diff** is a tool designed to compare IAVL tree databases in Cosmos SDK-based blockchains. | ||
|
|
||
| It identifies differences between two IAVL tree versions across multiple prefixes and highlights key-value discrepancies. This is particularly useful for debugging non-deterministic issues in distributed systems where consistency is critical. | ||
|
|
||
| ## Features | ||
|
|
||
| - **Prefix Extraction**: Automatically extracts prefixes used in the databases for different modules. | ||
| - **Version Comparison**: Compares two databases at the latest common version for each prefix. | ||
| - **Tree Hash Comparison**: Computes and compares tree hashes to quickly detect discrepancies. | ||
| - **Key-Value Difference Detection**: Identifies and outputs differences in keys and values when tree hashes differ. | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. **Prefix Detection**: The tool scans the provided databases to extract all relevant prefixes. | ||
| 2. **Version Selection**: For each prefix, it determines the latest common version between the two databases. | ||
| 3. **Tree Comparison**: The tool compares the tree hashes at the selected version. If the hashes differ, it proceeds to identify specific key-value differences. | ||
| 4. **Output**: The differences, if any, are printed to the console, showing the exact keys and values that vary between the two databases. | ||
|
|
||
| ## Usage | ||
|
|
||
| Ensure you're in the `iavl-tree-diff` directory: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/pokt-network/poktroll.git && cd poktroll | ||
| cd tools/iavl-tree-diff | ||
| ``` | ||
|
|
||
| Then, simply run the tool with the following command: | ||
|
|
||
| ```bash | ||
| go run . <path_to_db1> <path_to_db2> | ||
| ``` | ||
|
|
||
| Example: | ||
|
|
||
| ```bash | ||
| go run . $HOME/pocket/testnet/halt-08-26/data-val/application.db $HOME/pocket/testnet/halt-08-26/data-fullnode/application.db | ||
| ``` | ||
|
|
||
| ## Example Output | ||
|
|
||
| ```text | ||
| Checking prefix: s/k:group/ | ||
| Using lower version: 15910 | ||
| Tree hash from db1: A89113C07AE262E84FA49E6D9111BF7C638679604357FA388526A976CFD021E0 | ||
| Tree hash from db2: A89113C07AE262E84FA49E6D9111BF7C638679604357FA388526A976CFD021E0 | ||
| Tree hashes are identical. | ||
| Checking prefix: s/k:supplier/ | ||
| Using lower version: 15910 | ||
| Tree hash from db1: 148180BFC904425FDBDD7046D4ECAA7238281DC2E76C66E079DBA76E5BCCD15C | ||
| Tree hash from db2: 31E6BCCD1B2712342B634C29651CE4DF4072080B70EBF26185BB4836D0A89B68 | ||
| Hashes differ, checking for differences in keys/values... | ||
| Key: Supplier/operator_address/pokt10a2lwlrkraqx6sud6gc8mk4ewvmtany6e9z7mp/ | ||
| Value in db1: 0A2B706F6B7431... | ||
| Value in db2: 0A2B706F6B74313061326C776C72... | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| module github.com/pokt-network/poktroll/tools/iavl-tree-diff | ||
|
|
||
| go 1.23 | ||
|
|
||
| require github.com/cosmos/iavl v0.0.0-20240827134643-70519d07c07e | ||
|
|
||
| require ( | ||
| cosmossdk.io/core v0.12.1-0.20240813134434-072a29c838a5 // indirect | ||
| github.com/cosmos/gogoproto v1.7.0 // indirect | ||
| github.com/cosmos/ics23/go v0.10.0 // indirect | ||
| github.com/emicklei/dot v1.6.2 // indirect | ||
| github.com/gogo/protobuf v1.3.2 // indirect | ||
| github.com/golang/snappy v0.0.4 // indirect | ||
| github.com/google/btree v1.1.3 // indirect | ||
| github.com/google/go-cmp v0.6.0 // indirect | ||
| github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect | ||
| golang.org/x/crypto v0.26.0 // indirect | ||
| golang.org/x/sys v0.24.0 // indirect | ||
| google.golang.org/protobuf v1.34.2 // indirect | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,120 @@ | ||
| cosmossdk.io/core v0.12.1-0.20240813134434-072a29c838a5 h1:aIfrfJUronk2qHPH4N7M5nGzijdSWkRwHHqqPGnQBrk= | ||
| cosmossdk.io/core v0.12.1-0.20240813134434-072a29c838a5/go.mod h1:B8JQN1vmGCPSVFlmRb/22n1T736P4C2qfEsrSKDX1iM= | ||
| github.com/cosmos/gogoproto v1.7.0 h1:79USr0oyXAbxg3rspGh/m4SWNyoz/GLaAh0QlCe2fro= | ||
| github.com/cosmos/gogoproto v1.7.0/go.mod h1:yWChEv5IUEYURQasfyBW5ffkMHR/90hiHgbNgrtp4j0= | ||
| github.com/cosmos/iavl v0.0.0-20240827134643-70519d07c07e h1:ezHALk3OcOaiK3IwrspWg9bpM2lSsaMxxlA7eL2tuto= | ||
| github.com/cosmos/iavl v0.0.0-20240827134643-70519d07c07e/go.mod h1:DQgcIbHHc+ONuUZqKj4h/6ojtCrPnrMIVhCV/NCcnOE= | ||
| github.com/cosmos/ics23/go v0.10.0 h1:iXqLLgp2Lp+EdpIuwXTYIQU+AiHj9mOC2X9ab++bZDM= | ||
| github.com/cosmos/ics23/go v0.10.0/go.mod h1:ZfJSmng/TBNTBkFemHHHj5YY7VAU/MBU980F4VU1NG0= | ||
| github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= | ||
| github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= | ||
| github.com/emicklei/dot v1.6.2 h1:08GN+DD79cy/tzN6uLCT84+2Wk9u+wvqP+Hkx/dIR8A= | ||
| github.com/emicklei/dot v1.6.2/go.mod h1:DeV7GvQtIw4h2u73RKBkkFdvVAz0D9fzeJrgPW6gy/s= | ||
| github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= | ||
| github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= | ||
| github.com/fsnotify/fsnotify v1.5.4 h1:jRbGcIw6P2Meqdwuo0H1p6JVLbL5DHKAKlYndzMwVZI= | ||
| github.com/fsnotify/fsnotify v1.5.4/go.mod h1:OVB6XrOHzAwXMpEM7uPOzcehqUV2UqJxmVXmkdnm1bU= | ||
| github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= | ||
| github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= | ||
| github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= | ||
| github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= | ||
| github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= | ||
| github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= | ||
| github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= | ||
| github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= | ||
| github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= | ||
| github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= | ||
| github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= | ||
| github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= | ||
| github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= | ||
| github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg= | ||
| github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4= | ||
| github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= | ||
| github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= | ||
| github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= | ||
| github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= | ||
| github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= | ||
| github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= | ||
| github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= | ||
| github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= | ||
| github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78= | ||
| github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= | ||
| github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= | ||
| github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= | ||
| github.com/onsi/ginkgo v1.14.0 h1:2mOpI4JVVPBN+WQRa0WKH2eXR+Ey+uK4n7Zj0aYpIQA= | ||
| github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY= | ||
| github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= | ||
| github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= | ||
| github.com/onsi/gomega v1.26.0 h1:03cDLK28U6hWvCAns6NeydX3zIm4SF3ci69ulidS32Q= | ||
| github.com/onsi/gomega v1.26.0/go.mod h1:r+zV744Re+DiYCIPRlYOTxn0YkOLcAnW8k1xXdMPGhM= | ||
| github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= | ||
| github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= | ||
| github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= | ||
| github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= | ||
| github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY= | ||
| github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7/go.mod h1:q4W45IWZaF22tdD+VEXcAWRA037jwmWEB5VWYORlTpc= | ||
| github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= | ||
| github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= | ||
| go.uber.org/mock v0.4.0 h1:VcM4ZOtdbR4f6VXfiOpwpVJDL6lCReaZ6mw31wqh7KU= | ||
| go.uber.org/mock v0.4.0/go.mod h1:a6FSlNadKUHUa9IP5Vyt1zh4fC7uAwxMutEAscFbkZc= | ||
| golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= | ||
| golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= | ||
| golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= | ||
| golang.org/x/crypto v0.26.0 h1:RrRspgV4mU+YwB4FYnuBoKsUapNIL5cohGAmSH3azsw= | ||
| golang.org/x/crypto v0.26.0/go.mod h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn54= | ||
| golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= | ||
| golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= | ||
| golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= | ||
| golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= | ||
| golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= | ||
| golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= | ||
| golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= | ||
| golang.org/x/net v0.0.0-20200813134508-3edf25e44fcc/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= | ||
| golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= | ||
| golang.org/x/net v0.21.0 h1:AQyQV4dYCvJ7vGmJyKki9+PBdyvhkSd8EIx/qb0AYv4= | ||
| golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= | ||
| golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= | ||
| golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= | ||
| golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= | ||
| golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= | ||
| golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= | ||
| golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= | ||
| golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.0.0-20200814200057-3d37ad5750ed/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= | ||
| golang.org/x/sys v0.24.0 h1:Twjiwq9dn6R1fQcyiK+wQyHWfaz/BJB+YIpzU/Cv3Xg= | ||
| golang.org/x/sys v0.24.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= | ||
| golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= | ||
| golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= | ||
| golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= | ||
| golang.org/x/text v0.17.0 h1:XtiM5bkSOt+ewxlOE/aE/AKEHibwj/6gvWMl9Rsh0Qc= | ||
| golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= | ||
| golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= | ||
| golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= | ||
| golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= | ||
| golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= | ||
| golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= | ||
| golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= | ||
| golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= | ||
| golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= | ||
| google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= | ||
| google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= | ||
| google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= | ||
| google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= | ||
| google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= | ||
| google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= | ||
| google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg= | ||
| google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw= | ||
| gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= | ||
| gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= | ||
| gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= | ||
| gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= | ||
| gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= | ||
| gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= | ||
| gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= | ||
| gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= |
Oops, something went wrong.