[Documentation] Python Automation script and workflow for github wiki (…

…#419)

## Description

### Motivation

Create a [GitHub wiki](https://docs.github.com/en/communities/documenting-your-project-with-wikis/about-wikis) (a separate git repo relative to this repo) for the documentation across this GitHub repo. GitHub wikis have better visibility and navigation for users.

### Background

GitHub wikis lack version control and are entirely separate relative to this source repository. To solve both of these issues, we use the documentation/markdown files from the main repo to update the Github wiki repository.

To maximize the benefit of the wiki, a hierarchical view is created in a sidebar. However, the mapping of the markdowns files to the sidebar poses some challenges. 

The PR is an attempt at a very simple solution. A GitHub action that tracks changes to a set of files and triggers a python script. The script picks up pre-written comments in the set of files to generate a sidebar. From the sidebars linking reference, it copies over the files from the main repo to the wiki repo.

### Pre-Requisites to Merge this in

- [ ] Create a repository secret. Based on the legacy access rules, tick the `repo` permission box.

### Outstanding issues

- [ ] Improve the format of the sidebar, so categories don't repeat. Rolling the main README of a category into the category heading so that the heading is also a link and not just text.
- [ ] Change the internal links of the Markdown files in the wiki to point to other paths in the wiki by either:
   1.  A regex change to the current wiki file names (aka wiki titles)
   2. Changing the wiki file mapping to align with the internal links.

## Issue

Fixes #133 

Remaining/new issues

## Type of change

Please mark the relevant option(s):

- [x] New feature, functionality or library
- [ ] Bug fix
- [x] Code health or cleanup
- [ ] Major breaking change
- [x] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- (Apologies in advance) Some formatting changes from auto linting of markdowns files. I don't like this as it has blown out the diff. I can try to remove if it's too messy.
- Added a GitHub action that tracks change to `.md` files, sets env variables and runs a python script
- Added a python script in a new directory called tools. Finds the path of `.md` files, maps them to the specified format described in the files, generates a GitHub wiki sidebar file and respective wiki files. Pulls the current wiki, syncs the files and pushes them. 
- Added comments at the bottom of `.md` files found from `find . -name "*.md" | grep -v -e "vendor" -e "app"` in the form of `<!-- GITHUB_WIKI: category/subcategory/.../file_name -->`.
- Brief description of maintaining the wiki format comment in the `docs/development/readme`
- Brief mention and linking to the wiki in the main readme.
 
## Testing

- [ ] `make develop_test`
- [ ] [LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md) w/ all of the steps outlined in the `README`
d

Screenshot of output from example wiki on a personal fork: [example wiki](https://github.com/profishional/pocket/wiki)
<img width="234" alt="image" src="https://user-images.githubusercontent.com/93366432/209815735-879b17c4-8e64-4b41-b75a-54ea4dc5c9fc.png">

## Required Checklist

- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist
- [x] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added, or updated, [mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding README(s)
- [ ] I have added, or updated, documentation and [mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*` if I updated `shared/*`README(s)

---

Co-authored-by: Daniel Olshansky <[email protected]>
Co-authored-by: Dmitry Knyazev <[email protected]>
Fixes #133

.github/workflows/wiki_sync_process.yml

@@ -0,0 +1,24 @@
+name: Publish docs to Wiki
+
+on:
+  push:
+    branches:
+      - main # 1. When changes are merged to the main branch
+    paths:
+      - '**/*.md' # 2. It detects changes to Markdowns files in the repository
+
+env: # 3. Set variables which the Python script uses for git commands
+  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # TODO_IN_THIS_COMMIT: provision repo secret through Pocket account
+  USER_NAME: <user_name>
+  USER_EMAIL: <user_email>
+  OWNER: ${{ github.event.repository.owner.name }}
+  REPOSITORY_NAME: ${{ github.event.repository.name }}
+
+jobs:
+  publish_docs_to_wiki:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Generate and push Wiki
+      run: | # 4. Runs a python script that uploads md files to Pocket Github Wiki
+        python tools/wiki_sync.py

README.md

@@ -14,7 +14,9 @@ The official implementation of the [V1 Pocket Network Protocol Specification](ht
 
 ---
 
-All the links you'll need are listed below. If you'd like to contribute to the Pocket V1 Protocol, start by:
+Some relevant links are listed below. Refer to the complete ongoing documentation at the **[Pocket GitHub Wiki](https://github.com/pokt-network/pocket/wiki)**.
+
+If you'd like to contribute to the Pocket V1 Protocol, start by:
 
 1. Get up and running by reading the [Development Guide](docs/development/README.md)
 2. Find a task by reading the [Contribution Guide](docs/contributing/README.md)
@@ -72,3 +74,5 @@ All the links you'll need are listed below. If you'd like to contribute to the P
 ## License
 
 This project is licensed under the MIT License; see the [LICENSE](LICENSE) file for details.
+
+<!-- GITHUB_WIKI: home/readme -->

build/config/README.md

@@ -47,3 +47,5 @@ The config and genesis files located in the `./build/config/` directory are need
 These builds currently expect four (validator) `config.json` file and a single `genesis.json` file.
 
 Until #186 is implemented, take caution when overwriting / deleting the files with different configurations.
+
+<!-- GITHUB_WIKI: build/config/readme -->

consensus/doc/CHANGELOG.md

@@ -191,3 +191,5 @@ General
 - Benchmarking via `make benchmark_sortition`
 - VRF Wrapper library in `consensus/leader_election/vrf/` of github.com/ProtonMail/go-ecvrf/ecvrf
 - Implementation of Algorand's Leader Election sortition algorithm in `consensus/leader_election/sortition/`
+
+<!-- GITHUB_WIKI: changelog/consensus -->

consensus/doc/PROTOCOL_STATE_SYNC.md

@@ -396,3 +396,5 @@ Aptos follow an **async "fire-and-forget"** pattern as can be seen [here](https:
   - A detailed list of the type of requests Chia uses for communication between peers
 - [https://docs.chia.net/node-syncing](https://docs.chia.net/node-syncing)
   - An explanation of the configurations Chia exposes for node synching
+
+<!-- GITHUB_WIKI: consensus/state_sync_protocol -->

docs/contributing/README.md

@@ -9,3 +9,5 @@ Once V1 is closer to mainnet launch, we will have an [extensive contribution gui
 ![](./badge.png)
 
 3. Collaborate with the team and other core contributors on Discord in the [#v1-dev channel](https://discord.com/channels/553741558869131266/986789914379186226) or [#v1-research channel](https://discord.com/channels/553741558869131266/936666517498650644).
+
+<!-- GITHUB_WIKI: guides/contributing/readme -->

docs/demos/iteration_3_end_to_end_tx.md

@@ -163,3 +163,5 @@ go run app/client/*.go --path_to_private_key_file=/tmp/val2.json Account Send 67
 ```bash
 make swagger-ui
 ```
+
+<!-- GITHUB_WIKI: guides/demos/iteration_three_end_to_end -->

docs/deps/README.md

@@ -48,3 +48,5 @@ This is okay in the case of Pocket's Leader Election Algorithm because the seed
 2. The network leverages consensus messages to distribute the keys throughout the network in `O(N)`
 3. The VRF keys begin to be used for leader election at some `height (N+M)` where `M > 0`
 4. The input to the VRF for each `height (N+M')` where `M' ≥ M` will use publicly known information (e.g. appHash, byzValidators, etc..) known at `height (N+M'-1)` and therefore satisfy the security notice above.
+
+<!-- GITHUB_WIKI: guides/dependencies/readme -->

docs/development/FAQ.md

@@ -42,3 +42,5 @@ sudo chcon -Rt svirt_sandbox_file_t ./pocket
 ```
 
 See [this stackoverflow post](https://stackoverflow.com/questions/24288616/permission-denied-on-accessing-host-directory-in-docker) for more details.
+
+<!-- GITHUB_WIKI: guides/development/FAQ -->

docs/development/README.md

@@ -231,6 +231,30 @@ Pocket
 └── Makefile                          # The source of targets used to develop, build and test
 ```
 
+## Maintaining Documentation
+
+Documentation files currently found by the following command `find . -name "*.md" | grep -v -e "vendor" -e "app"` are added to the [Github Wiki Repository](https://github.com/pokt-network/pocket/wiki). The Wiki will be improved overtime but in its current form, provides an organized overview of the Repository.
+
+To keep the Wiki organized, a comment is added at the end of each `.md` file. For example, you can find the following one at the end of this file `<!-- GITHUB_WIKI: guides/development/readme -->`. The structure of the comment indicates the category (guides), subcategory(ies) (development) and filename (readme): `<!-- GITHUB_WIKI: <category>/<subcategory 1>/.../<filename>`. You can see the example output in the [Wiki Page](https://github.com/pokt-network/pocket/wiki/Development-Readme).
+
+If you are adding a new `.md` file for documentation please included a similar comment. Use your best judgment for the category and subcategory if its a new directory. Otherwise, copy the comment from a similar file in the directory and choose a relevant filename.
+
+## Documentation Resources and Implementation
+
+### [Github Actions File](/.github/workflows/wiki_sync_process.yml)
+
+The Action is triggered when there is a change to any Markdown file on the main branch of the Repository. When triggered, environment variables are set for a Python script that updates the Github Wiki Repository based on Pocket Repository files.
+
+### [Python Script](/tools/wiki_sync.py)
+
+The script finds the relevant Markdown files in the repository and organizes them for the Wiki Repository. Currently, the find command is filtered to exclude the `./app` and `./vendor` directories. Based on the list of `.md` file paths, it maps the formatting spec from [above](##Maintaining-Documentation) to some information about the file. Using the map, it creates a Sidebar file format which Github uses as a Table of Contents for the wiki. Also, from the Pocket repo we copy over the files with titles linking to the Sidebar format.
+
+Below, you can see some of the patterns between the Sidebar format, folder of markdowns in the Wiki Repository, and final sidebar/table of contents display.
+
+|Format|Folder|Wiki|
+|---|---|---|
+|![format](/tools/images/sidebar_format.png)|![Folder](/tools/images/folder.png)|![wiki](/tools/images/sidebar.png)|
+
 ### Linters
 
 We utilize `golangci-lint` to run the linters. It is a wrapper around a number of linters and is configured to run many at once. The linters are configured to run on every commit and pull request via CI, and all code issues are populated as GitHub annotations to let developers and reviewers easily locate an issue.
@@ -271,3 +295,5 @@ The official documentation includes a list of different linters and their config
 We can write custom linters using [`go-ruleguard`](https://go-ruleguard.github.io/). The rules are located in the [`build/linters`](../../build/linters) directory. The rules are written in the [Ruleguard DSL](https://github.com/quasilyte/go-ruleguard/blob/master/_docs/dsl.md), if you've never worked with ruleguard in the past, it makes sense to go through [introduction article](https://quasilyte.dev/blog/post/ruleguard/) and [Ruleguard by example tour](https://go-ruleguard.github.io/by-example/).
 
 Ruleguard is run via `gocritic` linter which is a part of `golangci-lint`, so if you wish to change configuration or debug a particular rule, you can modify the `.golangci.yml` file.
+
+<!-- GITHUB_WIKI: guides/development/readme -->

docs/learning/README.md

@@ -218,3 +218,5 @@ We do not consider them bad, but time is limited so it is important to focus on
 
 - [Blockchains Meet Distributed Hash Tables: Decoupling Validation from State Storage](https://arxiv.org/abs/1904.01935)
   - An "extended abstract" of how Authenticated Data Structures (i.e. Merkle Trees) could be _"sharded"_ across nodes using Distributed Hash Tables (DHTs) to reduce the state required to be maintained and synched by each node.
+
+<!-- GITHUB_WIKI: guides/learning/readme -->

docs/releases/README.md

@@ -43,3 +43,5 @@ Extended images with additional tooling are built to aid in troubleshoot and deb
 ## [TODO] Magefile build system
 
 Once the V1 implementation reaches the stage of testable binaries, we are looking to use [Mage](https://magefile.org/) which is being tracked in [pocket/issues/43](https://github.com/pokt-network/pocket/issues/43) that'll inject a version with the `-version` flag.
+
+<!-- GITHUB_WIKI: guides/releases/readme -->

docs/roadmap/README.md

@@ -103,3 +103,5 @@ gantt
 ## M7. Pocket NoS (North Star)
 
 Shoot for the ✨ and we will land on the 🌕
+
+<!-- GITHUB_WIKI: guides/roadmap/readme -->

p2p/CHANGELOG.md

@@ -131,3 +131,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [0.0.0.0] - 2022-06-16
 
 - RainTree first iteration in Pre2P module (no cleanup or redundancy)
+
+<!-- GITHUB_WIKI: changelog/p2p -->

p2p/README.md

@@ -124,3 +124,5 @@ You can read the documentation in the [python simulator](https://github.com/pokt
 
 - **Deterministic Private Key Generation**: Since RainTree is dependant on the lexicographic order of the addresses, the generation of the private keys (and in turn the public keys and addresses) is important and cannot be randomized for the time being.
 - **Variable Coupling**:There is an implicit coupling between `validatorId`, `serviceUrl` and `genericParam` that requires understanding of the codebase. Reach out to @olshansk or @andrewnguyen22 for clarity on this.
+
+<!-- GITHUB_WIKI: p2p/readme -->

persistence/docs/CHANGELOG.md

@@ -284,3 +284,5 @@ Pocket Persistence 1st Iteration (https://github.com/pokt-network/pocket/pull/73
 - Update to the Persistence module interface to enable historical height queries
 - Library / infrastructure for persistence unit fuzz testing
 - Tests triggered via `make test_persistence`
+
+<!-- GITHUB_WIKI: changelog/persistence -->

persistence/docs/PROTOCOL_STATE_HASH.md

@@ -136,3 +136,5 @@ _TODO: If an error occurs at any step, all of the operations must be reverted in
 ## Failed Commitments
 
 TODO: Failed commitments and the implementation of rollbacks is tracked in #327 and #329.
+
+<!-- GITHUB_WIKI: persistence/state_hash_protocol -->

persistence/docs/README.md

@@ -225,7 +225,7 @@ Short-term (i.e. simpler starter) tasks:
 - [ ] DOCUMENT: Need to do a better job at documenting the process of paused apps being turned into unstaking apps.
 - [ ] CLEANUP: Remove unused parameters from `the PostgresContext` interface (i.e. see where \_ is used in the implementation such as in `InsertFisherman`)
 - [ ] IMPROVE: Consider converting all address params from bytes to string to avoid unnecessary encoding
-- [ ] CLEANUP(https://github.com/pokt-network/pocket/issues/76): Review all the `gov_*.go` related files and simplify the code
+- [ ] CLEANUP(#76): Review all the `gov_*.go` related files and simplify the code
 - [ ] REFACTOR/DISCUSS: Should we prefix the functions in the `PersistenceModule` with the Param / Actor it's impacting to make autocomplete in implementation better?
 - [ ] DISCUSS: Consider removing all `Set` methods (e.g. `SetAccountAmount`) and replace with `Add` (e.g. `AddAccountAmount`) by having it leverage a "default zero".
 - [ ] REFACTOR(https://github.com/pokt-network/pocket/issues/102): Split `account` and `pool` into a shared actor (e.g. like fisherman/validator/serviceNode/application) and simplify the code in half
@@ -242,3 +242,5 @@ Long-term (i.e. design) tasks
 
 - [ ] INVESTIGATE: Expand the existing fuzzing approach to push random changes in state transitions to its limit.
 - [ ] INVESTIGATE: Use a DSL-like approach to design complex "user stories" for state transitions between protocol actors in different situations.
+
+<!-- GITHUB_WIKI: persistence/readme -->

persistence/indexer/doc/README.md

@@ -9,8 +9,10 @@
 | HASHKEY      | `h/SHA3(TxResultProtoBytes)` | TxResultProtoBytes | store value by hash (the key here is equivalent to the VALs below) |
 | HEIGHTKEY    | `b/height/txIndex`             | HASHKEY            | store hashKey by height                                            |
 | SENDERKEY    | `s/senderAddr`               | HASHKEY            | store hashKey by sender                                            |
-| RECIPIENTKEY | `r/recipientAddr`            | HASHKEY            | store hashKey by recipient (if not empty)                          
+| RECIPIENTKEY | `r/recipientAddr`            | HASHKEY            | store hashKey by recipient (if not empty)
 
 ## ELEN Index
 
-The height/txIndex store uses [ELEN](https://github.com/jordanorelli/lexnum/blob/master/elen.pdf). This is to ensure the results are stored sorted (assuming the `KVStore` uses a byte-wise lexicographical sorting).
+The height/txIndex store uses [ELEN](https://github.com/jordanorelli/lexnum/blob/master/elen.pdf). This is to ensure the results are stored sorted (assuming the `KVStore` uses a byte-wise lexicographical sorting).
+
+<!-- GITHUB_WIKI: persistence/indexer/readme -->

rpc/doc/CHANGELOG.md

@@ -48,3 +48,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Spec: basic Openapi.yaml
   - Codegen: code generation for the Server + DTOs
   - Codegen: code generation for the Client
+
+<!-- GITHUB_WIKI: changelog/rpc -->

rpc/doc/README.md

@@ -109,3 +109,5 @@ Definitely we'll need ways to retrieve transactions as well so we can envisage:
 └── v1
     └── openapi.yaml         # OpenAPI v3.0 spec (source for the generated files above)
 ```
+
+<!-- GITHUB_WIKI: rpc/readme -->

runtime/docs/CHANGELOG.md

@@ -70,3 +70,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Modules now accept `interfaces` instead of paths.
 - Unmarshalling is done in a new `runtime` package (runtime because what we do in there affects the runtime of the application)
 - We are now able to accept configuration via environment variables (thanks to @okdas for inspiration and [sp13 for Viper]("github.com/spf13/viper"))
+
+<!-- GITHUB_WIKI: changelog/runtime -->

runtime/docs/README.md

@@ -47,3 +47,5 @@ This module includes the following components:
   Clock is a drop-in replacement for some of the features offered by the `time` package, it acts as an injectable clock implementation used to provide time manipulation while testing.
 
   By default, the **real** clock is used and while testing it's possible to override it by using the "option" `WithClock(...)`
+
+<!-- GITHUB_WIKI: runtime/readme -->

shared/CHANGELOG.md

@@ -131,3 +131,5 @@ Configs:
 - Shared interfaces and general 'base' configuration located here
 - Moved make client code to 'debug' to clarify that the event distribution is for the temporary local net
 - Left multiple `TODO` for remaining code in test_artifacts to think on removal of shared testing code
+
+<!-- GITHUB_WIKI: changelog/shared -->

shared/README.md

@@ -85,3 +85,5 @@ Intentionally left blank for now. Reach out to the team if you have questions.
 ## Interfaces
 
 Intentionally left blank for now. Reach out to the team if you have questions.
+
+<!-- GITHUB_WIKI: shared/readme -->

shared/docs/PROTOCOL_STATE_HASH.md

@@ -173,3 +173,5 @@ sequenceDiagram
     %% Validate the computed hash
     C->>C: Compare local hash<br>against proposed hash
 ```
+
+<!-- GITHUB_WIKI: shared/state_hash_protocol -->

shared/docs/flows/README.md

@@ -1,3 +1,5 @@
 # Flows
 
 The purpose of [shared/flows](./) is to document cross-module communication for end-to-end behaviour that has cross-module dependencies and and depends on the interfaces exposed by the interfaces exposed in [shared/modules](../modules).
+
+<!-- GITHUB_WIKI: shared/flows_readme -->

shared/modules/doc/CHANGELOG.md

@@ -88,3 +88,5 @@ UtilityModule
 - Deprecated old build files
 - Use new config and genesis files for make compose_and_watch
 - Use new config and genesis files for make client_start && make client_connect
+
+<!-- GITHUB_WIKI: changelog/shared_modules -->

shared/modules/doc/README.md

@@ -103,4 +103,6 @@ err := newModule.Stop()
 if err != nil {
 	// handle error
 }
-```
+```
+
+<!-- GITHUB_WIKI: shared/modules/readme -->

telemetry/README.md

@@ -194,3 +194,5 @@ const NODES_ALIVE_COUNTER_TIME_SERIES_DESCRIPTION = "Number of nodes online"
 ```
 
 This makes referencing the metrics easier, and makes it easy to change the name of the metric later on, in addition to the fact that it gathers metric definitions in a single file.
+
+<!-- GITHUB_WIKI: guides/telemetry/readme -->