[Development] Add LocalDev pod debugging

Makefile

@@ -625,3 +625,10 @@ ggshield_secrets_scan: ## Scans the project for secrets using ggshield
 .PHONY: ggshield_secrets_add
 ggshield_secrets_add: ## A helper that adds the last results from `make ggshield_secrets_scan`, store in `.cache_ggshield` to `.gitguardian.yaml`. See `ggshield for more configuratiosn`
 	ggshield secret ignore --last-found
+
+# The validator's pod name to debug
+debugPod=validator-001-pocket-0
+
+.PHONY: kube_debug
+kube_debug: ## Forward dlv client connections to debugPod that should be running a dlv server at the same port
+	kubectl port-forward ${debugPod} 7081:7081

build/localnet/README.md

@@ -16,9 +16,15 @@ This guide shows how to deploy a LocalNet using [pocket-operator](https://github
   - [Interacting w/ LocalNet](#interacting-w-localnet)
     - [Make Targets](#make-targets)
   - [Addresses and keys on LocalNet](#addresses-and-keys-on-localnet)
+    - [Applications staked on LocalNet](#applications-staked-on-localnet)
+    - [Servicers staked on LocalNet](#servicers-staked-on-localnet)
 - [How to change configuration files](#how-to-change-configuration-files)
   - [Overriding default values for localnet with Tilt](#overriding-default-values-for-localnet-with-tilt)
 - [How does it work?](#how-does-it-work)
+- [Debug with dlv](#debug-with-dlv)
+  - [k8s LocalNet: Connect to a node debugging server](#k8s-localnet-connect-to-a-node-debugging-server)
+    - [Configure VSCode debugger](#configure-vscode-debugger)
+  - [Debug tests](#debug-tests)
 - [Troubleshooting](#troubleshooting)
   - [Why?](#why)
   - [Force Trigger an Update](#force-trigger-an-update)
@@ -206,6 +212,55 @@ The k8s manifests that `tilt` submits to the cluster can be found in [this direc
 
 Tilt continuously monitors files on local filesystem in [specific directories](Tiltfile#L27), and it rebuilds the binary and distributes it to the pods on every code change. This allows developers to iterate on the code and see the changes immediately (i.e. hot-reloading).
 
+## Debug with dlv
+
+### k8s LocalNet: Connect to a node debugging server
+
+LocalNet Pocket nodes have a `dlv` debugging server opened on port `7081`. In order to connect a debugging client (VSCode, GoLand) and be able to set breakpoints, we need to setup a tunnel to the pod we want to debug.
+
+```bash
+make kube_config
+```
+
+#### Configure VSCode debugger
+
+Add the following to your VSCode debugging configuration file `.vscode/launch.json`. You can add the file from IDE if it does not exist.
+
+![image](https://github.com/pokt-network/pocket/assets/231488/2882008e-3632-469c-9d1e-2f77ad785dfe)
+
+```json
+{
+  "name": "Debug Pocket Node",
+  "type": "go",
+  "request": "attach",
+  "mode": "remote",
+  "remotePath": "${workspaceFolder}",
+  "port": 7081,
+  "host": "localhost"
+}
+```
+
+Then you have just to set your breakpoints and start the debugging session (no need to rebuild or restart the binary).
+
+[Watch demo](https://github.com/pokt-network/pocket/assets/231488/3525161f-2098-488a-8f36-8747e40320a6)
+
+k8s runs liveness and readiness checks will restart the pods that are not responsive. In order to not have the debugging session dropped because of the pod restarting, you have to comment `livenessProbe` and `readinessProbe` sections in `charts/pocket/templates/statefulset.yaml`
+
+```yaml
+# Comment this section disable liveness checks for debugging
+livenessProbe:
+  httpGet:
+    path: /v1/health
+    port: rpc
+# Comment this section disable readiness checks for debugging
+readinessProbe:
+  httpGet:
+    path: /v1/health
+    port: rpc
+```
+
+### Debug tests
+
 ## Troubleshooting
 
 ### Why?

build/localnet/Tiltfile

@@ -102,15 +102,24 @@ local_resource(
 docker_build_with_restart(
     "pocket-image",
     root_dir,
-    dockerfile_contents="""FROM debian:bullseye
+    dockerfile_contents="""FROM golang:1.20-bullseye
+RUN go install github.com/go-delve/delve/cmd/dlv@latest
 COPY bin/pocket-linux /usr/local/bin/pocket
 WORKDIR /
 """,
     only=["./bin/pocket-linux"],
     entrypoint=[
+        "dlv",
+        "exec",
         "/usr/local/bin/pocket",
-        "-config=/pocket/configs/config.json",
-        "-genesis=/pocket/configs/genesis.json",
+        "--headless",
+        "--listen=:7081",
+        "--api-version=2",
+        "--accept-multiclient",
+        "--continue",
+        "--",
+        "--config=/pocket/configs/config.json",
+        "--genesis=/pocket/configs/genesis.json",
     ],
     live_update=[sync("bin/pocket-linux", "/usr/local/bin/pocket")],
 )