[Observability] Foundation for load testing telemetry #832

okdas · 2024-09-24T00:29:21Z

Summary

Refactor the foundation for E2E tokenomics observability w/ lots of new data points.

Key changes include:

x/tokenomics telemetry
Begin/End blockers execution time management
Custom poktroll telemetry config in app.toml

Issue

[Observability] Permissionless demand observability #762

okdas · 2024-10-04T00:30:25Z

Current dashboard:

Unfortunately, unable to make more requests without breaking the network.

Olshansk · 2024-10-04T02:41:15Z

@okdas

unable to make more requests without breaking the network.

What breaks the network?

okdas · 2024-10-04T17:02:09Z

What breaks the network?

Same issue - #841

okdas · 2024-10-04T23:31:18Z

telemetry/tokens.go

+		return
+	}
+
+	// CosmosSDK has a metric called `minted_tokens` (as a part of `mint` module), however it is wrongfully marked a `gauge`.


Going to open a cosmos-sdk pr to address this

@okdas Send the link and #PUC once you have it.

I'll help push it through (for review et al) on their end.

gitguardian · 2024-10-24T18:06:37Z

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12819930	Triggered	Generic Password	`d966241`	localnet/kubernetes/values-pocketdex-postgres.yaml	View secret
12819930	Triggered	Generic Password	`ce77cc7`	localnet/kubernetes/observability-prometheus-stack.yaml	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

…for-dashboards

Tiltfile

x/tokenomics/module/abci.go

Olshansk · 2024-10-30T16:38:51Z

cmd/poktrolld/cmd/config.go

 )

 var once sync.Once

+// PoktrollAdditionalConfig represents a poktroll-specific part of `app.toml` file.
+// See the `customAppConfigTemplate()` for additional information about each setting.
+type PoktrollAdditionalConfig struct {


I looked in Cosmos & Comet and the names for these are usually RPCConfig, CustomAppConfig, serverconfig.Config, etc...

Wdyt of PoktrollAppConfig?

Additional feels weird...

CN(aming)O at work

cmd/poktrolld/cmd/config.go

x/proof/keeper/msg_server_create_claim.go

x/proof/keeper/msg_server_submit_proof.go

x/tokenomics/keeper/settle_pending_claims.go

telemetry/tokens.go

Co-authored-by: Daniel Olshansky <[email protected]>

okdas · 2024-10-30T20:46:42Z

@Olshansk ready for another pass!

Olshansk · 2024-10-30T20:51:04Z

x/tokenomics/keeper/settle_pending_claims.go

+// finalizeTelemetry logs telemetry metrics for a claim based on its stage (e.g., EXPIRED, SETTLED).
+// Meant to run deferred.
+func (k Keeper) finalizeTelemetry(
+	proofStage prooftypes.ClaimProofStage,


s/proofStage/claimProofStage

Olshansk · 2024-10-30T20:52:47Z

telemetry/common.go

-// MetricNameKeys constructs the full metric name by prefixing with a defined
-// prefix and appending any additional metrics provided as variadic arguments.
+// MetricNameKeys prefixes metrics with `poktroll` for easy identification.
+// E.g., `("hodlers", "regret_level")` yields `poktroll_hodlers_regret_level` — great for tracking FOMO as hodlers rethink choices.


okdas · 2024-10-31T04:18:48Z

e2e tests passed locally. Merging this in.

…tlement # By Redouane Lakrache (3) and others # Via GitHub * pokt/main: [Observability] Foundation for load testing telemetry (#832) [Tokenomics] Implement Global Mint Reimbursement Request (#878) [Docs] Assign owners to all `TODO_BETA` (#898) [Supplier] Add supplier staking fee (#883) [Application] Single Service Applications (#886) # Conflicts: # testutil/keeper/tokenomics.go # x/tokenomics/keeper/settle_pending_claims.go # x/tokenomics/keeper/token_logic_modules.go # x/tokenomics/keeper/token_logic_modules_test.go # x/tokenomics/module/abci.go # x/tokenomics/types/errors.go

# By Redouane Lakrache (3) and others # Via GitHub * pokt/main: [Observability] Foundation for load testing telemetry (#832) [Tokenomics] Implement Global Mint Reimbursement Request (#878) [Docs] Assign owners to all `TODO_BETA` (#898) [Supplier] Add supplier staking fee (#883) [Application] Single Service Applications (#886) # Conflicts: # x/supplier/keeper/msg_server_stake_supplier.go # x/supplier/keeper/msg_server_stake_supplier_test.go # x/supplier/keeper/msg_server_unstake_supplier_test.go

## Summary Refactor the foundation for E2E tokenomics observability w/ lots of new data points. Key changes include: - `x/tokenomics` telemetry - Begin/End blockers execution time management - Custom `poktroll` telemetry config in `app.toml` ## Issue - #762 --------- Co-authored-by: Daniel Olshansky <[email protected]>

…pdates * pokt/main: (51 commits) [RelayMiner] Supplier rate limiting (#895) [Tokenomics] refactor: `MintAllocationApplication` var usage to param usage (#918) [Tokenomics] feat: add `mint_allocation_application` param to tokenomics module (#917) [Tokenomics] refactor: `MintAllocationSourceOwner` var usage to param usage (#916) [Tokenomics] feat: add `mint_allocation_source_owner` param to tokenomics module (#915) [Tokenomics] refactor: `MintAllocationSupplier` var usage to param usage (#914) [Tokenomics] feat: add `mint_allocation_supplier` param to tokenomics module (#913) [Tokenomics] refactor: `MintAllocationProposer` var usage to param usage (#912) [Tokenomics] feat: add `mint_allocation_proposer` param to tokenomics module (#911) [Tokenomics] refactor: `MintAllocationDao` var usage to param usage (#910) [Tokenomics] Add `mint_allocation_dao` tokenomics module param (#909) [quick-pr] change full-node.sh script links (#920) [Supplier] refactor: `NumSuppliersPerSession` var usage to param usage (#908) [Session Params] Add `num_suppliers_per_session` param to session module (#907) [Session Params] Add `MsgUpdateParam` to session module (#906) Updated some compiled proto files [TODOs] Update #2 to `TODO_BETA` to only reflect ACTUAL blockers (#900) [Workflow] Enforce `TODO_UPNEXT` comments to have an associated username (#896) [Observability] Foundation for load testing telemetry (#832) [Tokenomics] Implement Global Mint Reimbursement Request (#878) ...

measure begin/end blockers

ff762ea

okdas added the tooling Tooling - CLI, scripts, helpers, off-chain, etc... label Sep 24, 2024

okdas self-assigned this Sep 24, 2024

okdas changed the title ~~measure begin/end blockers~~ [Observability] measure begin/end blockers Sep 24, 2024

okdas added 5 commits September 23, 2024 18:01

--wip-- [skip ci]

1aad6ca

Merge remote-tracking branch 'origin/main' into add-metrics

ee84b91

metrixxx

025698a

--wip-- [skip ci]

b7c5bd3

bring back original

549710e

okdas changed the title ~~[Observability] measure begin/end blockers~~ [Observability] Tweak metrics and logs Sep 26, 2024

okdas added 2 commits September 26, 2024 19:38

TODO: figure out why prometheus doesn't scrape anymore

208f824

log level and reduce verbosity of some logs

90624df

okdas mentioned this pull request Sep 27, 2024

[Bug] UnbondSupplier with 0 balance causes consensus failure #841

Closed

7 tasks

increase the stakes to run the load-test

8fa8e04

This was referenced Sep 30, 2024

[Demand Scalability] Permissionless demand load testing & validation #742

Open

[Observability] Permissionless demand observability #762

Closed

okdas and others added 7 commits October 2, 2024 16:24

Merge branch 'main' into add-metrics

8bf2212

--wip-- [skip ci]

0262062

--wip-- [skip ci]

d39fb8f

--wip-- [skip ci]

69f3df5

--wip-- [skip ci]

fbfe6d1

--wip-- [skip ci]

cfa7dc7

add histogram

050687a

add custom metrics config

4786097

okdas commented Oct 4, 2024

View reviewed changes

okdas added 2 commits October 4, 2024 16:43

break proofs by app and supplier

83f1b7d

--wip-- [skip ci]

10eb2be

okdas and others added 2 commits October 24, 2024 10:03

change retention time on localnet

2cd91de

Merge branch 'main' into add-metrics

d966241

add psql datasource to grafana

ce77cc7

okdas mentioned this pull request Oct 25, 2024

[LocalNet] Use pocketdex for dashboards #893

Merged

15 tasks

okdas added 5 commits October 28, 2024 15:34

Merge remote-tracking branch 'origin/main' into add-metrics

e5cc4f6

localnet_up after merge

4a0b12b

Merge remote-tracking branch 'origin/add-metrics' into use-pocketdex-…

497e0e3

…for-dashboards

more dashboards

814d0a2

Merge remote-tracking branch 'origin/main' into add-metrics

cd438dc

okdas requested a review from Olshansk October 30, 2024 01:29

Olshansk requested changes Oct 30, 2024

View reviewed changes

Olshansk changed the title ~~[Observability] Tweak metrics and logs~~ [Observability] Foundation for load testing telemetry Oct 30, 2024

okdas and others added 5 commits October 30, 2024 10:55

Update Tiltfile

54d6c0f

Co-authored-by: Daniel Olshansky <[email protected]>

address the feedback

dd5276d

address feedback

fc53e96

clarify comments

a247a8e

clarify comments

5f693e5

okdas requested a review from Olshansk October 30, 2024 20:46

Olshansk approved these changes Oct 30, 2024

View reviewed changes

okdas added 3 commits October 30, 2024 16:40

Merge remote-tracking branch 'origin/main' into add-metrics

6cac26c

address feedback

a13c2ec

fix the cycle

6cd43f0

okdas merged commit bae452a into main Oct 31, 2024
9 of 10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Observability] Foundation for load testing telemetry #832

[Observability] Foundation for load testing telemetry #832

okdas commented Sep 24, 2024 •

edited by Olshansk

Loading

okdas commented Oct 4, 2024

Olshansk commented Oct 4, 2024

okdas commented Oct 4, 2024

okdas Oct 4, 2024

Olshansk Oct 5, 2024

gitguardian bot commented Oct 24, 2024 •

edited

Loading

Olshansk Oct 30, 2024

okdas Oct 30, 2024

okdas commented Oct 30, 2024

Olshansk Oct 30, 2024

Olshansk Oct 30, 2024

okdas commented Oct 31, 2024

[Observability] Foundation for load testing telemetry #832

[Observability] Foundation for load testing telemetry #832

Conversation

okdas commented Sep 24, 2024 • edited by Olshansk Loading

Summary

Issue

okdas commented Oct 4, 2024

Olshansk commented Oct 4, 2024

okdas commented Oct 4, 2024

okdas Oct 4, 2024

Choose a reason for hiding this comment

Olshansk Oct 5, 2024

Choose a reason for hiding this comment

gitguardian bot commented Oct 24, 2024 • edited Loading

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Olshansk Oct 30, 2024

Choose a reason for hiding this comment

okdas Oct 30, 2024

Choose a reason for hiding this comment

okdas commented Oct 30, 2024

Olshansk Oct 30, 2024

Choose a reason for hiding this comment

Olshansk Oct 30, 2024

Choose a reason for hiding this comment

okdas commented Oct 31, 2024

okdas commented Sep 24, 2024 •

edited by Olshansk

Loading

gitguardian bot commented Oct 24, 2024 •

edited

Loading