Observability of validators for relayers #3057
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… trevor/relayer-validator-metrics
… trevor/relayer-validator-metrics
… trevor/relayer-validator-metrics
|
tkporter
commented
Dec 14, 2023
jmrossy
reviewed
Dec 14, 2023
… trevor/relayer-validator-metrics
daniel-savu
approved these changes
Dec 18, 2023
daniel-savu
reviewed
Dec 18, 2023
… trevor/relayer-validator-metrics
daniel-savu
added a commit
that referenced
this pull request
Mar 13, 2024
### Description Includes the `app_context` classification in `PendingMessage`, and adds trait methods on `PendingOperation` to require always having such a label on `OpQueue` operations. This is done by reusing the matching list logic from the validator checkpoint labels (#3057). The nice thing is that this enables later support for retrying a group of `OpQueue` operations just by specifying the `app_context` label, without adding any new logic, since these labels are essentially matching list results. One downside to using `app_context` for retries is that the endpoint caller is constrained to only the matching lists defined by the relayer operator - however imo only the relayer operator that should be able to trigger retries. ### Drive-by changes The `OpQueue` type alias is converted to an actual struct, that stores the queue label (for metrics purposes), and also the `IntGaugeVec` metric: the generic group of metrics associated with that queue (basically only `submitter_queue_length` currently). ### Related issues - Fixes #3240 ### Backward compatibility Yes ### Testing Manual, by spinning up a relayer for injective and inevm. Sample metrics, from `--metricAppContexts '[{"name": "injectivelabel", "matchingList": [{"destination_domain": 6909546 }] }, {"name": "inevmlabel", "matchingList": [{"destination_domain": 2525 }] }]'` ``` hyperlane_submitter_queue_length{agent="relayer",app_context="inevmlabel",hyperlane_baselib_version="0.1.0",queue_name="confirm_queue",remote="inevm"} 11 hyperlane_submitter_queue_length{agent="relayer",app_context="inevmlabel",hyperlane_baselib_version="0.1.0",queue_name="prepare_queue",remote="inevm"} 0 hyperlane_submitter_queue_length{agent="relayer",app_context="injectivelabel",hyperlane_baselib_version="0.1.0",queue_name="confirm_queue",remote="injective"} 63 hyperlane_submitter_queue_length{agent="relayer",app_context="injectivelabel",hyperlane_baselib_version="0.1.0",queue_name="prepare_queue",remote="injective"} 13281 ```
yorhodes
pushed a commit
that referenced
this pull request
Mar 22, 2024
### Description Includes the `app_context` classification in `PendingMessage`, and adds trait methods on `PendingOperation` to require always having such a label on `OpQueue` operations. This is done by reusing the matching list logic from the validator checkpoint labels (#3057). The nice thing is that this enables later support for retrying a group of `OpQueue` operations just by specifying the `app_context` label, without adding any new logic, since these labels are essentially matching list results. One downside to using `app_context` for retries is that the endpoint caller is constrained to only the matching lists defined by the relayer operator - however imo only the relayer operator that should be able to trigger retries. ### Drive-by changes The `OpQueue` type alias is converted to an actual struct, that stores the queue label (for metrics purposes), and also the `IntGaugeVec` metric: the generic group of metrics associated with that queue (basically only `submitter_queue_length` currently). ### Related issues - Fixes #3240 ### Backward compatibility Yes ### Testing Manual, by spinning up a relayer for injective and inevm. Sample metrics, from `--metricAppContexts '[{"name": "injectivelabel", "matchingList": [{"destination_domain": 6909546 }] }, {"name": "inevmlabel", "matchingList": [{"destination_domain": 2525 }] }]'` ``` hyperlane_submitter_queue_length{agent="relayer",app_context="inevmlabel",hyperlane_baselib_version="0.1.0",queue_name="confirm_queue",remote="inevm"} 11 hyperlane_submitter_queue_length{agent="relayer",app_context="inevmlabel",hyperlane_baselib_version="0.1.0",queue_name="prepare_queue",remote="inevm"} 0 hyperlane_submitter_queue_length{agent="relayer",app_context="injectivelabel",hyperlane_baselib_version="0.1.0",queue_name="confirm_queue",remote="injective"} 63 hyperlane_submitter_queue_length{agent="relayer",app_context="injectivelabel",hyperlane_baselib_version="0.1.0",queue_name="prepare_queue",remote="injective"} 13281 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Goal of this was to have insight into validators of important sets being "up"
Introduces a new metric used by relayers:
hyperlane_observed_validator_latest_index, e.g.:Tapping into metadata building for multisig ISMs, the relayer will update the metric with the latest indices for the validators in a set. In order to prevent the cardinality being ridiculously high, only certain validator sets are tracked. This is done by introducing an
app_contextlabel (I'm very open to other names here, for some reason whenever idk how to name some kind of identifier I end up calling it a context 😆)The app context can either be:
--metricAppContexts '[{"name": "testapp", "matchingList": [{"recipient_address": "0xd84379ceae14aa33c123af12424a37803f885889", "destination_domain": 13371 }] }]'. This is nice for e.g. warp route deployments, where the ISM is maybe not a default ISM, and can be changedThis way we have insight in to the default ISM and any application-specific ISMs.
Some things to note:
Drive-by changes
.min(1)insteadRelated issues
Backward compatibility
yes
Testing
Ran locally - I think i'll probably add something in e2e tests, but opening now