Context

Sequenced events like HyperlaneMessage and MerkleTreeInsertion are indexed using a forward-backward cursor which is initialized to the sequence number at the blockchain tip, when the agent is started.

For instance, if the last HyperlaneMessage had nonce 123, the forward-backward cursor would instantiate:

• a forward cursor that looks for messages in new blocks, with nonces 124 and up
• a backward cursor that iterates blocks from the tip back to genesis (or to whatever is configured in index.from)

Problem

When an RPC becomes healthy after an outage, it's hard to know if agents are making any progress indexing the missed events over that period. If they're running but not making any progress, the RPC may still be having issues. The only way to know this is by checking raw logs, which requires knowledge of what to look for.

Suggested Solution

A simple way to provide better observability here is to have some new Prometheus metrics for indexed events:

• for all event types (regardless if sequenced or not): the block number the cursor(s) are currently at
• for sequenced events: the sequence number that the backward and forward cursors are on, and the max known sequence

To continue with the HyperlaneMessage example, the metrics at a random point for a given chain would look something like:

• cursor_current_block{eventType: 'HyperlaneMessage', cursorType: 'ForwardSequenced'}: 118000
• cursor_current_sequence{eventType: 'HyperlaneMessage', cursorType: 'ForwardSequenced'}: 240
• cursor_current_block{eventType: 'HyperlaneMessage', cursorType: 'BackwardSequenced'}: 91021
• cursor_current_sequence{eventType: 'HyperlaneMessage', cursorType: 'BackwardSequenced'}: 82
• cursor_max_sequence{eventType: 'HyperlaneMessage'}: 345

Based on this, it will be much easier to have dashboard showing the indexing progress agents make after each RPC rotation and restart, per indexed event and chain.