Firehose sidecars for payments

[abourget]

This issue describes the way we can make Firehose available on The Graph network, with payments using the TAP protocol.

It was produced by @Jannis and I over the last few days.

---

Roles

CFO

Putting money in the payer_address for the Firehose Network Client is his responsibility, and it done prior to running these pieces of software, and out of band.

• It is their responsibility to replenish it for uninterrupted service.
• They can use whatever notification service on the web3 land, monitor.
• They can initiate a withdraw at any point in time, but complete it only after a thawing period (to allow for inflight vouchers to be claimed).

Firehose Client

• Program that initiates a Blocks(Request) to a Firehose service provider endpoint.
• Written in any language
• Custom built, generated form gRPC proto files

Firehose Network Client (sidecar)

Responsibilities

• Proxies the request altogether. Goal: keep the Firehose Client as simple as possible.
• Implement the client TAP dance:
  • Has a wallet, with a CLI flag --wallet-adddress 0x1231231098 --wallet-privkey-env-file
• Side-channel
• --network-subgraph-endpoint [default: https://thegraph.com/hosted-service/subgraph/graphprotocol/graph-network-mainnet]
• --preselect-indexer 0x123123123123110293781029832 // queried from the network subgraph above, this provides the network endpoint.
• --max-cost-per-unit 5.45 GRT // USD useable, converted
• --billing-unit BYTES_EGRESS // default value
• --escrow-release-increments ? (corresponds to the receipt amount each time it is requested)
• `--max-release-rate "50 GRT/h"
• Perhaps we need a JSON-RPC endpoint to set an anchor of the latest block, to be able to ignore indexers where StatusResponse will be too far (according to our own --max-block-lag-for-consideration).
• Targets a certain network / allocation_id which points to a network.

Behavior

Receive a Firehose request

Starts with an incoming proxied request from a Firehose Client

• When selecting the indexer, you can use the start_block and stop_block within the request, to determine if you need to filter out indexers that have not sync'd up to the chain head.

Init

• If pre-selected, just make sure it alive and go to connect
• If not pre-selected:
  • Hit Status endpoints everywhere? Compare all, threshold of latency, , relative to most recent (if required by the current Firehose::Block(Request)'s stop_block)'
  • Hit Cost endpoints? Elimitate those off bounds
  • Take what's left, round-robin?!

Some metrics to consider:

• Time to first byte
  • Can be assessed by the Firehose Network Client, between the Request to the remove Firehose, and the first respobse.
• Egress throughput
  • Again, can be measured with time and incoming bytes.
• Economic security
  • Indexer's total stake.
  • Staking percentage.
  • Ratio between the indexer's and total stake: who can you slash the most here.
• Latency
  • Unsure of the first anchor to measure latency, so unsure about this metric.

Firehose Server

Responsibilities

• Serves requests native Firehose requests
• Communicates with its sidecar on the server, for authentication and continuous authentication dauth
  • It also pushes to the sidecar, information about the running streams, like metrics. dmetric

Firehose Indexer Service (sidecar)

Responsibilities

• Listens for Network Client streams
  • Negotiates authentication
  • and payments
• Does not track collateral for the users. TAP agent does it all.

Flags

• --indexer-address, on-chain identity,
• --tap-agent-endpoint

Behavior

• Upon boot, fetch form the Network Subgraph, get the allocation for a given Deployment ID?

  • Watch for it changing.

• Upon receiving requests, it will take the payer_address, check against the tap-agent to know whether we've busted limits or any other error code worthy to be returned to the user.

• Upon receiving a new receipt or voucher, it will push into TAP agent.

• This communicates with the tap-agent:

  • Runs as its own network service? Library?
  • Is it exclusively writing and reading from the database?
  • Is it aware of the units?
  • Do we communicate this to it?

• Monitors the allocations attached to my indexer-address

  • Alllocation ID changes each 28 days
    • Points to a Deployment ID (which contains that firehose service manifest)

TAP agent

This is a separate process, networked.

Responsibilities

Listen for Indexer Service requests.

• Has a model of risks indexer is willing to take.
  • How much money am I willing to lose if the consumer doesn't collaborate in providing vouchers.

It receives receipts and vouchers and keeps track of them.

• Implement all the TAP dance
• Persistent storage of receipts and vouchers
• Keeps them in memory and is always able to give answers about the state of collateral, and whether service can be continued.

It has the additional role to collect the vouchers on-chain. Out of scope for the current discussion.

Flags

• --json-rpc-endpoint to be able to check on some balances or on-chain state for the addresses in the incoming requests?
• --indexer-address, identifies on chain, and our chunk of money in the escrow subgraph. The TAP agent is not capable of signing on behalf of that address, as it uses the operator address indirection.
• --operator-address for signing transactions, claiming vouchers, distinct from the controlling account for the whole indexer operations.
• --operator-mnemonic-env-file ?

• --dsn-database for durability of receipts and vouchers.
• --tap-escrow-subgraph-endpoint, to track collateral pairs for each consumers, and compare them with sums of vouchers and receipts
• --some-config-about-a-rate-of-voucher-claiming (how it interacts with withdrawals, and claiming and stuff, out of scope for our current concern)
• --per-consumer-receipt-aggregation-threshold 50 GRT

Behavior

• Watching the tap collateral for depletion, alongside withdrawal requests. Potentially cutting off the connection, with a proper error message saying "collateral depleted", with the address, in the
  • If collateral - pending_receipts - withdrawl_requests < 0 upon connection, or at any point during period updates, we error out and close connection.

Contracts

Escrow contract has a pair of payer_address (consumer, or senders in TAP verbiage), and indexer_address (receivers in TAP verbiage, identifying the service provider that can claim Vouchers out of that bucket of money).
The TAP collateral is the money deposited from a sender for a given receiver.

Service definitions

Standard Firehose service

// Implemented by `Firehose Client` as a client, and `Firehose Server` as a server.
service Firehose {
  rpc Block(Request) stream BlockResponse // this is the native Fierhose
}

Network Service

// served on indexer:/mainnet

service FirehoseNetworkService {
  rpc Status(StatusRequest) StatusResponse // status API
  rpc DiscoverCost(CostRequest) CostResponse // cost API
  
  rpc Stream(stream ClientMessage) stream ServiceMessage
}

message StatusRequest {
  
}
message StatusResponse {
  uint64 block_num = 1;
  string block_hash = 2;
}

message CostRequest {
  
}

message CostResponse {
  repeated CostModel cost_models = 1;
}

message CostModel {
  Unit unit = 1;
  double amount = 2;
}

message ClientMessage {
  oneof {
    InitSession init_session = 1;
    Receipt receipt = 2;
  }
  message InitSession {
	string payer_address = 1; // the on-chain address the guy is going to pay from? 
    string network = 2; // mainnet, polygon

    string continue_with_request_id = 5; // a request_id would represent here the long-running connection between Firehose CLient and Firehose Serve.. to be able to identify the calls beween the Firehose Server and the Firehose Network Service.

    double assumed_price = 3; // More predictable for the client. The server can shutdown the connection if his price got out of hand.
    // It'll be renegotiated upon reconnection here.
    Unit unit = 4;
  }
  message Receipt {
    // Indexer receives, and calls `tap_library`, and from there gets the amount of GRT?
    // allocation_id
    // value (fixed point 128 bits),
    // Timestamp
    // Random nonce
    // Signature
    bytes tap_payload = 1;
    // This assumes the pre-negotiated price
  }
  message Voucher {
    bytes tap_receipt_aggregate_voucher = 1; // Always signed by the Firehose Network Client
  }
}

enum Unit {
  BYTES_EGRESS = 1;
  BYTES_READ = 2;
}

// server sidecar:
//   -- 
message ServiceMessage {
  oneof {
    Error error = 1;
    OpenSession open_sess = 1;
    Progress progress = 3;
    ReceiptRequest receipt_request = 2; // could come before OpenSession
    RAVRequest rav_request = 3;
  }
  message Error {
    uint32 code = 1;
    string message = 2;
  }
  // When the service confirms it _has_ a receipt 
  message OpenSession {
    string url = 1;
    string auth_token = 2;

    string request_id = 3;  // to pick back up on a future `ClientMessage::InitSession`
  }
  message Progress {
    uint64 consumed_units = 1;
    uint64 remaining_units = 2;
    Unit unit = 2;
  }
  // RAV = Receipt Aggregate Voucher
  // it contains multiple Receipts
  message ReceiptRequest {
    // This would allow the client to not have to track the chain and its changing allocations, provided it is trustworthy and not a risk.
    bytes allocation_id = 1; // 160 bits
  }
  
  message RAVRequest {
    bytes tap_opaque_rav_request = 2; // containing the previous voucher, alongside a bunch of receipts.. even though the Client have sent them to you before, but maybe not. It includes the allocation_id, and we'll let the library handle that.
  }

  bytes sign_this_voucher_please = 3;
}

TAP agent

service {
  rpc ContinuityCheck(ContinuityRequest) ContinuityResponse;
  rpc SubmitReceipt(SubmitReceiptRequest) ContinuityResponse;
  rpc SubmitVoucher(SubmitReceiptRequest) ContinuityResponse; 
}

message ContinuityRequest {
  strong payer_address = 1;
}
message ContinuityResponse {
  uint64 remaining_collateral = 2; // on-chain collateral with all the vouchers and additional receipts I've received.
}
message SubmitReceiptRequest {
  bytes tap_payload_for_a_receipt = 1;
}
message SubmitVoucherRequest {
  bytes tap_payload_for_a_voucher = 1;
}

Open questions

• Initial deposit into tap escrow contract, needs there to write to chat. Needs json-rpc endpoint. Run through this.

• Why is it necessary to obtain signed receipts, if the indexer can't act on them, because it requires a voucher (and even sends the consumer those receipts back for aggregation).

  • Is it to collect sibling receipts, ensuring it is spent only once?

• Can we push following functionality into the tap-agent:

  • Isolating the writing to the database to the tap-agent, if it makes sense?

• Can the consumer (Network Client) trust an incoming allocation_id that would potentially be provided by the Indexer Service, or does that pose a security risk?

  • If we could do this, the Network Client wouldn't need to keep its own track of the allocations shifting and all..