diff --git a/api/docs/churner.md b/api/docs/churner.md new file mode 100644 index 0000000000..124f3d3d62 --- /dev/null +++ b/api/docs/churner.md @@ -0,0 +1,141 @@ +# Protocol Documentation + + +## Table of Contents + +- [churner.proto](#churner-proto) + - [ChurnReply](#churner-ChurnReply) + - [ChurnRequest](#churner-ChurnRequest) + - [OperatorToChurn](#churner-OperatorToChurn) + - [SignatureWithSaltAndExpiry](#churner-SignatureWithSaltAndExpiry) + + - [Churner](#churner-Churner) + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## churner.proto + + + + + +### ChurnReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| signature_with_salt_and_expiry | [SignatureWithSaltAndExpiry](#churner-SignatureWithSaltAndExpiry) | | The signature signed by the Churner. | +| operators_to_churn | [OperatorToChurn](#churner-OperatorToChurn) | repeated | A list of existing operators that get churned out. Note: it's possible an operator gets churned out just for one or more quorums (rather than entirely churned out for all quorums). | + + + + + + + + +### ChurnRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| operator_to_register_pubkey_g1 | [bytes](#bytes) | | The operator making the churn request. | +| operator_to_register_pubkey_g2 | [bytes](#bytes) | | | +| operator_request_signature | [bytes](#bytes) | | The operator's BLS signature signed on the keccak256 hash of concat("ChurnRequest", g1, g2, salt). | +| salt | [bytes](#bytes) | | The salt used as part of the message to sign on for operator_request_signature. | +| quorum_ids | [uint32](#uint32) | repeated | The quorums to register for. Note: - If any of the quorum here has already been registered, this entire request will fail to proceed. - If any of the quorum fails to register, this entire request will fail. The IDs must be in range [0, 255]. | + + + + + + + + +### OperatorToChurn +This describes an operator to churn out for a quorum. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| quorum_id | [uint32](#uint32) | | The ID of the quorum of the operator to churn out. | +| operator | [bytes](#bytes) | | The address of the operator. | +| pubkey | [bytes](#bytes) | | BLS pubkey (G1 point) of the operator. | + + + + + + + + +### SignatureWithSaltAndExpiry + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| signature | [bytes](#bytes) | | Churner's signature on the Operator's attributes. | +| salt | [bytes](#bytes) | | Salt is the keccak256 hash of concat("churn", time.Now(), operatorToChurn's OperatorID, Churner's ECDSA private key) | +| expiry | [int64](#int64) | | When this churn decision will expire. | + + + + + + + + + + + + + + +### Churner +The Churner is a service that handles churn requests from new operators trying to +join the EigenDA network. +When the EigenDA network reaches the maximum number of operators, any new operator +trying to join will have to make a churn request to this Churner, which acts as the +sole decision maker to decide whether this new operator could join, and if so, which +existing operator will be churned out (so the max number of operators won't be +exceeded). +The max number of operators, as well as the rules to make churn decisions, are +defined onchain, see details in OperatorSetParam at: +https://github.com/Layr-Labs/eigenlayer-middleware/blob/master/src/interfaces/IBLSRegistryCoordinatorWithIndices.sol#L24. + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| Churn | [ChurnRequest](#churner-ChurnRequest) | [ChurnReply](#churner-ChurnReply) | | + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | +| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | +| double | | double | double | float | float64 | double | float | Float | +| float | | float | float | float | float32 | float | float | Float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | +| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | + diff --git a/api/docs/disperser.md b/api/docs/disperser.md new file mode 100644 index 0000000000..cadc1995bd --- /dev/null +++ b/api/docs/disperser.md @@ -0,0 +1,321 @@ +# Protocol Documentation + + +## Table of Contents + +- [disperser.proto](#disperser-proto) + - [BatchHeader](#disperser-BatchHeader) + - [BatchMetadata](#disperser-BatchMetadata) + - [BlobHeader](#disperser-BlobHeader) + - [BlobInfo](#disperser-BlobInfo) + - [BlobQuorumParam](#disperser-BlobQuorumParam) + - [BlobStatusReply](#disperser-BlobStatusReply) + - [BlobStatusRequest](#disperser-BlobStatusRequest) + - [BlobVerificationProof](#disperser-BlobVerificationProof) + - [DisperseBlobReply](#disperser-DisperseBlobReply) + - [DisperseBlobRequest](#disperser-DisperseBlobRequest) + - [RetrieveBlobReply](#disperser-RetrieveBlobReply) + - [RetrieveBlobRequest](#disperser-RetrieveBlobRequest) + - [SecurityParams](#disperser-SecurityParams) + + - [BlobStatus](#disperser-BlobStatus) + + - [Disperser](#disperser-Disperser) + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## disperser.proto + + + + + +### BatchHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_root | [bytes](#bytes) | | The root of the merkle tree with the hashes of blob headers as leaves. | +| quorum_numbers | [bytes](#bytes) | | All quorums associated with blobs in this batch. | +| quorum_signed_percentages | [bytes](#bytes) | | The percentage of stake that has signed for this batch. The quorum_signed_percentages[i] is percentage for the quorum_numbers[i]. | +| reference_block_number | [uint32](#uint32) | | The Ethereum block number at which the batch was created. The Disperser will encode and disperse the blobs based on the onchain info (e.g. operator stakes) at this block number. | + + + + + + + + +### BatchMetadata + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_header | [BatchHeader](#disperser-BatchHeader) | | | +| signatory_record_hash | [bytes](#bytes) | | The hash of all public keys of the operators that did not sign the batch. | +| fee | [bytes](#bytes) | | The gas fee of confirming this batch. It's the bytes representation of a big.Int value. | +| confirmation_block_number | [uint32](#uint32) | | The Ethereum block number at which the batch is confirmed onchain. | +| batch_header_hash | [bytes](#bytes) | | This is the hash of the ReducedBatchHeader defined onchain, see: https://github.com/Layr-Labs/eigenda/blob/master/contracts/src/interfaces/IEigenDAServiceManager.sol#L43 The is the message that the operators will sign their signatures on. | + + + + + + + + +### BlobHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| commitment | [bytes](#bytes) | | KZG commitment to the blob. | +| data_length | [uint32](#uint32) | | The length of the blob in symbols (each symbol is 31 bytes). | +| blob_quorum_params | [BlobQuorumParam](#disperser-BlobQuorumParam) | repeated | The params of the quorums that this blob participates in. | + + + + + + + + +### BlobInfo +BlobInfo contains information needed to confirm the blob against the EigenDA contracts + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_header | [BlobHeader](#disperser-BlobHeader) | | | +| blob_verification_proof | [BlobVerificationProof](#disperser-BlobVerificationProof) | | | + + + + + + + + +### BlobQuorumParam + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| quorum_number | [uint32](#uint32) | | The ID of the quorum. | +| adversary_threshold_percentage | [uint32](#uint32) | | Same as SecurityParams.adversary_threshold. | +| quorum_threshold_percentage | [uint32](#uint32) | | Same as SecurityParams.quorum_threshold. | +| quantization_param | [uint32](#uint32) | | This determines the nominal number of chunks for the blob, which is nominal_num_chunks = quantization_param * num_operators. A chunk is the smallest unit that's distributed to DA Nodes, corresponding to a set of evaluations of the polynomial (representing the blob) and a KZG multiproof. See more details in data model of EigenDA: https://github.com/Layr-Labs/eigenda/blob/master/docs/spec/data-model.md | +| encoded_length | [uint64](#uint64) | | The length of the blob after encoding (in number of symbols). | + + + + + + + + +### BlobStatusReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| status | [BlobStatus](#disperser-BlobStatus) | | The status of the blob. | +| info | [BlobInfo](#disperser-BlobInfo) | | The blob info needed for clients to confirm the blob against the EigenDA contracts. | + + + + + + + + +### BlobStatusRequest +BlobStatusRequest is used to query the status of a blob. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| request_id | [bytes](#bytes) | | | + + + + + + + + +### BlobVerificationProof + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_id | [uint32](#uint32) | | batch_id is an incremental ID assigned to a batch by EigenDAServiceManager | +| blob_index | [uint32](#uint32) | | The index of the blob in the batch (which is logically an ordered list of blobs). | +| batch_metadata | [BatchMetadata](#disperser-BatchMetadata) | | | +| inclusion_proof | [bytes](#bytes) | | inclusion_proof is a merkle proof for a blob header's inclusion in a batch | +| quorum_indexes | [bytes](#bytes) | | indexes of quorums in BatchHeader.quorum_numbers that match the quorums in BlobHeader.blob_quorum_params Ex. BlobHeader.blob_quorum_params = [ { quorum_number = 0, ... }, { quorum_number = 3, ... }, { quorum_number = 5, ... }, ] BatchHeader.quorum_numbers = [0, 5, 3] => 0x000503 Then, quorum_indexes = [0, 2, 1] => 0x000201 | + + + + + + + + +### DisperseBlobReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| result | [BlobStatus](#disperser-BlobStatus) | | The status of the blob associated with the request_id. | +| request_id | [bytes](#bytes) | | The request ID generated by the disperser. Once a request is accepted (although not processed), a unique request ID will be generated. Two different DisperseBlobRequests (determined by the hash of the DisperseBlobRequest) will have different IDs, and the same DisperseBlobRequest sent repeatedly at different times will also have different IDs. The client should use this ID to query the processing status of the request (via the GetBlobStatus API). | + + + + + + + + +### DisperseBlobRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| data | [bytes](#bytes) | | The data to be dispersed. The size of data must be <= 512KiB. | +| security_params | [SecurityParams](#disperser-SecurityParams) | repeated | Security parameters allowing clients to customize the safety (via adversary threshold) and liveness (via quorum threshold). Clients can define one SecurityParams per quorum, and specify multiple quorums. The disperser will ensure that the encoded blobs for each quorum are all processed within the same batch. | + + + + + + + + +### RetrieveBlobReply +RetrieveBlobReply contains the retrieved blob data + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| data | [bytes](#bytes) | | | + + + + + + + + +### RetrieveBlobRequest +RetrieveBlobRequest contains parameters to retrieve the blob. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_header_hash | [bytes](#bytes) | | | +| blob_index | [uint32](#uint32) | | | + + + + + + + + +### SecurityParams +SecurityParams contains the security parameters for a given quorum. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| quorum_id | [uint32](#uint32) | | The ID of the quorum. The quorum must be already registered on EigenLayer. The ID must be in range [0, 255]. | +| adversary_threshold | [uint32](#uint32) | | The max percentage of stake within the quorum that can be held by or delegated to adversarial operators. + +Clients use this to customize the trust assumption (safety). + +Requires: 1 <= adversary_threshold < 100 | +| quorum_threshold | [uint32](#uint32) | | The min percentage of stake that must attest in order to consider the dispersal is successful. + +Clients use this to customize liveness requirement. The higher this number, the more operators may need to be up for attesting the blob, so the chance the dispersal request to fail may be higher (liveness for dispersal). + +Requires: 1 <= quorum_threshld <= 100 quorum_threshld > adversary_threshold. + +Note: The adversary_threshold and quorum_threshold will directly influence the cost of encoding for the blob to be dispersed, roughly by a factor of 100 / (quorum_threshold - adversary_threshold). See the spec for more details: https://github.com/Layr-Labs/eigenda/blob/master/docs/spec/protocol-modules/storage/overview.md | + + + + + + + + + + +### BlobStatus + + +| Name | Number | Description | +| ---- | ------ | ----------- | +| UNKNOWN | 0 | | +| PROCESSING | 1 | PROCESSING means that the blob is currently being processed by the disperser | +| CONFIRMED | 2 | CONFIRMED means that the blob has been dispersed to DA Nodes and the dispersed batch containing the blob has been confirmed onchain | +| FAILED | 3 | FAILED means that the blob has failed permanently (for reasons other than insufficient signatures, which is a separate state) | +| FINALIZED | 4 | FINALIZED means that the block containing the blob's confirmation transaction has been finalized on Ethereum | +| INSUFFICIENT_SIGNATURES | 5 | INSUFFICIENT_SIGNATURES means that the quorum threshold for the blob was not met for at least one quorum. | + + + + + + + + + +### Disperser +Disperser defines the public APIs for dispersing blobs. + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| DisperseBlob | [DisperseBlobRequest](#disperser-DisperseBlobRequest) | [DisperseBlobReply](#disperser-DisperseBlobReply) | This API accepts blob to disperse from clients. This executes the dispersal async, i.e. it returns once the request is accepted. The client could use GetBlobStatus() API to poll the the processing status of the blob. | +| GetBlobStatus | [BlobStatusRequest](#disperser-BlobStatusRequest) | [BlobStatusReply](#disperser-BlobStatusReply) | This API is meant to be polled for the blob status. | +| RetrieveBlob | [RetrieveBlobRequest](#disperser-RetrieveBlobRequest) | [RetrieveBlobReply](#disperser-RetrieveBlobReply) | This retrieves the requested blob from the Disperser's backend. This is a more efficient way to retrieve blobs than directly retrieving from the DA Nodes (see detail about this approach in api/proto/retriever/retriever.proto). The blob should have been initially dispersed via this Disperser service for this API to work. | + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | +| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | +| double | | double | double | float | float64 | double | float | Float | +| float | | float | float | float | float32 | float | float | Float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | +| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | + diff --git a/api/docs/node.md b/api/docs/node.md new file mode 100644 index 0000000000..390c3c1772 --- /dev/null +++ b/api/docs/node.md @@ -0,0 +1,288 @@ +# Protocol Documentation + + +## Table of Contents + +- [node.proto](#node-proto) + - [BatchHeader](#node-BatchHeader) + - [Blob](#node-Blob) + - [BlobHeader](#node-BlobHeader) + - [BlobQuorumInfo](#node-BlobQuorumInfo) + - [Bundle](#node-Bundle) + - [GetBlobHeaderReply](#node-GetBlobHeaderReply) + - [GetBlobHeaderRequest](#node-GetBlobHeaderRequest) + - [MerkleProof](#node-MerkleProof) + - [RetrieveChunksReply](#node-RetrieveChunksReply) + - [RetrieveChunksRequest](#node-RetrieveChunksRequest) + - [StoreChunksReply](#node-StoreChunksReply) + - [StoreChunksRequest](#node-StoreChunksRequest) + + - [Dispersal](#node-Dispersal) + - [Retrieval](#node-Retrieval) + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## node.proto + + + + + +### BatchHeader +BatchHeader (see core/data.go#BatchHeader) + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_root | [bytes](#bytes) | | The root of the merkle tree with hashes of blob headers as leaves. | +| reference_block_number | [uint32](#uint32) | | The Ethereum block number at which the batch is dispersed. | + + + + + + + + +### Blob +In EigenDA, the original blob to disperse is encoded as a polynomial via taking +taking different point evaluations (i.e. erasure coding). These points are split +into disjoint subsets which are assigned to different operator nodes in the EigenDA +network. +The data in this message is a subset of these points that are assigned to a +single operator node. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| header | [BlobHeader](#node-BlobHeader) | | Which (original) blob this is for. | +| bundles | [Bundle](#node-Bundle) | repeated | Each bundle contains all chunks for a single quorum of the blob. The number of bundles must be equal to the total number of quorums associated with the blob, and the ordering must be the same as BlobHeader.quorum_headers. | + + + + + + + + +### BlobHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| commitment | [bytes](#bytes) | | The KZG commitment to the polynomial representing the blob. | +| length_proof | [bytes](#bytes) | | The low degree proof. It's the KZG commitment to the polynomial shifted to the largest SRS degree. | +| length | [uint32](#uint32) | | The length of the original blob in number of symbols (in the field where the polynomial is defined). | +| quorum_headers | [BlobQuorumInfo](#node-BlobQuorumInfo) | repeated | The params of the quorums that this blob participates in. | +| account_id | [string](#string) | | The ID of the user who is dispersing this blob to EigenDA. | + + + + + + + + +### BlobQuorumInfo +See BlobQuorumParam as defined in +api/proto/disperser/disperser.proto + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| quorum_id | [uint32](#uint32) | | | +| adversary_threshold | [uint32](#uint32) | | | +| quantization_factor | [uint32](#uint32) | | | +| encoded_blob_length | [uint32](#uint32) | | | +| quorum_threshold | [uint32](#uint32) | | | +| ratelimit | [uint32](#uint32) | | | + + + + + + + + +### Bundle +A Bundle is the collection of chunks associated with a single blob, for a single +operator and a single quorum. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| chunks | [bytes](#bytes) | repeated | Each chunk corresponds to a collection of points on the polynomial. Each chunk has same number of points. | + + + + + + + + +### GetBlobHeaderReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_header | [BlobHeader](#node-BlobHeader) | | The header of the blob requested per GetBlobHeaderRequest. | +| proof | [MerkleProof](#node-MerkleProof) | | Merkle proof that returned blob header belongs to the batch and is the batch's MerkleProof.index-th blob. This can be checked against the batch root on chain. | + + + + + + + + +### GetBlobHeaderRequest +See RetrieveChunksRequest for documentation of each parameter of GetBlobHeaderRequest. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_header_hash | [bytes](#bytes) | | | +| blob_index | [uint32](#uint32) | | | +| quorum_id | [uint32](#uint32) | | | + + + + + + + + +### MerkleProof + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| hashes | [bytes](#bytes) | repeated | The proof itself. | +| index | [uint32](#uint32) | | Which index (the leaf of the Merkle tree) this proof is for. | + + + + + + + + +### RetrieveChunksReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| chunks | [bytes](#bytes) | repeated | All chunks the Node is storing for the requested blob per RetrieveChunksRequest. | + + + + + + + + +### RetrieveChunksRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_header_hash | [bytes](#bytes) | | The hash of the ReducedBatchHeader defined onchain, see: https://github.com/Layr-Labs/eigenda/blob/master/contracts/src/interfaces/IEigenDAServiceManager.sol#L43 This identifies which batch to retrieve for. | +| blob_index | [uint32](#uint32) | | Which blob in the batch to retrieve for (note: a batch is logically an ordered list of blobs). | +| quorum_id | [uint32](#uint32) | | Which quorum of the blob to retrieve for (note: a blob can have multiple quorums and the chunks for different quorums at a Node can be different). The ID must be in range [0, 255]. | + + + + + + + + +### StoreChunksReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| signature | [bytes](#bytes) | | The operator's BLS signature signed on the batch header hash. | + + + + + + + + +### StoreChunksRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_header | [BatchHeader](#node-BatchHeader) | | Which batch this request is for. | +| blobs | [Blob](#node-Blob) | repeated | The chunks for each blob in the batch to be stored in an EigenDA Node. | + + + + + + + + + + + + + + +### Dispersal + + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| StoreChunks | [StoreChunksRequest](#node-StoreChunksRequest) | [StoreChunksReply](#node-StoreChunksReply) | StoreChunks validates that the chunks match what the Node is supposed to receive ( different Nodes are responsible for different chunks, as EigenDA is horizontally sharded) and is correctly coded (e.g. each chunk must be a valid KZG multiproof) according to the EigenDA protocol. It also stores the chunks along with metadata for the protocol-defined length of custody. It will return a signature at the end to attest to the data in this request it has processed. | + + + + +### Retrieval + + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| RetrieveChunks | [RetrieveChunksRequest](#node-RetrieveChunksRequest) | [RetrieveChunksReply](#node-RetrieveChunksReply) | RetrieveChunks retrieves the chunks for a blob custodied at the Node. | +| GetBlobHeader | [GetBlobHeaderRequest](#node-GetBlobHeaderRequest) | [GetBlobHeaderReply](#node-GetBlobHeaderReply) | Similar to RetrieveChunks, this just returns the header of the blob. | + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | +| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | +| double | | double | double | float | float64 | double | float | Float | +| float | | float | float | float | float32 | float | float | Float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | +| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | + diff --git a/api/docs/retriever.md b/api/docs/retriever.md new file mode 100644 index 0000000000..412188853c --- /dev/null +++ b/api/docs/retriever.md @@ -0,0 +1,106 @@ +# Protocol Documentation + + +## Table of Contents + +- [retriever.proto](#retriever-proto) + - [BlobReply](#retriever-BlobReply) + - [BlobRequest](#retriever-BlobRequest) + + - [Retriever](#retriever-Retriever) + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## retriever.proto + + + + + +### BlobReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| data | [bytes](#bytes) | | The blob retrieved and reconstructed from the EigenDA Nodes per BlobRequest. | + + + + + + + + +### BlobRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_header_hash | [bytes](#bytes) | | The hash of the ReducedBatchHeader defined onchain, see: https://github.com/Layr-Labs/eigenda/blob/master/contracts/src/interfaces/IEigenDAServiceManager.sol#L43 This identifies the batch that this blob belongs to. | +| blob_index | [uint32](#uint32) | | Which blob in the batch this is requesting for (note: a batch is logically an ordered list of blobs). | +| reference_block_number | [uint32](#uint32) | | The Ethereum block number at which the batch for this blob was constructed. | +| quorum_id | [uint32](#uint32) | | Which quorum of the blob this is requesting for (note a blob can participate in multiple quorums). | + + + + + + + + + + + + + + +### Retriever +The Retriever is a service for retrieving chunks corresponding to a blob from +the EigenDA operator nodes and reconstructing the original blob from the chunks. +This is a client-side library that the users are supposed to operationalize. + +Note: Users generally have two ways to retrieve a blob from EigenDA: + 1) Retrieve from the Disperser that the user initially used for dispersal: the API + is Disperser.RetrieveBlob() as defined in api/proto/disperser/disperser.proto + 2) Retrieve directly from the EigenDA Nodes, which is supported by this Retriever. + +The Disperser.RetrieveBlob() (the 1st approach) is generally faster and cheaper as the +Disperser manages the blobs that it has processed, whereas the Retriever.RetrieveBlob() +(the 2nd approach here) removes the need to trust the Disperser, with the downside of +worse cost and performance. + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| RetrieveBlob | [BlobRequest](#retriever-BlobRequest) | [BlobReply](#retriever-BlobReply) | This fans out request to EigenDA Nodes to retrieve the chunks and returns the reconstructed original blob in response. | + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | +| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | +| double | | double | double | float | float64 | double | float | Float | +| float | | float | float | float | float32 | float | float | Float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | +| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | +