From 6e46b696598a7f691b11659248c03adb40523459 Mon Sep 17 00:00:00 2001 From: Jian Xiao Date: Mon, 2 Dec 2024 20:36:37 +0000 Subject: [PATCH] Update API docs --- api/docs/disperser.md | 69 ++++++++- api/docs/disperserv2.md | 317 ++++++++++++++++++++++++++++++++++++++++ api/docs/lightnode.md | 165 +++++++++++++++++++++ api/docs/node.md | 50 ++++++- api/docs/nodev2.md | 257 ++++++++++++++++++++++++++++++++ api/docs/relay.md | 274 ++++++++++++++++++++++++++++++++++ api/docs/retriever.md | 38 +++++ 7 files changed, 1159 insertions(+), 11 deletions(-) create mode 100644 api/docs/disperserv2.md create mode 100644 api/docs/lightnode.md create mode 100644 api/docs/nodev2.md create mode 100644 api/docs/relay.md diff --git a/api/docs/disperser.md b/api/docs/disperser.md index 6def32751..ce2ab8a6d 100644 --- a/api/docs/disperser.md +++ b/api/docs/disperser.md @@ -18,6 +18,7 @@ - [BlobVerificationProof](#disperser-BlobVerificationProof) - [DisperseBlobReply](#disperser-DisperseBlobReply) - [DisperseBlobRequest](#disperser-DisperseBlobRequest) + - [DispersePaidBlobRequest](#disperser-DispersePaidBlobRequest) - [RetrieveBlobReply](#disperser-RetrieveBlobReply) - [RetrieveBlobRequest](#disperser-RetrieveBlobRequest) @@ -26,7 +27,9 @@ - [Disperser](#disperser-Disperser) - [common/common.proto](#common_common-proto) + - [BlobCommitment](#common-BlobCommitment) - [G1Commitment](#common-G1Commitment) + - [PaymentHeader](#common-PaymentHeader) - [Scalar Value Types](#scalar-value-types) @@ -254,7 +257,7 @@ BlobStatusRequest is used to query the status of a blob. | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | -| result | [BlobStatus](#disperser-BlobStatus) | | The status of the blob associated with the request_id. | +| result | [BlobStatus](#disperser-BlobStatus) | | The status of the blob associated with the request_id. Will always be PROCESSING. | | request_id | [bytes](#bytes) | | The request ID generated by the disperser. Once a request is accepted (although not processed), a unique request ID will be generated. Two different DisperseBlobRequests (determined by the hash of the DisperseBlobRequest) will have different IDs, and the same DisperseBlobRequest sent repeatedly at different times will also have different IDs. The client should use this ID to query the processing status of the request (via the GetBlobStatus API). | @@ -270,7 +273,7 @@ BlobStatusRequest is used to query the status of a blob. | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | -| data | [bytes](#bytes) | | The data to be dispersed. The size of data must be <= 2MiB. Every 32 bytes of data chunk is interpreted as an integer in big endian format where the lower address has more significant bits. The integer must stay in the valid range to be interpreted as a field element on the bn254 curve. The valid range is 0 <= x < 21888242871839275222246405745257275088548364400416034343698204186575808495617 containing slightly less than 254 bits and more than 253 bits. If any one of the 32 bytes chunk is outside the range, the whole request is deemed as invalid, and rejected. | +| data | [bytes](#bytes) | | The data to be dispersed. The size of data must be <= 16MiB. Every 32 bytes of data is interpreted as an integer in big endian format where the lower address has more significant bits. The integer must stay in the valid range to be interpreted as a field element on the bn254 curve. The valid range is 0 <= x < 21888242871839275222246405745257275088548364400416034343698204186575808495617 If any one of the 32 bytes elements is outside the range, the whole request is deemed as invalid, and rejected. | | custom_quorum_numbers | [uint32](#uint32) | repeated | The quorums to which the blob will be sent, in addition to the required quorums which are configured on the EigenDA smart contract. If required quorums are included here, an error will be returned. The disperser will ensure that the encoded blobs for each quorum are all processed within the same batch. | | account_id | [string](#string) | | The account ID of the client. This should be a hex-encoded string of the ECSDA public key corresponding to the key used by the client to sign the BlobAuthHeader. | @@ -279,6 +282,24 @@ BlobStatusRequest is used to query the status of a blob. + + +### DispersePaidBlobRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| data | [bytes](#bytes) | | The data to be dispersed. Same requirements as DisperseBlobRequest. | +| quorum_numbers | [uint32](#uint32) | repeated | The quorums to which the blob to be sent | +| payment_header | [common.PaymentHeader](#common-PaymentHeader) | | Payment header contains AccountID, BinIndex, and CumulativePayment | +| payment_signature | [bytes](#bytes) | | signature of payment_header | + + + + + + ### RetrieveBlobReply @@ -332,10 +353,10 @@ Terminal states are states that will not be updated to a different state: | UNKNOWN | 0 | | | PROCESSING | 1 | PROCESSING means that the blob is currently being processed by the disperser | | CONFIRMED | 2 | CONFIRMED means that the blob has been dispersed to DA Nodes and the dispersed batch containing the blob has been confirmed onchain | -| FAILED | 3 | FAILED means that the blob has failed permanently (for reasons other than insufficient signatures, which is a separate state) | +| FAILED | 3 | FAILED means that the blob has failed permanently (for reasons other than insufficient signatures, which is a separate state). This status is somewhat of a catch-all category, containg (but not necessarily exclusively as errors can be added in the future): - blob has expired - internal logic error while requesting encoding - blob retry has exceeded its limit while waiting for blob finalization after confirmation. Most likely triggered by a chain reorg: see https://github.com/Layr-Labs/eigenda/blob/master/disperser/batcher/finalizer.go#L179-L189. | | FINALIZED | 4 | FINALIZED means that the block containing the blob's confirmation transaction has been finalized on Ethereum | | INSUFFICIENT_SIGNATURES | 5 | INSUFFICIENT_SIGNATURES means that the confirmation threshold for the blob was not met for at least one quorum. | -| DISPERSING | 6 | DISPERSING means that the blob is currently being dispersed to DA Nodes and being confirmed onchain | +| DISPERSING | 6 | The DISPERSING state is comprised of two separate phases: - Dispersing to DA nodes and collecting signature - Submitting the transaction on chain and waiting for tx receipt | @@ -350,7 +371,9 @@ Disperser defines the public APIs for dispersing blobs. | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| -| DisperseBlob | [DisperseBlobRequest](#disperser-DisperseBlobRequest) | [DisperseBlobReply](#disperser-DisperseBlobReply) | This API accepts blob to disperse from clients. This executes the dispersal async, i.e. it returns once the request is accepted. The client could use GetBlobStatus() API to poll the processing status of the blob. | +| DisperseBlob | [DisperseBlobRequest](#disperser-DisperseBlobRequest) | [DisperseBlobReply](#disperser-DisperseBlobReply) | DisperseBlob accepts a single blob to be dispersed. This executes the dispersal async, i.e. it returns once the request is accepted. The client should use GetBlobStatus() API to poll the processing status of the blob. + +If DisperseBlob returns the following error codes: INVALID_ARGUMENT (400): request is invalid for a reason specified in the error msg. RESOURCE_EXHAUSTED (429): request is rate limited for the quorum specified in the error msg. user should retry after the specified duration. INTERNAL (500): serious error, user should NOT retry. | | DisperseBlobAuthenticated | [AuthenticatedRequest](#disperser-AuthenticatedRequest) stream | [AuthenticatedReply](#disperser-AuthenticatedReply) stream | DisperseBlobAuthenticated is similar to DisperseBlob, except that it requires the client to authenticate itself via the AuthenticationData message. The protoco is as follows: 1. The client sends a DisperseBlobAuthenticated request with the DisperseBlobRequest message 2. The Disperser sends back a BlobAuthHeader message containing information for the client to verify and sign. 3. The client verifies the BlobAuthHeader and sends back the signed BlobAuthHeader in an AuthenticationData message. 4. The Disperser verifies the signature and returns a DisperseBlobReply message. | | GetBlobStatus | [BlobStatusRequest](#disperser-BlobStatusRequest) | [BlobStatusReply](#disperser-BlobStatusReply) | This API is meant to be polled for the blob status. | | RetrieveBlob | [RetrieveBlobRequest](#disperser-RetrieveBlobRequest) | [RetrieveBlobReply](#disperser-RetrieveBlobReply) | This retrieves the requested blob from the Disperser's backend. This is a more efficient way to retrieve blobs than directly retrieving from the DA Nodes (see detail about this approach in api/proto/retriever/retriever.proto). The blob should have been initially dispersed via this Disperser service for this API to work. | @@ -366,6 +389,25 @@ Disperser defines the public APIs for dispersing blobs. + + +### BlobCommitment +BlobCommitment represents commitment of a specific blob, containing its +KZG commitment, degree proof, the actual degree, and data length in number of symbols. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| commitment | [bytes](#bytes) | | | +| length_commitment | [bytes](#bytes) | | | +| length_proof | [bytes](#bytes) | | | +| length | [uint32](#uint32) | | | + + + + + + ### G1Commitment @@ -381,6 +423,23 @@ Disperser defines the public APIs for dispersing blobs. + + + +### PaymentHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| account_id | [string](#string) | | | +| bin_index | [uint32](#uint32) | | | +| cumulative_payment | [bytes](#bytes) | | | + + + + + diff --git a/api/docs/disperserv2.md b/api/docs/disperserv2.md new file mode 100644 index 000000000..6f982684c --- /dev/null +++ b/api/docs/disperserv2.md @@ -0,0 +1,317 @@ +# Protocol Documentation + + +## Table of Contents + +- [disperser/v2/disperser_v2.proto](#disperser_v2_disperser_v2-proto) + - [Attestation](#disperser-v2-Attestation) + - [BlobCommitmentReply](#disperser-v2-BlobCommitmentReply) + - [BlobCommitmentRequest](#disperser-v2-BlobCommitmentRequest) + - [BlobStatusReply](#disperser-v2-BlobStatusReply) + - [BlobStatusRequest](#disperser-v2-BlobStatusRequest) + - [BlobVerificationInfo](#disperser-v2-BlobVerificationInfo) + - [DisperseBlobReply](#disperser-v2-DisperseBlobReply) + - [DisperseBlobRequest](#disperser-v2-DisperseBlobRequest) + - [SignedBatch](#disperser-v2-SignedBatch) + + - [BlobStatus](#disperser-v2-BlobStatus) + + - [Disperser](#disperser-v2-Disperser) + +- [common/common.proto](#common_common-proto) + - [BlobCommitment](#common-BlobCommitment) + - [G1Commitment](#common-G1Commitment) + - [PaymentHeader](#common-PaymentHeader) + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## disperser/v2/disperser_v2.proto + + + + + +### Attestation + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| non_signer_pubkeys | [bytes](#bytes) | repeated | Serialized bytes of non signer public keys (G1 points) | +| apk_g2 | [bytes](#bytes) | | Serialized bytes of G2 point that represents aggregate public key of all signers | +| quorum_apks | [bytes](#bytes) | repeated | Serialized bytes of aggregate public keys (G1 points) from all nodes for each quorum | +| sigma | [bytes](#bytes) | | Serialized bytes of aggregate signature | +| quorum_numbers | [uint32](#uint32) | repeated | Relevant quorum numbers for the attestation | + + + + + + + + +### BlobCommitmentReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_commitment | [common.BlobCommitment](#common-BlobCommitment) | | | + + + + + + + + +### BlobCommitmentRequest +Utility method used to generate the commitment of blob given its data. +This can be used to construct BlobHeader.commitment + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| data | [bytes](#bytes) | | | + + + + + + + + +### BlobStatusReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| status | [BlobStatus](#disperser-v2-BlobStatus) | | The status of the blob. | +| signed_batch | [SignedBatch](#disperser-v2-SignedBatch) | | The signed batch | +| blob_verification_info | [BlobVerificationInfo](#disperser-v2-BlobVerificationInfo) | | | + + + + + + + + +### BlobStatusRequest +BlobStatusRequest is used to query the status of a blob. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_key | [bytes](#bytes) | | | + + + + + + + + +### BlobVerificationInfo +BlobVerificationInfo is the information needed to verify the inclusion of a blob in a batch. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_certificate | [common.v2.BlobCertificate](#common-v2-BlobCertificate) | | | +| blob_index | [uint32](#uint32) | | blob_index is the index of the blob in the batch | +| inclusion_proof | [bytes](#bytes) | | inclusion_proof is the inclusion proof of the blob in the batch | + + + + + + + + +### DisperseBlobReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| result | [BlobStatus](#disperser-v2-BlobStatus) | | The status of the blob associated with the blob key. | +| blob_key | [bytes](#bytes) | | | + + + + + + + + +### DisperseBlobRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| data | [bytes](#bytes) | | The data to be dispersed. The size of data must be <= 16MiB. Every 32 bytes of data is interpreted as an integer in big endian format where the lower address has more significant bits. The integer must stay in the valid range to be interpreted as a field element on the bn254 curve. The valid range is 0 <= x < 21888242871839275222246405745257275088548364400416034343698204186575808495617 If any one of the 32 bytes elements is outside the range, the whole request is deemed as invalid, and rejected. | +| blob_header | [common.v2.BlobHeader](#common-v2-BlobHeader) | | | + + + + + + + + +### SignedBatch +SignedBatch is a batch of blobs with a signature. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| header | [common.v2.BatchHeader](#common-v2-BatchHeader) | | header contains metadata about the batch | +| attestation | [Attestation](#disperser-v2-Attestation) | | attestation on the batch | + + + + + + + + + + +### BlobStatus +BlobStatus represents the status of a blob. +The status of a blob is updated as the blob is processed by the disperser. +The status of a blob can be queried by the client using the GetBlobStatus API. +Intermediate states are states that the blob can be in while being processed, and it can be updated to a differet state: +- QUEUED +- ENCODED +Terminal states are states that will not be updated to a different state: +- CERTIFIED +- FAILED +- INSUFFICIENT_SIGNATURES + +| Name | Number | Description | +| ---- | ------ | ----------- | +| UNKNOWN | 0 | | +| QUEUED | 1 | QUEUED means that the blob has been queued by the disperser for processing | +| ENCODED | 2 | ENCODED means that the blob has been encoded and is ready to be dispersed to DA Nodes | +| CERTIFIED | 3 | CERTIFIED means the blob has been dispersed and attested by the DA nodes | +| FAILED | 4 | FAILED means that the blob has failed permanently (for reasons other than insufficient signatures, which is a separate state) | +| INSUFFICIENT_SIGNATURES | 5 | INSUFFICIENT_SIGNATURES means that the confirmation threshold for the blob was not met for at least one quorum. | + + + + + + + + + +### Disperser +Disperser defines the public APIs for dispersing blobs. + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| DisperseBlob | [DisperseBlobRequest](#disperser-v2-DisperseBlobRequest) | [DisperseBlobReply](#disperser-v2-DisperseBlobReply) | DisperseBlob accepts blob to disperse from clients. This executes the dispersal asynchronously, i.e. it returns once the request is accepted. The client could use GetBlobStatus() API to poll the the processing status of the blob. | +| GetBlobStatus | [BlobStatusRequest](#disperser-v2-BlobStatusRequest) | [BlobStatusReply](#disperser-v2-BlobStatusReply) | GetBlobStatus is meant to be polled for the blob status. | +| GetBlobCommitment | [BlobCommitmentRequest](#disperser-v2-BlobCommitmentRequest) | [BlobCommitmentReply](#disperser-v2-BlobCommitmentReply) | GetBlobCommitment is a utility method that calculates commitment for a blob payload. | + + + + + + +

Top

+ +## common/common.proto + + + + + +### BlobCommitment +BlobCommitment represents commitment of a specific blob, containing its +KZG commitment, degree proof, the actual degree, and data length in number of symbols. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| commitment | [bytes](#bytes) | | | +| length_commitment | [bytes](#bytes) | | | +| length_proof | [bytes](#bytes) | | | +| length | [uint32](#uint32) | | | + + + + + + + + +### G1Commitment + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| x | [bytes](#bytes) | | The X coordinate of the KZG commitment. This is the raw byte representation of the field element. | +| y | [bytes](#bytes) | | The Y coordinate of the KZG commitment. This is the raw byte representation of the field element. | + + + + + + + + +### PaymentHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| account_id | [string](#string) | | | +| bin_index | [uint32](#uint32) | | | +| cumulative_payment | [bytes](#bytes) | | | + + + + + + + + + + + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | +| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | +| double | | double | double | float | float64 | double | float | Float | +| float | | float | float | float | float32 | float | float | Float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | +| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | + diff --git a/api/docs/lightnode.md b/api/docs/lightnode.md new file mode 100644 index 000000000..532c8b7ba --- /dev/null +++ b/api/docs/lightnode.md @@ -0,0 +1,165 @@ +# Protocol Documentation + + +## Table of Contents + +- [lightnode/lightnode.proto](#lightnode_lightnode-proto) + - [StreamChunkAvailabilityReply](#lightnode-StreamChunkAvailabilityReply) + - [StreamChunkAvailabilityRequest](#lightnode-StreamChunkAvailabilityRequest) + + - [LightNode](#lightnode-LightNode) + +- [common/common.proto](#common_common-proto) + - [BlobCommitment](#common-BlobCommitment) + - [G1Commitment](#common-G1Commitment) + - [PaymentHeader](#common-PaymentHeader) + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## lightnode/lightnode.proto + + + + + +### StreamChunkAvailabilityReply +A reply to a StreamAvailabilityStatus request. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| header_hash | [bytes](#bytes) | | The hash of a blob header corresponding to a chunk the agent received and verified. From the light node's perspective, the blob is available if all chunks the light node wants to sample are available. | + + + + + + + + +### StreamChunkAvailabilityRequest +A request from a DA node to an agent light node to stream the availability status of all chunks +assigned to the light node. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| authentication_token | [bytes](#bytes) | | | + + + + + + + + + + + + + + +### LightNode + + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| StreamBlobAvailability | [StreamChunkAvailabilityRequest](#lightnode-StreamChunkAvailabilityRequest) | [StreamChunkAvailabilityReply](#lightnode-StreamChunkAvailabilityReply) stream | StreamBlobAvailability streams the availability status blobs from the light node's perspective. A light node considers a blob to be available if all chunks it wants to sample are available. This API is for use by a DA node for monitoring the availability of chunks through its constellation of agent light nodes. | + + + + + + +

Top

+ +## common/common.proto + + + + + +### BlobCommitment +BlobCommitment represents commitment of a specific blob, containing its +KZG commitment, degree proof, the actual degree, and data length in number of symbols. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| commitment | [bytes](#bytes) | | | +| length_commitment | [bytes](#bytes) | | | +| length_proof | [bytes](#bytes) | | | +| length | [uint32](#uint32) | | | + + + + + + + + +### G1Commitment + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| x | [bytes](#bytes) | | The X coordinate of the KZG commitment. This is the raw byte representation of the field element. | +| y | [bytes](#bytes) | | The Y coordinate of the KZG commitment. This is the raw byte representation of the field element. | + + + + + + + + +### PaymentHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| account_id | [string](#string) | | | +| bin_index | [uint32](#uint32) | | | +| cumulative_payment | [bytes](#bytes) | | | + + + + + + + + + + + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | +| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | +| double | | double | double | float | float64 | double | float | Float | +| float | | float | float | float | float32 | float | float | Float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | +| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | + diff --git a/api/docs/node.md b/api/docs/node.md index d36a0b5f8..bd5be0076 100644 --- a/api/docs/node.md +++ b/api/docs/node.md @@ -24,13 +24,15 @@ - [StoreChunksReply](#node-StoreChunksReply) - [StoreChunksRequest](#node-StoreChunksRequest) - - [ChunkEncoding](#node-ChunkEncoding) + - [ChunkEncodingFormat](#node-ChunkEncodingFormat) - [Dispersal](#node-Dispersal) - [Retrieval](#node-Retrieval) - [common/common.proto](#common_common-proto) + - [BlobCommitment](#common-BlobCommitment) - [G1Commitment](#common-G1Commitment) + - [PaymentHeader](#common-PaymentHeader) - [Scalar Value Types](#scalar-value-types) @@ -274,7 +276,7 @@ Node info request | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | | chunks | [bytes](#bytes) | repeated | All chunks the Node is storing for the requested blob per RetrieveChunksRequest. | -| encoding | [ChunkEncoding](#node-ChunkEncoding) | | How the above chunks encoded. | +| chunk_encoding_format | [ChunkEncodingFormat](#node-ChunkEncodingFormat) | | How the above chunks are encoded. | @@ -362,9 +364,9 @@ Node info request - + -### ChunkEncoding +### ChunkEncodingFormat This describes how the chunks returned in RetrieveChunksReply are encoded. Used to facilitate the decoding of chunks. @@ -388,8 +390,8 @@ Used to facilitate the decoding of chunks. | Method Name | Request Type | Response Type | Description | | ----------- | ------------ | ------------- | ------------| | StoreChunks | [StoreChunksRequest](#node-StoreChunksRequest) | [StoreChunksReply](#node-StoreChunksReply) | StoreChunks validates that the chunks match what the Node is supposed to receive ( different Nodes are responsible for different chunks, as EigenDA is horizontally sharded) and is correctly coded (e.g. each chunk must be a valid KZG multiproof) according to the EigenDA protocol. It also stores the chunks along with metadata for the protocol-defined length of custody. It will return a signature at the end to attest to the data in this request it has processed. | -| StoreBlobs | [StoreBlobsRequest](#node-StoreBlobsRequest) | [StoreBlobsReply](#node-StoreBlobsReply) | StoreBlobs is simiar to StoreChunks, but it stores the blobs using a different storage schema so that the stored blobs can later be aggregated by AttestBatch method to a bigger batch. StoreBlobs + AttestBatch will eventually replace and deprecate StoreChunks method. | -| AttestBatch | [AttestBatchRequest](#node-AttestBatchRequest) | [AttestBatchReply](#node-AttestBatchReply) | AttestBatch is used to aggregate the batches stored by StoreBlobs method to a bigger batch. It will return a signature at the end to attest to the aggregated batch. | +| StoreBlobs | [StoreBlobsRequest](#node-StoreBlobsRequest) | [StoreBlobsReply](#node-StoreBlobsReply) | StoreBlobs is simiar to StoreChunks, but it stores the blobs using a different storage schema so that the stored blobs can later be aggregated by AttestBatch method to a bigger batch. StoreBlobs + AttestBatch will eventually replace and deprecate StoreChunks method. DEPRECATED: StoreBlobs method is not used | +| AttestBatch | [AttestBatchRequest](#node-AttestBatchRequest) | [AttestBatchReply](#node-AttestBatchReply) | AttestBatch is used to aggregate the batches stored by StoreBlobs method to a bigger batch. It will return a signature at the end to attest to the aggregated batch. DEPRECATED: AttestBatch method is not used | | NodeInfo | [NodeInfoRequest](#node-NodeInfoRequest) | [NodeInfoReply](#node-NodeInfoReply) | Retrieve node info metadata | @@ -415,6 +417,25 @@ Used to facilitate the decoding of chunks. + + +### BlobCommitment +BlobCommitment represents commitment of a specific blob, containing its +KZG commitment, degree proof, the actual degree, and data length in number of symbols. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| commitment | [bytes](#bytes) | | | +| length_commitment | [bytes](#bytes) | | | +| length_proof | [bytes](#bytes) | | | +| length | [uint32](#uint32) | | | + + + + + + ### G1Commitment @@ -430,6 +451,23 @@ Used to facilitate the decoding of chunks. + + + +### PaymentHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| account_id | [string](#string) | | | +| bin_index | [uint32](#uint32) | | | +| cumulative_payment | [bytes](#bytes) | | | + + + + + diff --git a/api/docs/nodev2.md b/api/docs/nodev2.md new file mode 100644 index 000000000..c882eb258 --- /dev/null +++ b/api/docs/nodev2.md @@ -0,0 +1,257 @@ +# Protocol Documentation + + +## Table of Contents + +- [node/v2/node_v2.proto](#node_v2_node_v2-proto) + - [GetChunksReply](#node-v2-GetChunksReply) + - [GetChunksRequest](#node-v2-GetChunksRequest) + - [NodeInfoReply](#node-v2-NodeInfoReply) + - [NodeInfoRequest](#node-v2-NodeInfoRequest) + - [StoreChunksReply](#node-v2-StoreChunksReply) + - [StoreChunksRequest](#node-v2-StoreChunksRequest) + + - [Dispersal](#node-v2-Dispersal) + - [Retrieval](#node-v2-Retrieval) + +- [common/v2/common.proto](#common_v2_common-proto) + - [Batch](#common-v2-Batch) + - [BatchHeader](#common-v2-BatchHeader) + - [BlobCertificate](#common-v2-BlobCertificate) + - [BlobHeader](#common-v2-BlobHeader) + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## node/v2/node_v2.proto + + + + + +### GetChunksReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| chunks | [bytes](#bytes) | repeated | All chunks the Node is storing for the requested blob per RetrieveChunksRequest. | + + + + + + + + +### GetChunksRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_key | [bytes](#bytes) | | | +| quorum_id | [uint32](#uint32) | | Which quorum of the blob to retrieve for (note: a blob can have multiple quorums and the chunks for different quorums at a Node can be different). The ID must be in range [0, 254]. | + + + + + + + + +### NodeInfoReply +Node info reply + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| semver | [string](#string) | | | +| arch | [string](#string) | | | +| os | [string](#string) | | | +| num_cpu | [uint32](#uint32) | | | +| mem_bytes | [uint64](#uint64) | | | + + + + + + + + +### NodeInfoRequest +Node info request + + + + + + + + +### StoreChunksReply + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| signature | [bytes](#bytes) | | | + + + + + + + + +### StoreChunksRequest + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch | [common.v2.Batch](#common-v2-Batch) | | batch of blobs to store | + + + + + + + + + + + + + + +### Dispersal +WARNING: the following RPCs are experimental and subject to change. + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| StoreChunks | [StoreChunksRequest](#node-v2-StoreChunksRequest) | [StoreChunksReply](#node-v2-StoreChunksReply) | | +| NodeInfo | [NodeInfoRequest](#node-v2-NodeInfoRequest) | [NodeInfoReply](#node-v2-NodeInfoReply) | | + + + + +### Retrieval + + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| GetChunks | [GetChunksRequest](#node-v2-GetChunksRequest) | [GetChunksReply](#node-v2-GetChunksReply) | GetChunks retrieves the chunks for a blob custodied at the Node. | +| NodeInfo | [NodeInfoRequest](#node-v2-NodeInfoRequest) | [NodeInfoReply](#node-v2-NodeInfoReply) | Retrieve node info metadata | + + + + + + +

Top

+ +## common/v2/common.proto + + + + + +### Batch +Batch is a batch of blob certificates + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| header | [BatchHeader](#common-v2-BatchHeader) | | header contains metadata about the batch | +| blob_certificates | [BlobCertificate](#common-v2-BlobCertificate) | repeated | blob_certificates is the list of blob certificates in the batch | + + + + + + + + +### BatchHeader +BatchHeader is the header of a batch of blobs + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_root | [bytes](#bytes) | | batch_root is the root of the merkle tree of the hashes of blob certificates in the batch | +| reference_block_number | [uint64](#uint64) | | reference_block_number is the block number that the state of the batch is based on for attestation | + + + + + + + + +### BlobCertificate +BlobCertificate is what gets attested by the network + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_header | [BlobHeader](#common-v2-BlobHeader) | | | +| relays | [uint32](#uint32) | repeated | | + + + + + + + + +### BlobHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| version | [uint32](#uint32) | | Blob version | +| quorum_numbers | [uint32](#uint32) | repeated | | +| commitment | [common.BlobCommitment](#common-BlobCommitment) | | | +| payment_header | [common.PaymentHeader](#common-PaymentHeader) | | | +| signature | [bytes](#bytes) | | signature over keccak hash of the blob_header that can be verified by blob_header.account_id | + + + + + + + + + + + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | +| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | +| double | | double | double | float | float64 | double | float | Float | +| float | | float | float | float | float32 | float | float | Float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | +| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | + diff --git a/api/docs/relay.md b/api/docs/relay.md new file mode 100644 index 000000000..1ae4113f4 --- /dev/null +++ b/api/docs/relay.md @@ -0,0 +1,274 @@ +# Protocol Documentation + + +## Table of Contents + +- [relay/relay.proto](#relay_relay-proto) + - [ChunkRequest](#node-ChunkRequest) + - [ChunkRequestByIndex](#node-ChunkRequestByIndex) + - [ChunkRequestByRange](#node-ChunkRequestByRange) + - [GetBlobReply](#node-GetBlobReply) + - [GetBlobRequest](#node-GetBlobRequest) + - [GetChunksReply](#node-GetChunksReply) + - [GetChunksRequest](#node-GetChunksRequest) + + - [Relay](#node-Relay) + +- [common/v2/common.proto](#common_v2_common-proto) + - [Batch](#common-v2-Batch) + - [BatchHeader](#common-v2-BatchHeader) + - [BlobCertificate](#common-v2-BlobCertificate) + - [BlobHeader](#common-v2-BlobHeader) + +- [Scalar Value Types](#scalar-value-types) + + + + +

Top

+ +## relay/relay.proto + + + + + +### ChunkRequest +A request for chunks within a specific blob. Requests are fulfilled in all-or-nothing fashion. If any of the +requested chunks are not found or are unable to be fetched, the entire request will fail. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| by_index | [ChunkRequestByIndex](#node-ChunkRequestByIndex) | | Request chunks by their individual indices. | +| by_range | [ChunkRequestByRange](#node-ChunkRequestByRange) | | Request chunks by a range of indices. | + + + + + + + + +### ChunkRequestByIndex +A request for chunks within a specific blob. Each chunk is requested individually by its index. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_key | [bytes](#bytes) | | The blob key. | +| chunk_indices | [uint32](#uint32) | repeated | The index of the chunk within the blob. | + + + + + + + + +### ChunkRequestByRange +A request for chunks within a specific blob. Each chunk is requested a range of indices. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_key | [bytes](#bytes) | | The blob key. | +| start_index | [uint32](#uint32) | | The first index to start fetching chunks from. | +| end_index | [uint32](#uint32) | | One past the last index to fetch chunks from. Similar semantics to golang slices. | + + + + + + + + +### GetBlobReply +The reply to a GetBlobs request. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob | [bytes](#bytes) | | The blob requested. | + + + + + + + + +### GetBlobRequest +A request to fetch one or more blobs. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_key | [bytes](#bytes) | | The key of the blob to fetch. | + + + + + + + + +### GetChunksReply +The reply to a GetChunks request. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| data | [bytes](#bytes) | repeated | The chunks requested. The order of these chunks will be the same as the order of the requested chunks. data is the raw data of the bundle (i.e. serialized byte array of the frames) | + + + + + + + + +### GetChunksRequest +Request chunks from blobs stored by this relay. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| chunk_requests | [ChunkRequest](#node-ChunkRequest) | repeated | The chunk requests. Chunks are returned in the same order as they are requested. | +| operator_id | [bytes](#bytes) | | If this is an authenticated request, this should hold the ID of the operator. If this is an unauthenticated request, this field should be empty. Relays may choose to reject unauthenticated requests. | +| operator_signature | [bytes](#bytes) | | If this is an authenticated request, this field will hold a BLS signature by the requester on the hash of this request. Relays may choose to reject unauthenticated requests. + +The following describes the schema for computing the hash of this request This algorithm is implemented in golang using relay.auth.HashGetChunksRequest(). + +All integers are encoded as unsigned 4 byte big endian values. + +Perform a keccak256 hash on the following data in the following order: 1. the operator id 2. for each chunk request: a. if the chunk request is a request by index: i. a one byte ASCII representation of the character "i" (aka Ox69) ii. the blob key iii. the start index iv. the end index b. if the chunk request is a request by range: i. a one byte ASCII representation of the character "r" (aka Ox72) ii. the blob key iii. each requested chunk index, in order | + + + + + + + + + + + + + + +### Relay +Relay is a service that provides access to public relay functionality. + +| Method Name | Request Type | Response Type | Description | +| ----------- | ------------ | ------------- | ------------| +| GetBlob | [GetBlobRequest](#node-GetBlobRequest) | [GetBlobReply](#node-GetBlobReply) | GetBlob retrieves a blob stored by the relay. | +| GetChunks | [GetChunksRequest](#node-GetChunksRequest) | [GetChunksReply](#node-GetChunksReply) | GetChunks retrieves chunks from blobs stored by the relay. | + + + + + + +

Top

+ +## common/v2/common.proto + + + + + +### Batch +Batch is a batch of blob certificates + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| header | [BatchHeader](#common-v2-BatchHeader) | | header contains metadata about the batch | +| blob_certificates | [BlobCertificate](#common-v2-BlobCertificate) | repeated | blob_certificates is the list of blob certificates in the batch | + + + + + + + + +### BatchHeader +BatchHeader is the header of a batch of blobs + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| batch_root | [bytes](#bytes) | | batch_root is the root of the merkle tree of the hashes of blob certificates in the batch | +| reference_block_number | [uint64](#uint64) | | reference_block_number is the block number that the state of the batch is based on for attestation | + + + + + + + + +### BlobCertificate +BlobCertificate is what gets attested by the network + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| blob_header | [BlobHeader](#common-v2-BlobHeader) | | | +| relays | [uint32](#uint32) | repeated | | + + + + + + + + +### BlobHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| version | [uint32](#uint32) | | Blob version | +| quorum_numbers | [uint32](#uint32) | repeated | | +| commitment | [common.BlobCommitment](#common-BlobCommitment) | | | +| payment_header | [common.PaymentHeader](#common-PaymentHeader) | | | +| signature | [bytes](#bytes) | | signature over keccak hash of the blob_header that can be verified by blob_header.account_id | + + + + + + + + + + + + + + + +## Scalar Value Types + +| .proto Type | Notes | C++ | Java | Python | Go | C# | PHP | Ruby | +| ----------- | ----- | --- | ---- | ------ | -- | -- | --- | ---- | +| double | | double | double | float | float64 | double | float | Float | +| float | | float | float | float | float32 | float | float | Float | +| int32 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| int64 | Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| uint32 | Uses variable-length encoding. | uint32 | int | int/long | uint32 | uint | integer | Bignum or Fixnum (as required) | +| uint64 | Uses variable-length encoding. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum or Fixnum (as required) | +| sint32 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sint64 | Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| fixed32 | Always four bytes. More efficient than uint32 if values are often greater than 2^28. | uint32 | int | int | uint32 | uint | integer | Bignum or Fixnum (as required) | +| fixed64 | Always eight bytes. More efficient than uint64 if values are often greater than 2^56. | uint64 | long | int/long | uint64 | ulong | integer/string | Bignum | +| sfixed32 | Always four bytes. | int32 | int | int | int32 | int | integer | Bignum or Fixnum (as required) | +| sfixed64 | Always eight bytes. | int64 | long | int/long | int64 | long | integer/string | Bignum | +| bool | | bool | boolean | boolean | bool | bool | boolean | TrueClass/FalseClass | +| string | A string must always contain UTF-8 encoded or 7-bit ASCII text. | string | String | str/unicode | string | string | string | String (UTF-8) | +| bytes | May contain any arbitrary sequence of bytes. | string | ByteString | str | []byte | ByteString | string | String (ASCII-8BIT) | + diff --git a/api/docs/retriever.md b/api/docs/retriever.md index 6d56d256d..983536391 100644 --- a/api/docs/retriever.md +++ b/api/docs/retriever.md @@ -10,7 +10,9 @@ - [Retriever](#retriever-Retriever) - [common/common.proto](#common_common-proto) + - [BlobCommitment](#common-BlobCommitment) - [G1Commitment](#common-G1Commitment) + - [PaymentHeader](#common-PaymentHeader) - [Scalar Value Types](#scalar-value-types) @@ -94,6 +96,25 @@ worse cost and performance. + + +### BlobCommitment +BlobCommitment represents commitment of a specific blob, containing its +KZG commitment, degree proof, the actual degree, and data length in number of symbols. + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| commitment | [bytes](#bytes) | | | +| length_commitment | [bytes](#bytes) | | | +| length_proof | [bytes](#bytes) | | | +| length | [uint32](#uint32) | | | + + + + + + ### G1Commitment @@ -109,6 +130,23 @@ worse cost and performance. + + + +### PaymentHeader + + + +| Field | Type | Label | Description | +| ----- | ---- | ----- | ----------- | +| account_id | [string](#string) | | | +| bin_index | [uint32](#uint32) | | | +| cumulative_payment | [bytes](#bytes) | | | + + + + +