From cdf886c65000704cf3d5b8396336520b03ab7263 Mon Sep 17 00:00:00 2001 From: indirection42 Date: Fri, 25 Oct 2024 08:22:51 +0800 Subject: [PATCH 01/12] introduce xcq --- text/introduce-xcq.md | 345 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 345 insertions(+) create mode 100644 text/introduce-xcq.md diff --git a/text/introduce-xcq.md b/text/introduce-xcq.md new file mode 100644 index 000000000..81e844450 --- /dev/null +++ b/text/introduce-xcq.md @@ -0,0 +1,345 @@ +# RFC-0000: Introduce XCQ(Cross Consensus Query) + +| | | +| --------------- | ------------------------------------------------------------------------------------------- | +| **Start Date** | Oct 24 2024 | +| **Description** | Introduce XCQ (Cross Consensus Query) | +| **Authors** | Bryan Chen, Jiyuan Zheng | + +## Summary + +This proposal introduces XCQ (Cross Consensus Query), which is a system aims to reduce code duplication between runtime and UI implementations while providing a unified interface for cross-chain queries. +`XCQ` abstracts away concrete implementations across chains and supports custom query computations. + +Use cases benefiting from `XCQ` include: +- XCM bridge UI: + - Query asset balances + - Query XCM weight and fee from hop and dest chains +- Wallets: + - Query asset balances + - Query weights and fees for operations across chains +- Universal dApp that supports all the parachains: + - Perform Feature discovery + - Query pallet-specific features + - Construct extrinsics by querying pallet index, call index, etc + +## Motivation + +In Substrate, runtime APIs facilitate off-chain clients in reading the state of the consensus system. +However, different chains may expose different APIs or have varying data types, such as differing `AccountId` types. +This diversity extends to client-side, which may require custom computations over runtime APIs in various use cases. +Substrate UIs often access storage directly and reimplement additional logic to convert data into user-friendly representations, leading to duplicated code between Rust runtime logic and UI JS/TS logic. +This duplication increases workload and potential for bugs. + +Therefore, A system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs. + +## Stakeholders + +- Runtime Developers +- Tools/UI Developers + +## Explanation + +The overall query pattern of the XCQ consists of three components: + +- Runtime: View-functions across different pallets are amalgamated through an extension-based system. +- XCQ query: Custom computations over view-function results are encapsulated via PolkaVM programs. +- XCQ query arguments: Query arguments like accounts to be queried are also passed. + +### XCQ Runtime API + +The runtime API for off-chain query usage includes two methods: + +- `execute_query`: Executes the query and returns the result. It takes the query, input, and weight limit as arguments. +The query is the query program in PolkaVM program binary format. +The input is the query arguments that is SCALE-encoded. +The weight limit is the maximum weight allowed for the query execution. +- `metadata`: Return metadata of supported extensions, serving as a feature discovery functionality + +**Example XCQ Runtime API**: +```rust +decl_runtime_apis! { + pub trait XcqApi { + fn execute_query(query: Vec, input: Vec, weight_limit: u64) -> XcqResult; + fn metadata() -> Vec; + } +} +type XcqResult = Result; +type XcqResponse = Vec; +enum XcqError { + Custom(String), +} +``` + +**Example metadata**: +```rust +Metadata { + extensions: vec![ + ExtensionMetadata { + name: "ExtensionCore", + methods: vec![MethodMetadata { + name: "has_extension", + inputs: vec![MethodParamMetadata { + name: "id", + ty: XcqType::Primitive(PrimitiveType::U64) + }], + output: XcqType::Primitive(PrimitiveType::Bool) + }] + }, + ExtensionMetadata { + name: "ExtensionFungibles", + methods: vec![ + MethodMetadata { + name: "total_supply", + inputs: vec![MethodParamMetadata { + name: "asset", + ty: XcqType::Primitive(PrimitiveType::U32) + }], + output: XcqType::Primitive(PrimitiveType::U64) + }, + MethodMetadata { + name: "balance", + inputs: vec![ + MethodParamMetadata { + name: "asset", + ty: XcqType::Primitive(PrimitiveType::U32) + }, + MethodParamMetadata { + name: "who", + ty: XcqType::Primitive(PrimitiveType::H256) + } + ], + output: XcqType::Primitive(PrimitiveType::U64) + } + ] + } + ] +} +``` + +Note: `ty` is represented by a meta-type system called `xcq-types` + +#### xcq-types + +`xcq-types` is a meta-type system similar to `scale-info` but simpler. +It enables different chains with different type definitions to work via a common operation. +Front-end codes constructs call data to XCQ programs according to metadata provided by different chains. + +### XCQ Executor + +An XCQ executor is a runtime module that executes XCQ queries. +It has a core method `execute` that takes a PolkaVM program binary, +method name of the exported functions in the PolkaVM program, input arguments, and weight limit that the PolkaVM program can consume. + +```rust +pub fn execute( + &mut self, + raw_blob: &[u8], + method: &str, + input: &[u8], + weight_limit: u64, +) -> Result, XcqExecutorError> {...} +``` + +### XCQ Extension + +An extension-based design is essential for several reasons: +- Diffent chains may have different data types for semantically similar queries, making it challenging to standardize function calls across them. +An extension-based design with optional associated types allows these diverse data types to be specified and utilized effectively. +- Function calls distributed across various pallets can be amalgamated into a single extension, simplifying the development process and ensuring a more cohesive and maintainable codebase. +- Extensions are identified via an extension ID, a hash based on the extension name and method sets. An update to an extension is treated as a new extension. +- New functionalities can be added without upgrading the core part of the XCQ. +- Ensure the core part is in a minimal scope. + +Essential components of an XCQ extension system include: + +- `decl_extension` macro: Defines an extension as a Rust trait with optional associated types. + +**Example usage**: +```rust +use xcq_extension::decl_extension; + +pub trait Config { + type AssetId: Codec; + type AccountId: Codec; + type Balance: Codec; +} +decl_extensions! { + pub trait ExtensionFungibles { + type Config: Config; + fn total_supply(asset: ::AssetId) -> ::Balance; + fn balance(asset: ::AssetId, who: ::AccountId) -> ::Balance; + } +} +``` + +- `impl_extensions` macro: Generates extension implementations and extension-level metadata. + +**Example Usage**: + +```rust +// ExtensionImpl is an aggregate struct to impl different extensions +struct ExtensionImpl; +impl extension_fungibles::Config for ExtensionImpl { + type AssetId = u32; + type AccountId = [u8; 32]; + type Balance = u64; +} +impl_extensions! { + impl extension_core::ExtensionCore for ExtensionImpl { + type Config = ExtensionImpl; + fn has_extension(id: ::ExtensionId) -> bool { + matches!(id, 0 | 1) + } + } + + impl extension_fungibles::ExtensionFungibles for ExtensionImpl { + type Config = ExtensionImpl; + #[allow(unused_variables)] + fn total_supply(asset: ::AssetId) -> ::Balance { + 200 + } + #[allow(unused_variables)] + fn balance(asset: ::AssetId, who: ::AccountId) -> ::Balance { + 100 + } + } +} +``` + +- `ExtensionExecutor`: Connects extension implementations and `xcq-executor`. +All methods of all extensions that a chain supports are amalgamated into a single `host_call` entry. +Then this entry is registered as a typed function entry in PolkaVM Linker within the `xcq-executor`. +Given the extension ID and call data encoded in SCALE format, call requests from the guest XCQ program are dispatched to corresponding extensions: +```rust +linker + .define_typed( + "host_call", + move |caller: Caller<'_, Self::UserData>, + extension_id: u64, + call_ptr: u32, + call_len: u32| + -> Result { + ... + }); +``` +- `PermController`: Filters guest XCQ program calling requests, useful for host chains to disable some queries by filtering invoking sources. +```rust +pub trait PermController { + fn is_allowed(extension_id: ExtensionIdTy, call: &[u8], source: InvokeSource) -> bool; +} +#[derive(Copy, Clone)] +pub enum InvokeSource { + RuntimeAPI, + XCM, + Extrinsic, + Runtime, +} +``` + +### XCQ Program Structure + +An XCQ program is structured as a PolkaVM program with the following key components: + +- Imported Functions: + - `host_call`: Dispatches call requests to the XCQ Extension Executor. + ```rust + #[polkavm_derive::polkavm_import] + extern "C" { + fn host_call(extension_id: u64, call_ptr: u32, call_len: u32) -> u64; + } + ``` + Results are SCALE-encoded bytes, with the pointer address (lower 32 bits) and length (higher 32 bits) packed into a u64. + + - `return_ty`: Returns the type of the function call result. + ```rust + #[polkavm_derive::polkavm_import] + extern "C" { + fn return_ty(extension_id: u64, call_index: u32) -> u64; + } + ``` + Results are SCALE-encoded bytes, with the pointer address and length packed similarly to `host_call`. + +- Exported Functions: + - `main`: The entry point of the XCQ program. It performs type checking and executes the query. + +### XCQ Program Execution Flow + +The interaction between an XCQ program and the XCQ Extension Executor follows these steps: + +1. Program Loading: The Executor loads the PolkaVM program binary. + +2. Environment Setup: The Executor configures the PolkaVM environment, registering host functions like `host_call` and `return_ty`. + +3. Main Function Execution: The Executor calls the program's `main` function, passing serialized query arguments. + +4. Program Execution: + a. Type Checking: The program uses the `return_ty` function to ensure compatibility with supported chain extensions. + b. Query Execution: The program executes the query using `host_call` and performs custom computations. + c. Result Serialization: The program serializes the result, writes it to shared memory, and returns the pointer and length to the executor. + +5. Result Retrieval: The Executor reads the result from shared memory and returns it to the caller. + +This structure allows for flexible, type-safe querying across different blockchain implementations while maintaining a consistent interface for developers. + +## Drawbacks + +### Performance issues + +- XCQ Query Program Size: The size of XCQ query programs should be optimized to ensure efficient storage and transmission via XCMP/HRMP. +The specification should define a target size limit for query programs and outline strategies for reducing program size without compromising functionality. This may include techniques such as: + - Exploring modular program structures that allow for separate storage and transmission of core logic and supporting elements. PolkaVM supports spliting the program into multiple modules. + - Establishing guidelines for optimizing dynamic memory usage within query programs + +### User experience issues + +- Debugging: Currently, there is no full-fledged debuggers for PolkaVM programs. The only debugging approach is to set the PolkaVM backend in interpreter mode and then log the operations at the assembly level, which is too low-level to debug efficiently. +- Gas computation: According to [this issue](https://github.com/koute/polkavm/issues/17), the gas cost model of PolkaVM is not accurate for now. + +## Testing, Security, and Privacy + +- Testing: + - A comprehensive test suite should be developed to cover various scenarios: + - Positive test cases: + - Basic queries with various extensions, data types, return values, custom computations, etc. + - Accurate conversion between given weight limit and the gas limit of PolkaVM + - Negative test cases: + - Queries exceeding weight limits + - Invoking queries from unauthorized sources + - Edge cases: + - Queries with minimal or maximum allowed input sizes + - Queries requesting data at the boundaries of available ranges + - Integration tests to ensure proper interaction with off-chain wallets/UI and on-chain XCM, including the aforementioned use cases in Motivation section. + +- Security: + - The XCQ system must enforce a strict read-only policy for all query operations. A mechanism should be implemented to prevent any state-changing operations within XCQ queries. + - The security model should include measures to prevent potential attack vectors such as resource exhaustion or malicious query injection. + - Clear guidelines and best practices should be provided for parachain developers to ensure secure implementation. + +## Performance, Ergonomics, and Compatibility + +### Performance + +It's a new functionality, which doesn't modify the existing implementations. + +### Ergonomics + +The proposal facilitate the wallets and dApps developers. Developers no longer need to examine every concrete implementation to support conceptually similar operations across different chains. Additionally, they gain a more modular development experience through encapsulating custom computations over the exposed APIs in PolkaVM programs. +### Compatibility + +The proposal defines new apis, which doesn't break compatibility with existing interfaces. + +## Prior Art and References + +There are several discussions related to the proposal, including: + +- is the original discussion about having a mechanism to avoid code duplications between the runtime and front-ends/wallets. In the original design, the custom computations are compiled as a wasm function. +- is the issue tracking the view functions implementation in runtime implementations +- is the on-going `view function` pull request. It works at pallet level. If two chains use two different pallets to provide similar functionalities, like pallet-assets and pallet-erc20, we still need to have different codes to support. Therefore, it doesn't conflict with XCQ, and can be utilized by XCQ. + +## Unresolved Questions + + +## Future Directions and Related Material + +Since XCQ are supported both in off-chain and on-chain, a related XCM-Format RFC should be proposed. From ae88d8d5db2f7ca99175cf05982026d9d0c304e9 Mon Sep 17 00:00:00 2001 From: indirection42 Date: Fri, 25 Oct 2024 09:13:51 +0800 Subject: [PATCH 02/12] update --- text/introduce-xcq.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/text/introduce-xcq.md b/text/introduce-xcq.md index 81e844450..332a5e9e7 100644 --- a/text/introduce-xcq.md +++ b/text/introduce-xcq.md @@ -8,7 +8,7 @@ ## Summary -This proposal introduces XCQ (Cross Consensus Query), which is a system aims to reduce code duplication between runtime and UI implementations while providing a unified interface for cross-chain queries. +This proposal introduces XCQ (Cross Consensus Query), which aims to serve as an intermediary layer between different chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries. `XCQ` abstracts away concrete implementations across chains and supports custom query computations. Use cases benefiting from `XCQ` include: @@ -26,12 +26,12 @@ Use cases benefiting from `XCQ` include: ## Motivation In Substrate, runtime APIs facilitate off-chain clients in reading the state of the consensus system. -However, different chains may expose different APIs or have varying data types, such as differing `AccountId` types. -This diversity extends to client-side, which may require custom computations over runtime APIs in various use cases. -Substrate UIs often access storage directly and reimplement additional logic to convert data into user-friendly representations, leading to duplicated code between Rust runtime logic and UI JS/TS logic. +However, different chains may expose different APIs for a similar query or have varying data types, such as doing custom transformations on direct data, or differing `AccountId` types. +This diversity also extends to client-side, which may require custom computations over runtime APIs in various use cases. +Therefore, tools and UI developers often access storage directly and reimplement custom computations to convert data into user-friendly representations, leading to duplicated code between Rust runtime logic and UI JS/TS logic. This duplication increases workload and potential for bugs. -Therefore, A system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs. +Therefore, A system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries. ## Stakeholders From 62174b193046c1b3af981865a957810051150d55 Mon Sep 17 00:00:00 2001 From: indirection42 Date: Fri, 25 Oct 2024 09:38:35 +0800 Subject: [PATCH 03/12] update --- text/introduce-xcq.md | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/text/introduce-xcq.md b/text/introduce-xcq.md index 332a5e9e7..5974e20a6 100644 --- a/text/introduce-xcq.md +++ b/text/introduce-xcq.md @@ -44,7 +44,7 @@ The overall query pattern of the XCQ consists of three components: - Runtime: View-functions across different pallets are amalgamated through an extension-based system. - XCQ query: Custom computations over view-function results are encapsulated via PolkaVM programs. -- XCQ query arguments: Query arguments like accounts to be queried are also passed. +- XCQ query arguments: Query arguments like accounts to be queried are also passed together with the query program. ### XCQ Runtime API @@ -54,7 +54,7 @@ The runtime API for off-chain query usage includes two methods: The query is the query program in PolkaVM program binary format. The input is the query arguments that is SCALE-encoded. The weight limit is the maximum weight allowed for the query execution. -- `metadata`: Return metadata of supported extensions, serving as a feature discovery functionality +- `metadata`: Return metadata of supported extensions (introduced in later section) and methods, serving as a feature discovery functionality **Example XCQ Runtime API**: ```rust @@ -144,15 +144,16 @@ pub fn execute( ### XCQ Extension An extension-based design is essential for several reasons: -- Diffent chains may have different data types for semantically similar queries, making it challenging to standardize function calls across them. +- Different chains may have different data types for semantically similar queries, making it challenging to standardize function calls across them. An extension-based design with optional associated types allows these diverse data types to be specified and utilized effectively. - Function calls distributed across various pallets can be amalgamated into a single extension, simplifying the development process and ensuring a more cohesive and maintainable codebase. -- Extensions are identified via an extension ID, a hash based on the extension name and method sets. An update to an extension is treated as a new extension. - New functionalities can be added without upgrading the core part of the XCQ. - Ensure the core part is in a minimal scope. Essential components of an XCQ extension system include: +- A hash-based extension id generation mechanism for addressing and versioning . The hash value derives from the extension name and its method sets. Any update to an extension is treated as a new extension. + - `decl_extension` macro: Defines an extension as a Rust trait with optional associated types. **Example usage**: @@ -164,7 +165,7 @@ pub trait Config { type AccountId: Codec; type Balance: Codec; } -decl_extensions! { +decl_extension! { pub trait ExtensionFungibles { type Config: Config; fn total_supply(asset: ::AssetId) -> ::Balance; @@ -261,7 +262,7 @@ An XCQ program is structured as a PolkaVM program with the following key compone Results are SCALE-encoded bytes, with the pointer address and length packed similarly to `host_call`. - Exported Functions: - - `main`: The entry point of the XCQ program. It performs type checking and executes the query. + - `main`: The entry point of the XCQ program. It performs type checking, arguments and results passing and executes the query. ### XCQ Program Execution Flow @@ -274,20 +275,18 @@ The interaction between an XCQ program and the XCQ Extension Executor follows th 3. Main Function Execution: The Executor calls the program's `main` function, passing serialized query arguments. 4. Program Execution: - a. Type Checking: The program uses the `return_ty` function to ensure compatibility with supported chain extensions. - b. Query Execution: The program executes the query using `host_call` and performs custom computations. - c. Result Serialization: The program serializes the result, writes it to shared memory, and returns the pointer and length to the executor. + 1. Type Checking: The program uses the `return_ty` function to ensure compatibility with supported chain extensions. + 2. Query Execution: The program executes the query using `host_call` and performs custom computations. + 3. Result Serialization: The program serializes the result, writes it to shared memory, and returns the pointer and length to the executor. 5. Result Retrieval: The Executor reads the result from shared memory and returns it to the caller. -This structure allows for flexible, type-safe querying across different blockchain implementations while maintaining a consistent interface for developers. - ## Drawbacks ### Performance issues - XCQ Query Program Size: The size of XCQ query programs should be optimized to ensure efficient storage and transmission via XCMP/HRMP. -The specification should define a target size limit for query programs and outline strategies for reducing program size without compromising functionality. This may include techniques such as: +Some strategies to address this issue include: - Exploring modular program structures that allow for separate storage and transmission of core logic and supporting elements. PolkaVM supports spliting the program into multiple modules. - Establishing guidelines for optimizing dynamic memory usage within query programs @@ -308,12 +307,10 @@ The specification should define a target size limit for query programs and outli - Invoking queries from unauthorized sources - Edge cases: - Queries with minimal or maximum allowed input sizes - - Queries requesting data at the boundaries of available ranges - - Integration tests to ensure proper interaction with off-chain wallets/UI and on-chain XCM, including the aforementioned use cases in Motivation section. + - Integration tests to ensure proper interaction with off-chain wallets/UI and on-chain XCM, including the aforementioned use cases in **Motivation** section. - Security: - The XCQ system must enforce a strict read-only policy for all query operations. A mechanism should be implemented to prevent any state-changing operations within XCQ queries. - - The security model should include measures to prevent potential attack vectors such as resource exhaustion or malicious query injection. - Clear guidelines and best practices should be provided for parachain developers to ensure secure implementation. ## Performance, Ergonomics, and Compatibility @@ -334,7 +331,7 @@ The proposal defines new apis, which doesn't break compatibility with existing i There are several discussions related to the proposal, including: - is the original discussion about having a mechanism to avoid code duplications between the runtime and front-ends/wallets. In the original design, the custom computations are compiled as a wasm function. -- is the issue tracking the view functions implementation in runtime implementations +- is the issue tracking the `view functions` implementation in runtime implementations - is the on-going `view function` pull request. It works at pallet level. If two chains use two different pallets to provide similar functionalities, like pallet-assets and pallet-erc20, we still need to have different codes to support. Therefore, it doesn't conflict with XCQ, and can be utilized by XCQ. ## Unresolved Questions From fa97e4a8cb49759cd32894f7e499283a5f6cf5cf Mon Sep 17 00:00:00 2001 From: indirection42 Date: Fri, 25 Oct 2024 09:40:48 +0800 Subject: [PATCH 04/12] update rfc number --- text/{introduce-xcq.md => 0126-introduce-xcq.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename text/{introduce-xcq.md => 0126-introduce-xcq.md} (99%) diff --git a/text/introduce-xcq.md b/text/0126-introduce-xcq.md similarity index 99% rename from text/introduce-xcq.md rename to text/0126-introduce-xcq.md index 5974e20a6..296310904 100644 --- a/text/introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -1,8 +1,8 @@ -# RFC-0000: Introduce XCQ(Cross Consensus Query) +# RFC-0126: Introduce XCQ(Cross Consensus Query) | | | | --------------- | ------------------------------------------------------------------------------------------- | -| **Start Date** | Oct 24 2024 | +| **Start Date** | Oct 25 2024 | | **Description** | Introduce XCQ (Cross Consensus Query) | | **Authors** | Bryan Chen, Jiyuan Zheng | From 740f5e0c0da5754d6294b6fd43b421fc60725131 Mon Sep 17 00:00:00 2001 From: indirection42 Date: Tue, 29 Oct 2024 19:55:14 +0800 Subject: [PATCH 05/12] add xcm section --- text/0126-introduce-xcq.md | 48 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/text/0126-introduce-xcq.md b/text/0126-introduce-xcq.md index 296310904..def901c1e 100644 --- a/text/0126-introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -31,7 +31,7 @@ This diversity also extends to client-side, which may require custom computation Therefore, tools and UI developers often access storage directly and reimplement custom computations to convert data into user-friendly representations, leading to duplicated code between Rust runtime logic and UI JS/TS logic. This duplication increases workload and potential for bugs. -Therefore, A system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries. +Therefore, a system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries. ## Stakeholders @@ -40,7 +40,7 @@ Therefore, A system is needed to serve as an intermediary layer between concrete ## Explanation -The overall query pattern of the XCQ consists of three components: +The overall query pattern of XCQ consists of three components: - Runtime: View-functions across different pallets are amalgamated through an extension-based system. - XCQ query: Custom computations over view-function results are encapsulated via PolkaVM programs. @@ -281,6 +281,50 @@ The interaction between an XCQ program and the XCQ Extension Executor follows th 5. Result Retrieval: The Executor reads the result from shared memory and returns it to the caller. +### XCM integration + +XCM usages is considered in XCQ. The integration is acheived by adding a new instruction to XCM, and a new variant is added to the `Response` type in `QueryResponse` message.: + +- A new `ReportQuery` instruction + +```rust +ReportQuery { + query: SizeLimitedXcq, + weight_limit: Option, + info: QueryResponseInfo, +} +``` + +Report to a given destination the results of an XCQ query. After query, a `QueryResponse` message of type `XcqResult` will be sent to the described destination. + +Operands: + +- `query: SizeLimitedXcq` has a size limit(2MB) + - `program: Vec`: A pre-built PVM program binary. + - `input: Vec`: The arguments of the program. +- `weight_limit: WeightLimit`: The maximum weight that the query should take. `WeightLimit` is an enum that can specify either `Limit(Weight)` or `Unlimited`. +- `info: QueryResponseInfo`: Information for making the response. + - `destination: Location`: The destination to which the query response message should be sent. + - `query_id: Compact`: The `query_id` field of the `QueryResponse` message + - `max_weight: Weight`: The `max_weight` field of the `QueryResponse` message + +- Add a new variant to the `Response` type in `QueryResponse` + +- `XcqResult = 6 (XcqResult)` + `XcqResult` is a enum + - `Ok = 0 (Vec)`: XCQ executes successfully with a SCALE-encoded response. + - `Err = 1 (ErrorCode)`: XCQ fails with an error code. +`ErrorCode` is a enum +- `ExceedMaxWeight = 0` +- `MemoryAllocationError = 1` +- `MemoryAccessError = 2` +- `CallError = 3` +- `OtherPVMError = 4` + +#### Errors +- `BadOrigin` +- `DestinationUnsupported` + ## Drawbacks ### Performance issues From bff10672bdf03f0fdee7948bb0ce2b7b47f65a4d Mon Sep 17 00:00:00 2001 From: indirection42 Date: Tue, 29 Oct 2024 20:02:29 +0800 Subject: [PATCH 06/12] rename PermissionController --- text/0126-introduce-xcq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0126-introduce-xcq.md b/text/0126-introduce-xcq.md index def901c1e..d02d8603d 100644 --- a/text/0126-introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -224,9 +224,9 @@ linker ... }); ``` -- `PermController`: Filters guest XCQ program calling requests, useful for host chains to disable some queries by filtering invoking sources. +- `PermissionController`: Filters guest XCQ program calling requests, useful for host chains to disable some queries by filtering invoking sources. ```rust -pub trait PermController { +pub trait PermissionController { fn is_allowed(extension_id: ExtensionIdTy, call: &[u8], source: InvokeSource) -> bool; } #[derive(Copy, Clone)] From 3678eee1f03d08dad02257f64a8d4cb469ab11c0 Mon Sep 17 00:00:00 2001 From: indirection42 Date: Wed, 30 Oct 2024 09:38:54 +0800 Subject: [PATCH 07/12] update --- text/0126-introduce-xcq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0126-introduce-xcq.md b/text/0126-introduce-xcq.md index d02d8603d..59694e347 100644 --- a/text/0126-introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -283,7 +283,7 @@ The interaction between an XCQ program and the XCQ Extension Executor follows th ### XCM integration -XCM usages is considered in XCQ. The integration is acheived by adding a new instruction to XCM, and a new variant is added to the `Response` type in `QueryResponse` message.: +The integration of XCQ into XCM is acheived by adding a new instruction to XCM, as well as a new variant of the `Response` type in `QueryResponse` message.: - A new `ReportQuery` instruction From 220f62d146a4043de9d103832776b0802a584120 Mon Sep 17 00:00:00 2001 From: indirection42 Date: Wed, 30 Oct 2024 10:18:15 +0800 Subject: [PATCH 08/12] add read-only execution suggestion --- text/0126-introduce-xcq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0126-introduce-xcq.md b/text/0126-introduce-xcq.md index 59694e347..cb8619cd0 100644 --- a/text/0126-introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -354,7 +354,7 @@ Some strategies to address this issue include: - Integration tests to ensure proper interaction with off-chain wallets/UI and on-chain XCM, including the aforementioned use cases in **Motivation** section. - Security: - - The XCQ system must enforce a strict read-only policy for all query operations. A mechanism should be implemented to prevent any state-changing operations within XCQ queries. + - The XCQ system must enforce a strict read-only policy for all query operations. A mechanism should be implemented to prevent any state-changing operations within XCQ queries. For example, perform a final rollback in `frame_support::storage::with_transaction` to ensure the storage won't be changed. - Clear guidelines and best practices should be provided for parachain developers to ensure secure implementation. ## Performance, Ergonomics, and Compatibility From 6c637c3f700c67574b726bbe62b09f09545eda2c Mon Sep 17 00:00:00 2001 From: indirection42 Date: Tue, 5 Nov 2024 09:19:04 +0800 Subject: [PATCH 09/12] merge xcm rfc --- text/0126-introduce-xcq.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/text/0126-introduce-xcq.md b/text/0126-introduce-xcq.md index cb8619cd0..efaf0f276 100644 --- a/text/0126-introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -382,5 +382,3 @@ There are several discussions related to the proposal, including: ## Future Directions and Related Material - -Since XCQ are supported both in off-chain and on-chain, a related XCM-Format RFC should be proposed. From 69e3320849a14cf034be411e80106fba6e5129af Mon Sep 17 00:00:00 2001 From: indirection42 Date: Fri, 6 Dec 2024 10:23:02 +0800 Subject: [PATCH 10/12] update metadata representation --- text/0126-introduce-xcq.md | 78 ++++++++++++-------------------------- 1 file changed, 24 insertions(+), 54 deletions(-) diff --git a/text/0126-introduce-xcq.md b/text/0126-introduce-xcq.md index efaf0f276..6f42526a0 100644 --- a/text/0126-introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -12,6 +12,7 @@ This proposal introduces XCQ (Cross Consensus Query), which aims to serve as an `XCQ` abstracts away concrete implementations across chains and supports custom query computations. Use cases benefiting from `XCQ` include: + - XCM bridge UI: - Query asset balances - Query XCM weight and fee from hop and dest chains @@ -54,9 +55,11 @@ The runtime API for off-chain query usage includes two methods: The query is the query program in PolkaVM program binary format. The input is the query arguments that is SCALE-encoded. The weight limit is the maximum weight allowed for the query execution. -- `metadata`: Return metadata of supported extensions (introduced in later section) and methods, serving as a feature discovery functionality +- `metadata`: Return metadata of supported extensions (introduced in later section) and methods, serving as a feature discovery functionality. +The representation and encoding mechanism is similar to the [`frame-metadata`](https://github.com/paritytech/frame-metadata/), using `scale-info`. **Example XCQ Runtime API**: + ```rust decl_runtime_apis! { pub trait XcqApi { @@ -71,60 +74,15 @@ enum XcqError { } ``` -**Example metadata**: +**Example Metadata (before SCALE-encoded)** + ```rust -Metadata { - extensions: vec![ - ExtensionMetadata { - name: "ExtensionCore", - methods: vec![MethodMetadata { - name: "has_extension", - inputs: vec![MethodParamMetadata { - name: "id", - ty: XcqType::Primitive(PrimitiveType::U64) - }], - output: XcqType::Primitive(PrimitiveType::Bool) - }] - }, - ExtensionMetadata { - name: "ExtensionFungibles", - methods: vec![ - MethodMetadata { - name: "total_supply", - inputs: vec![MethodParamMetadata { - name: "asset", - ty: XcqType::Primitive(PrimitiveType::U32) - }], - output: XcqType::Primitive(PrimitiveType::U64) - }, - MethodMetadata { - name: "balance", - inputs: vec![ - MethodParamMetadata { - name: "asset", - ty: XcqType::Primitive(PrimitiveType::U32) - }, - MethodParamMetadata { - name: "who", - ty: XcqType::Primitive(PrimitiveType::H256) - } - ], - output: XcqType::Primitive(PrimitiveType::U64) - } - ] - } - ] +pub struct Metadata { + pub types: PortableRegistry, + pub extensions: Vec>, } ``` -Note: `ty` is represented by a meta-type system called `xcq-types` - -#### xcq-types - -`xcq-types` is a meta-type system similar to `scale-info` but simpler. -It enables different chains with different type definitions to work via a common operation. -Front-end codes constructs call data to XCQ programs according to metadata provided by different chains. - ### XCQ Executor An XCQ executor is a runtime module that executes XCQ queries. @@ -144,6 +102,7 @@ pub fn execute( ### XCQ Extension An extension-based design is essential for several reasons: + - Different chains may have different data types for semantically similar queries, making it challenging to standardize function calls across them. An extension-based design with optional associated types allows these diverse data types to be specified and utilized effectively. - Function calls distributed across various pallets can be amalgamated into a single extension, simplifying the development process and ensuring a more cohesive and maintainable codebase. @@ -157,6 +116,7 @@ Essential components of an XCQ extension system include: - `decl_extension` macro: Defines an extension as a Rust trait with optional associated types. **Example usage**: + ```rust use xcq_extension::decl_extension; @@ -212,6 +172,7 @@ impl_extensions! { All methods of all extensions that a chain supports are amalgamated into a single `host_call` entry. Then this entry is registered as a typed function entry in PolkaVM Linker within the `xcq-executor`. Given the extension ID and call data encoded in SCALE format, call requests from the guest XCQ program are dispatched to corresponding extensions: + ```rust linker .define_typed( @@ -224,7 +185,9 @@ linker ... }); ``` + - `PermissionController`: Filters guest XCQ program calling requests, useful for host chains to disable some queries by filtering invoking sources. + ```rust pub trait PermissionController { fn is_allowed(extension_id: ExtensionIdTy, call: &[u8], source: InvokeSource) -> bool; @@ -243,26 +206,30 @@ pub enum InvokeSource { An XCQ program is structured as a PolkaVM program with the following key components: - Imported Functions: - - `host_call`: Dispatches call requests to the XCQ Extension Executor. + - `host_call`: Dispatches call requests to the XCQ Extension Executor. + ```rust #[polkavm_derive::polkavm_import] extern "C" { fn host_call(extension_id: u64, call_ptr: u32, call_len: u32) -> u64; } ``` + Results are SCALE-encoded bytes, with the pointer address (lower 32 bits) and length (higher 32 bits) packed into a u64. - - `return_ty`: Returns the type of the function call result. + - `return_ty`: Returns the type of the function call result. + ```rust #[polkavm_derive::polkavm_import] extern "C" { fn return_ty(extension_id: u64, call_index: u32) -> u64; } ``` + Results are SCALE-encoded bytes, with the pointer address and length packed similarly to `host_call`. - Exported Functions: - - `main`: The entry point of the XCQ program. It performs type checking, arguments and results passing and executes the query. + - `main`: The entry point of the XCQ program. It performs type checking, arguments and results passing and executes the query. ### XCQ Program Execution Flow @@ -322,6 +289,7 @@ Operands: - `OtherPVMError = 4` #### Errors + - `BadOrigin` - `DestinationUnsupported` @@ -366,6 +334,7 @@ It's a new functionality, which doesn't modify the existing implementations. ### Ergonomics The proposal facilitate the wallets and dApps developers. Developers no longer need to examine every concrete implementation to support conceptually similar operations across different chains. Additionally, they gain a more modular development experience through encapsulating custom computations over the exposed APIs in PolkaVM programs. + ### Compatibility The proposal defines new apis, which doesn't break compatibility with existing interfaces. @@ -380,5 +349,6 @@ There are several discussions related to the proposal, including: ## Unresolved Questions +- The metadata of the XCQ extensions can be integrated into `frame-metadata`'s `CustomMetadata` field, but the trade-offs (i.e. compatibility between versions) need examination. ## Future Directions and Related Material From 89e098fbcc273020dfa851f0af9c40dcb204ceaf Mon Sep 17 00:00:00 2001 From: indirection42 Date: Tue, 10 Dec 2024 11:07:57 +0800 Subject: [PATCH 11/12] fix weightlimit type --- text/0126-introduce-xcq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0126-introduce-xcq.md b/text/0126-introduce-xcq.md index 6f42526a0..830f02097 100644 --- a/text/0126-introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -257,7 +257,7 @@ The integration of XCQ into XCM is acheived by adding a new instruction to XCM, ```rust ReportQuery { query: SizeLimitedXcq, - weight_limit: Option, + weight_limit: WeightLimit, info: QueryResponseInfo, } ``` From 9def360c70f08310c30192f5bb2cb235d69264ef Mon Sep 17 00:00:00 2001 From: indirection42 Date: Wed, 11 Dec 2024 10:40:03 +0800 Subject: [PATCH 12/12] fix typos --- text/0126-introduce-xcq.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/text/0126-introduce-xcq.md b/text/0126-introduce-xcq.md index 830f02097..1ca83e28f 100644 --- a/text/0126-introduce-xcq.md +++ b/text/0126-introduce-xcq.md @@ -58,7 +58,7 @@ The weight limit is the maximum weight allowed for the query execution. - `metadata`: Return metadata of supported extensions (introduced in later section) and methods, serving as a feature discovery functionality. The representation and encoding mechanism is similar to the [`frame-metadata`](https://github.com/paritytech/frame-metadata/), using `scale-info`. -**Example XCQ Runtime API**: +#### Example XCQ Runtime API ```rust decl_runtime_apis! { @@ -74,7 +74,7 @@ enum XcqError { } ``` -**Example Metadata (before SCALE-encoded)** +#### Example Metadata (before SCALE-encoded) ```rust pub struct Metadata { @@ -250,7 +250,7 @@ The interaction between an XCQ program and the XCQ Extension Executor follows th ### XCM integration -The integration of XCQ into XCM is acheived by adding a new instruction to XCM, as well as a new variant of the `Response` type in `QueryResponse` message.: +The integration of XCQ into XCM is achieved by adding a new instruction to XCM, as well as a new variant of the `Response` type in `QueryResponse` message.: - A new `ReportQuery` instruction @@ -299,7 +299,7 @@ Operands: - XCQ Query Program Size: The size of XCQ query programs should be optimized to ensure efficient storage and transmission via XCMP/HRMP. Some strategies to address this issue include: - - Exploring modular program structures that allow for separate storage and transmission of core logic and supporting elements. PolkaVM supports spliting the program into multiple modules. + - Exploring modular program structures that allow for separate storage and transmission of core logic and supporting elements. PolkaVM supports splitting the program into multiple modules. - Establishing guidelines for optimizing dynamic memory usage within query programs ### User experience issues