Introduce XCQ #126

indirection42 · 2024-10-25T01:39:55Z

Looking for feedback.

Summary

This proposal introduces XCQ (Cross Consensus Query), which aims to serve as an intermediary layer between different chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries.

Related RFC

A related XCM-format RFC is drafting.

Related Discussions

https://forum.polkadot.network/t/wasm-view-functions/1045

PoC implementations

https://github.com/open-web3-stack/XCQ

franciscoaguirre · 2024-10-28T23:32:37Z

text/0126-introduce-xcq.md

+Therefore, tools and UI developers often access storage directly and reimplement custom computations to convert data into user-friendly representations, leading to duplicated code between Rust runtime logic and UI JS/TS logic.
+This duplication increases workload and potential for bugs.
+
+Therefore, A system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries.


Suggested change

Therefore, A system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries.

Therefore, a system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries.

franciscoaguirre · 2024-10-28T23:33:14Z

text/0126-introduce-xcq.md

+
+## Explanation
+
+The overall query pattern of the XCQ consists of three components:


Suggested change

The overall query pattern of the XCQ consists of three components:

The overall query pattern of XCQ consists of three components:

franciscoaguirre · 2024-10-28T23:38:29Z

text/0126-introduce-xcq.md

+```rust
+decl_runtime_apis! {
+    pub trait XcqApi {
+        fn execute_query(query: Vec<u8>, input: Vec<u8>, weight_limit: u64) -> XcqResult;


Suggested change

fn execute_query(query: Vec<u8>, input: Vec<u8>, weight_limit: u64) -> XcqResult;

fn execute_query(query: Vec<u8>, input: Vec<u8>, gas_limit: u64) -> XcqResult;

Is this limit the PVM gas instead of the weight?

It seems that the metrics of the PVM gas are in development now, and the gas value is an inner value that should not be exposed to users.

franciscoaguirre · 2024-10-28T23:45:23Z

text/0126-introduce-xcq.md

+```
+- `PermController`: Filters guest XCQ program calling requests, useful for host chains to disable some queries by filtering invoking sources.
+```rust
+pub trait PermController {


Maybe Authorizer is a better name? Or PermissionsController (not abbreviated)

franciscoaguirre · 2024-10-28T23:53:08Z

text/0126-introduce-xcq.md

+  - Integration tests to ensure proper interaction with off-chain wallets/UI and on-chain XCM, including the aforementioned use cases in **Motivation** section.
+
+- Security:
+  - The XCQ system must enforce a strict read-only policy for all query operations. A mechanism should be implemented to prevent any state-changing operations within XCQ queries.


Could the execution of the program be sandboxed?

Yes, we can do the query in a transaction, and rollback finally.

franciscoaguirre · 2024-10-28T23:54:32Z

text/0126-introduce-xcq.md

+
+## Future Directions and Related Material
+
+Since XCQ are supported both in off-chain and on-chain, a related XCM-Format RFC should be proposed.


We recently merged XCM RFCs into the fellowship RFCs. So the new instruction and behaviour of XCM could be described here.

jsdw · 2024-11-13T10:02:57Z

text/0126-introduce-xcq.md

+`xcq-types` is a meta-type system similar to `scale-info` but simpler.
+It enables different chains with different type definitions to work via a common operation.
+Front-end codes constructs call data to XCQ programs according to metadata provided by different chains.


I'd like to know more about xcq-types, ie what are the existing issues with scale-info here? Since it just describes the shapes of types, one can compare two sets of scale-info types across chains already, or is there more to it?

Adding a separate system to describe types adds overhead when everything else is built to work around scale-info already, so I wonder how necessary it is (ie we already have tools to do codegen, encoding and decoding with scale-info and it'd be a pain to reimplement them all for this).

Another point here: I'd assume that we would want XCQ metadata to be a part of the frame-metadata that we hand back when users call the Metadata_metadata_at_version runtime API? In that case, it feels beneficial to use the existing scale_info based type information used by the rest of the metadata over having a separate concept to understand and integrate there.

Yes, makes sense. we are experimenting with switching back to scale-info.

jsdw · 2024-11-13T10:14:06Z

text/0126-introduce-xcq.md

+}
+```
+
+- `ExtensionExecutor`: Connects extension implementations and `xcq-executor`.


If one wants to expose pallet view functions or runtime APIs to the XCQ programs, do extensions have to be written which call into them (and then those extensions are what is exposed in the XCQ program)?

Is the reason for having extensions instead of exposing raw runtime APIs and pallet view functions to XCQ programs that extensions have associated types which can be tweaked by different runtimes, whereas runtime APIs do not? (If that were the only reason, I wonder whether runtime APIs could be extended to optionally support associated types to achieve the same without a new thing being needed?)

A1: Yes, the fundamental APIs exposed to XCQ programs are organized at the extension level.

A2: It's not the only reason. Runtime APIs cannot be recognized via some identifications, while extensions can be recognized via extension ids.

Runtime APIs cannot be recognized via some identifications

Runtime API methods can all be identified as $trait_name ++ _ ++ $method_name, and so we could expose functions with similar names to the PolkaVM program? I suspect I am missing something :)

If Runtime APIs "just worked" natively, it would mean that we could provide a single stable interface to users (who can call the APIs themselves) and XCQ programs (which could call the same APIs), and wouldn't need to add another concept ro Runtimes.

The main reason we created XCQ extension is to have content based address. i.e. The extension ID is the hash of the extension definition, which includes the methods and argument types. So we can for sure know the compatibility with a matching extension ID. Runtime API is using versioning for that, which requires extra collaboration.
Besides, Runtime API are designed to be invoked from outside of wasm, which is only half of the use case of XCQ, the other half is XCM. We cannot assume every runtime API are safe to be dispatched by XCM.

Thankyou for your reply! I'm not super knowledgable in this area so apologies for more questions..

The extension ID is the hash of the extension definition, which includes the methods and argument types. So we can for sure know the compatibility with a matching extension ID. Runtime API is using versioning for that, which requires extra collaboration.

How does having a content based address help when compared with non content based addresses (ie the address just being the extension trait and method name, like Runtime APIs)?

Looking at some of the possibilities:

If I change the semantics but not the type information of some method:

Using content based address: the method will no longer do what is expected, but will still be callable via the same hash.

Using non-content based address: same as above

Verdict: Neither approach is any different here; we require social agreement (mostly a no brainer ofc) to not change these things.

If I change the semantics and type information (ie name, input, return types):

** Using content based address:** calling the "old" address will now fail with some error that it doesn't exist.

** Using non-content based address:** calling the "old" address will still work, but fail with some type error on encoding or decoding.

Verdict: Both approaches lead to an error; we require social agreement to not change/remove methods if we want to avoid breakage.

In other words, however we approach addressing things, we still need people to agree not to make breaking changes to these functions if we want to avoid breaking compatibility.

Do content based addresses help in some other way that I am missing (definitely likely!!)? Is the thinking here mainly around XCM ie making it more likely for queries that they call to no longer be accessible?

And a couple more Dumb Qs: what do extensions provide on top of runtime APIs that ensures that they are safe to be dispatched by XCM (or marks them as not being safe)? Might be there some extensions that we want to be able to call in XCQ programs but not via XCM?

The current polkadot-sdk runtime calls are name based and have many known issues and most of them can be addressed by content based address, which is more commonly used. e.g. EVM selector.

Some advantages:

The extension ID fixed the name and types, the name based approach only fixed the name

The extension ID is hash of the whole interface, whereas the name, well, is just whatever the name referring to.

e.g. 0x123456 could mean trait Fungibles { fn balanceOf(who: AccountId) -> U128; fn totalSupply() -> U128 } but with name, Fungibles means a trait named Fungibles but does not constraint the methods. A name of Fungibles.balanceOf is just a function name, and not telling anything about other members.

Unexpected change of type can be detected easily because it changes the extension ID. Not the case for name based ID.

There is no need to have central name registry. 3 different chains may all want to propose their version of Fungibles interface and who decide who got it? Not an issue for hash.

PermissionController defined in the RFC is the mechanism to only allow some extension in some context.

jsdw · 2024-11-13T10:17:47Z

text/0126-introduce-xcq.md

+```rust
+decl_runtime_apis! {
+    pub trait XcqApi {
+        fn execute_query(query: Vec<u8>, input: Vec<u8>, weight_limit: u64) -> XcqResult;


I guess any user can call this Runtime API and provide their XCQ program and input args? Is it expected that the runtime will have some default gas/weight limit to make it more difficult for users to supply long running/complex programs that DoS the node?

The weight_limit argument in this single API restricts the usage of a single execution. The overall resource limit is enforced by Runtime config and RPC rate limiting, which is out of the scope for a single RuntimeAPI. As for XCM usage, the gas/weight limit can be enforced by BuyExecution instruction.

indirection42 added 4 commits October 25, 2024 08:22

introduce xcq

cdf886c

update

ae88d8d

update

62174b1

update rfc number

fa97e4a

indirection42 changed the title ~~Introduce xcq~~ Introduce XCQ Oct 25, 2024

indirection42 mentioned this pull request Oct 28, 2024

RFC: Add an instruction about XCQ polkadot-fellows/xcm-format#71

Open

franciscoaguirre reviewed Oct 28, 2024

View reviewed changes

indirection42 added 4 commits October 29, 2024 19:55

add xcm section

740f5e0

rename PermissionController

bff1067

update

3678eee

add read-only execution suggestion

220f62d

indirection42 force-pushed the introduce-xcq branch from 0af3db3 to 220f62d Compare October 30, 2024 02:18

anaelleltd added the Proposed Is awaiting 3 formal reviews. label Nov 3, 2024

merge xcm rfc

6c637c3

xlc mentioned this pull request Nov 12, 2024

[tracking] Facade paritytech/polkadot-sdk#6441

Open

jsdw reviewed Nov 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce XCQ #126

Introduce XCQ #126

indirection42 commented Oct 25, 2024 •

edited

Loading

franciscoaguirre Oct 28, 2024

franciscoaguirre Oct 28, 2024

franciscoaguirre Oct 28, 2024

indirection42 Oct 29, 2024

franciscoaguirre Oct 28, 2024

franciscoaguirre Oct 28, 2024

indirection42 Oct 30, 2024

franciscoaguirre Oct 28, 2024

jsdw Nov 13, 2024 •

edited

Loading

jsdw Nov 22, 2024

indirection42 Nov 23, 2024

jsdw Nov 13, 2024 •

edited

Loading

indirection42 Nov 19, 2024

jsdw Nov 22, 2024

xlc Nov 25, 2024

jsdw Nov 29, 2024 •

edited

Loading

xlc Nov 30, 2024

xlc Nov 30, 2024

jsdw Nov 13, 2024

indirection42 Nov 19, 2024

	Therefore, A system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries.
	Therefore, a system is needed to serve as an intermediary layer between concrete chain runtime implementations and tools/UIs, to provide a unified interface for cross-chain queries.


		## Explanation

		The overall query pattern of the XCQ consists of three components:

	The overall query pattern of the XCQ consists of three components:
	The overall query pattern of XCQ consists of three components:

	fn execute_query(query: Vec<u8>, input: Vec<u8>, weight_limit: u64) -> XcqResult;
	fn execute_query(query: Vec<u8>, input: Vec<u8>, gas_limit: u64) -> XcqResult;


		## Future Directions and Related Material

		Since XCQ are supported both in off-chain and on-chain, a related XCM-Format RFC should be proposed.

Introduce XCQ #126

Are you sure you want to change the base?

Introduce XCQ #126

Conversation

indirection42 commented Oct 25, 2024 • edited Loading

Summary

Related RFC

Related Discussions

PoC implementations

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

indirection42 commented Oct 25, 2024 •

edited

Loading

jsdw Nov 13, 2024 •

edited

Loading

jsdw Nov 13, 2024 •

edited

Loading

jsdw Nov 29, 2024 •

edited

Loading