[tracking] Facade

[jsdw]

This is a tracking issue which collects any issues that are releated to the Facade project.

Please add any facade related issues to this as needed.

Tasks

Beta Give feedback

1. Add scaffolding to declare Facade APIs #6396. A place to define our runtime level Facade APIs.
2. Implement pallet view function queries #4722. Not strictly necessary in the MVP for declaring higher level facade APIs, but overall an important outcome to help move people away from direct storage accesses.

\cc @gui1117 @Jan-Jan

What is the Polkadot Facade?

Authors: @kianenigma, @jsdw

TL;DR

• Objective: Simplify the development and improve the stability of tooling and applications on Polkadot relay chain and its system parachains, and the DOT token.
• Key Results
  • Runtime Side: Create low level APIs, both at the runtime and at the pallet level, that expose a stable API of the pallet and the runtime.
• Client Side: Create high level and stable APIs around Polkadot as a multi-system-parachain network, allowing (read) access to DOT-native functionality[^1] without needing to interact with specific parachains.
  • Historic Support: The same APIs should work from genesis all the way up until now.

Terminology

• Runtime Side: Within the WASM runtime of a chain.
• Client Side: Anything outside of a single runtime, specifically the libraries, tooling and users that wish to interact with chains.
• Pallet View functions: Each pallet should define a set of view functions that abstract its underlying, complicated[^2] and volatile[^3] storage. When the storage format changes, the implementation of these functions would change, but the interface would remain stable for existing users. Therefore, view functions are meant to have a stable API by definition. We sometimes call these “Pallet APIs” for brevity.
• Facade Runtime APIs: The set of Runtime APIs specific to the Facade project. These should be stable across different runtimes, such that the Facade Client APIs can reliably use them to communicate with multiple chains.
• Facade Client APIs: The set of Facade APIs implemented on the client side, using libraries like PAPI or Subxt. These APIs primarily leverage the stable Facade Runtime APIs present across multiple chains in order to obtain and aggregate important information for users. They also contain the logic for using historic state to hand back the same information from the past.
• Historic Support: The facade client APIs need to also work when querying against older blocks. Thus, they need to include the shims and code required to obtain, decode and translate historic state or runtime API calls into a format compatible with the current APIs.
• Facade Runtime Spec: A specification which defines the shape and semantics of the Facade Runtime APIs. This allows individual chains to implement them such that they are compatible with the Facade Client APIs.
• Facade Client Spec: A specification which helps to ensure that, whichever language is used, we are providing the same set of functionality to all clients.

Scope

The scope of this project includes all the beige items pictured below:

Diagram 2: A system view of the pieces of work (beige) that comprises the Polkadot Facade

Implementation

This work can be broken into the following streams. Streams 2 and 3 mainly involves work in polkadot-sdk. The rest of the work will mostly happen elsewhere.

1. Facade Client API repository: Discover and spec out the Client Facade APIs.
2. Facade polkadot-sdk groundwork: Merge Pallet View Functions PR and other nice-to-haves to aid in defining the Runtime Facade APIs.
3. Facade Runtime API repository: To contain crate(s) which will house the Runtime Facade APIs, docs, spec and helpers.
4. Support historic decoding in TypeScript: Write a TS wrapper around frame-decode to enable decoding historic state in our Facade Client APIs, and prototype it.
5. Implement APIs E2E: When the above takes shape, we’ll be able to start implementing the APIs that we have decided on. This will build upon all of the above streams.

Steps 1-4 can roughly be worked on in parallel, and then as we define more APIs that we are ready to have a go at implementing in 1, we’ll be able to start progressing on 5.

More detail on each of these to follow:

Stream 1: Facade Client API Repository

This repository will contain a spec which defines the rough shape of the Facade Client APIs along with a reference implementation written using PAPI[^4]. We’ll:

• Pick a language/approach for defining the Facade Client API spec.
• Write tooling to generate some stub interface from this spec, and/or generate tests to check that an existing reference implementation matches it.
• Stub out the beginnings of a reference implementation in PAPI, ready to start adding methods to.

In conjunction with creating a repository, we’ll gather feedback on which APIs to actually have. To this end, we’ve opened more issues like this to gather feedback from ecosystem developers.

Stream 2: Facade polkadot-sdk groundwork

This covers the changes we’d like to have in polkadot-sdk in order to support this Facade work. These are:

1. Merging the Pallet View Functions PR.
2. Allow multiple impl_runtime_apis! blocks. This might enable us to better automate defining whole sections of Facade Runtime APIs.
3. Make Runtime APIs subscribable. Storage items can be subscribed to at present. If Runtime APIs and Pallet View Functions could also be subscribed to, then we’d remove a key reason why people may still want to directly access storage. Nice To Have in the context of the Facade work at the moment.

Stream 3: Facade Runtime API crates

We’ll create crates to house the Facade API definitions.

The MVP here is simply to expose some Facade Runtime APIs created with decl_runtime_apis!, once we start to know which APIs we want.

Building on this, we also want to expose some metadata about the Facade APIs we've defined. This will allow for libraries/tools to check a runtimes compatibility with the Facade APIs (eg what versions of which APIs does some runtime support) or generate documentation about the Facade APIs.

We should also think about how we can make implementing the Facade APIs as easy as possible, since many of them will simply point at some crate(s) and call the relevant Pallet View Functions to retrieve/aggregate an answer. To that end, we might also build:

• Macros (eg might look like facade::view_impl_helpers::balances_total_balance_impl!(BalancesPallet, OtherBalancesPallet)) which generate implementations of certain facade APIs (in this example, Balances::total_balance) which rely on the relevant Pallet View Functions.
• Some definition/spec of the Pallet View Functions that are required to exist in order that the macro can be used.

Stream 4: Support historic decoding in TypeScript

Any Client Facade API that we create should also be able to point back in time to an older block and provide back an answer. This requires us to be able to decode the state in historic blocks and transform it into the “modern” format that we’ve defined for the API.

We want a Client Facade API implementation to exist in TypeScript, and we now have a crate written in Rust that is responsible for decoding extrinsics and storage keys/values from the past. In theory, this crate compiles to WASM, and we can decode things into the scale_value::Value type, which can be serialized to JSON to be passed from WASM into TS to then be worked with.

Let’s:

• Build a TypeScript wrapper around frame-decode to allow historic state to be decoded in TypeScript.
• Build a prototype TS example in which we are able to decode some historic state given a block number. This will help to ensure that we have everything we need in our wrapper package.

Stream 5: Implement APIs E2E

Once the above has taken shape, we’ll start to settle on some APIs that we want to see implemented. To that end, we’ll:

• Add those APIs to the Facade Client API spec defined in stream 1.
• Add/spec the corresponding Facade Runtime APIs, enabled by stream 3.
• Implement the necessary Pallet View Functions and Runtime APIs in some runtime, enabled by stream 2.
• Implement the Facade Client API spec in our reference implementation, enabled by stream 1.
• Once stable, work to add historic support to the Facade Client APIs, enabled by stream 4.

This process is likely to be repeated as we settle on the definitions of more Facade Client APIs. We should also consider marking such APIs as unstable throughout the stack until we are happy with them and don’t intend to make any further changes.

[xlc]

what is this facade project?

[jsdw]

what is this facade project?

I added the rough plan we had in mind now; I hope that helps!

[xlc]

should XCQ be part of this? polkadot-fellows/RFCs#126

[bkchr]

We want a Client Facade API implementation to exist in TypeScript, and we now have a crate written in Rust that is responsible for decoding extrinsics and storage keys/values from the past. In theory, this crate compiles to WASM, and we can decode things into the scale_value::Value type, which can be serialized to JSON to be passed from WASM into TS to then be worked with.

Fascade itself should be written in Rust. Rust itself can be integrated almost anywhere. Like you said into typescript via WASM or directly into C, C++, python or whatever. The fascade Rust library should do all the heavy lifting, aka the decoding and ensuring there is a stable api etc. Then the best would be to generate bindings for this Rust library in different languages, while at the beginning the bindings could just be typescript and written manually. The end goal should be that we are able to use almost any language by generating the bindings for the Rust library.

[jsdw]

Fascade itself should be written in Rust. Rust itself can be integrated almost anywhere. Like you said into typescript via WASM or directly into C, C++, python or whatever. The fascade Rust library should do all the heavy lifting, aka the decoding and ensuring there is a stable api etc.

I'm mostly onboard with that! Given that most people using it will write TS apps, I wonder if it would make sense though to write a separate TS and Rust impl of the Facade Client library. That way, the TS version will be very idiomatic, have a small bundle size and be written in PAPI, which already has the relevant primitives for decoding etc. It'll make sense for most dApps and such.

The Rust version would then use Subxt and be the thing we bind to in other languages (and we could eventually maintain such bindings for a few langs).

Both would likely rely on the Rust stuff for historic block decoding, but this would only need to be pulled in in TS land when needed to avoid the bloat of the WASM blob.

[bkchr]

You could probably still provide some idiomatic TS library that uses the interfaces of the Rust library in a more typescript way. Same is for example done in Rust when you wrap a C library, you also expose Rust interfaces while under the hood using the C functions. But still you only write glue code in Rust and the actual implementation is in C. Same principle could be applied here.

[jsdw]

You could probably still provide some idiomatic TS library that uses the interfaces of the Rust library in a more typescript way.

Yep true; I guess my main argument was on the WASM bundle size. Josep reimplemented the merkleized-metadata stuff in TS for instance because of this, and I imagine importing a fair chunk of WASM may not appeal to some TS folks :D

[xlc]

XCQ extension already handles the API definition part and it provides metadata so client can generate whatever code it needs from the metadata

[jsdw]

should XCQ be part of this? polkadot-fellows/RFCs#126

My understanding at the moment is that this and XCQ are complementory:

• This facade work is like the foundation and "basic" APIs for doing some common things. It will lead to more Pallet View Functions being created and then, from these, additional Runtime APIs which use these to gather & return commonly wanted information. These Runtime APIs can be specced and implemented on many runtimes, so that a simple higher level client can call them across runtimes and aggregate the result. This work trades flexibility for simplicity; doing common things across supporting runtimes will be easy via the new APIs, but doing complex/custom things is still hard (still need to make a bunch of calls to gather data and and calculate things locally if the existing APIs don't give what you want).

• The XCQ proposal, IIUC, can then take advantage of these Pallet View Functions / Runtime APIs that we add here and allows people to build arbitrary PolkaVM programs to obtain more custom/complex data that isn't otherwise expressed by the APIs we add here in the Facade project. In other words, it covers the rest of the cases that the facade doesn't in exchange for the increased complexity of writing PolkaVM programs to achieve this.

[xlc]

yes I just want to ensure this project is designed and built in mind of XCQ to ensure they can nicely integrate together and avoid duplicate work
the client code generation for example, is not something we plan for XCQ but sounds like could benefit XCQ users if it supports XCQ

[jsdw]

yes I just want to ensure this project is designed and built in mind of XCQ to ensure they can nicely integrate together and avoid duplicate work

Yup that makes sense!

I think it'll be a net win overall to have both the "easy" APIs that users can use to obtain basic data across chains, plus a more advanced way to obtain and aggregate more custom/complex queries via XCQ programs. I can also imagine that when XCQ is ready we might start using it to aggregate some of this data together in place of traditional Runtime APIs where it makes sense!

Ultimately this work here shouldn't duplicate any XCQ related work. Instead. it'll provide the Pallet View Functions and new Runtime APIs which can then be fed into XCQ programs.