Document how to write a custom backend

[DemiMarie]

I was not able to find documentation on how to write custom backends.

[Drakulix]

Well I guess this issue has two sides:

1. If you assume there is some kind of generic abstraction over different backends in smithay and thus there should be documentation on how that works, then I have to tell you, that smithay has no such thing.

Smithays modules are very loosely coupled and especially more low-level modules usually expose a lot of implementation details to be as versatile as possible. That means you can plug different things together however you like and also easily insert your own code, but that also means that there is no common api for all the different backends. They share parts of the code (e.g. both the drm-backend as well as the winit-backend may utilize a Renderer), but getting there is very different for both of them. Any custom backend would likely again work differently in some aspects.

2. If your problem is, that you simply have no idea where to start, then yes, we are lacking documentation on that regard.

May I ask what backend you had in mind and where you would expect to find such documentation? That might provide a good ground to start writing something up.

[DemiMarie]

May I ask what backend you had in mind and where you would expect to find such documentation? That might provide a good ground to start writing something up.

Qubes OS wants to switch from X11 to Wayland. This will require a Wayland compositor that uses the Qubes GUI protocol as a rendering and input backend. While the Qubes OS team could write a simple Wayland compositor “by hand” (without using any toolkits), such a compositor would only support the most basic Wayland protocols, so most advanced functionality would not work. This would result in a poor user experience. Therefore, I have decided to use a compositor toolkit. Currently, I am trying to choose between wlroots and Smithay.

There are additional technical details that will result in this compositor being quite unusual:

1. The GUI qube (VM) has access to the GPU and thus to hardware acceleration. The frontend compositor (the one being discussed here) does not. Therefore, the frontend compositor should not actually do any compositing, but instead pass each surface to the GUI qube, which can perform the compositing using the GPU.
2. The frontend compositor needs to speak the Qubes GUI protocol. This will require custom rendering and input backends.
3. The Qubes GUI protocol includes support for a subset of GUI features, beyond just drawing and input handling. Currently, this is limited to the inter-qube clipboard, but more features (such as screensharing) may be added in the future.
4. Window decorations are the responsibility of the GUI daemon in the GUI qube. The frontend compositor does not draw server-side decorations itself, but must use the xdg-decoration and server-decoration protocols to tell clients to not draw client-side decorations, either.
5. Qubes OS uses Xen grant tables to share memory between the GUI agent in the frontend compositor and the GUI daemon. The library APIs for this require that the memory have been allocated specially. Supporting dma-bufs is possible but would require substantial additional work.

[elinorbgr]

Wow, thanks a lot for your interest!

Before I start, I should warn you that wlroots is a much more advanced and mature project than Smithay, which is on some regards still pretty experimental. That said, if you're interested anyway, we'd be glad to work in collaboration to bring to Smithay the missing pieces you may need.

From what you describe, it looks like the easiest route for you would be to simply not use the backend infrastructure of Smithay, and only use the logic for handling Wayland clients, that you'd combine with your on logic to plug into your specific drivers. Due to hwo Smithay is designed, it'd just be a matter of designing your own abstractions as you see fit. There would be close to no friction with the rest of Smithay thanks to how loosely coupled our modules are.

If you want to chat more directly about it, feel free to drop by #smithay:matrix.org on Matrix, or #smithay on libera.chat (both chat rooms are bridged together), @Drakulix and I are mostly active on Europe timezone.