RFC: Investigate integration of Xwayland rootful mode for improved Wayland functionality

[jrelvas-ipc]

Context

Currently, Vinegar has a major limitation in Wayland sessions. Xwayland cannot simulate cursor locks or constraints, preventing the cursor from locking in Roblox Player under certain circumstances and breaking camera rotation in Roblox Studio's edit mode.

These limitations may be overcome in the future with wine's upcoming native wayland driver, but there's no clear release date yet, nor if it even implements cursor constraints to begin with.

Gamescope has been suggested in the past as a workaround, but there are some issues which prevent its widespread adoption:

• Very inconsistent rates of success in actually grabbing the cursor properly.
• Bad compatibility with nvidia users.

And most cripplingly...

• No proper windowed functionality, preventing it from being an acceptable solution for Roblox Studio.

---

Introduction

With wine wayland still at least some months away and the unreliability of gamescope's cursor grabbing, the majority of users in wayland sessions are forced to either deal with these limitations or switch back to an X11 session just to use Roblox Studio or play Roblox experiences.

This is where Xwayland's "rootful" mode comes in. Although it has been present in Xwayland for a long time, it has gotten a lot of improvements in the "new" Xwayland 23.1 release. (with more planned soon, as of november)

---

What is Xwayland rootful and how can it help us?

Xwayland normally runs "rootless", where every X window integrates seamlessly into the Wayland session. This mostly works, but limits the functionality of certain programs, including Wine. Programs can't just change the cursor's position in Wayland, which is why Wine fails to lock Roblox's cursor properly.

Rootful fixes this by running a full X11 session, nested on top of wayland! X11 windows appear inside of the nested session, instead of blending into the wayland session. A window decoration for the nested session can be provided via libdecor.

$ Xwayland -geometry 1024x768 -retro -decorate :12

$ xterm -display :12 &
$ twm -display :12 &
$ xsetroot -solid dodgerblue -display :12

Now here's the kicker! This window can be resized by programs (for instance, xrandr) and even dynamically by the user!
https://www.youtube.com/watch?v=pbtoxyX4iTw
https://www.youtube.com/watch?v=Ndu4bwU_EZY

On supported wayland hosts, Xwayland can grab user input directly with the -host-grab option! This is likely the missing piece we need for proper cursor locking!

$ Xwayland -geometry 1024x768 -decorate -host-grab :12 &
$ DISPLAY=:12 SESSION_MANAGER= GDK_BACKEND=x11 dbus-run-session startxfce4

https://www.youtube.com/watch?v=wR_mhfmn5Bw

With the help of a X11 WM, you can automatically resize a window to fill the (nested) screen completely. This would allow us to provide a near-transparent windowed experience for Roblox Studio. For example:

$ Xwayland -geometry 1024x768 -decorate :12 &
$ matchbox-window-manager -display :12 &
$ GDK_BACKEND=x11 midori --display=:12

Lastly, the -fullscreen option can be used to make the nested session... fullscreen, allowing satisfactory Roblox Player use.

Sources:
https://ofourdan.blogspot.com/2023/10/xwayland-rootful-part1.html
https://ofourdan.blogspot.com/2023/11/xwayland-rootful-part-2.html

In short, Xwayland rootful could allow us to effectively workaround our current cursor issues and provide a wayland experience with no significant flaws or drawbacks in the very short-term, instead of making our users wait an undisclosed amount of months for wine wayland to be ready and (supposedly) provide cursor constraints.

---

The plan

Phase 1 - Validate Xwayland rootful state:

Check what functionality the latest Xwayland release has and if it's functional. Confirm the steps required to do the following:

• Launch Roblox Player in a fullscreen nested session.
  • Player should cover the entire screen, with no black borders or window decorations.
  • Cursor should lock. Shiftlock, entering first-person and moving the camera with right-click should keep the cursor stationary in the middle of the screen in a regular baseplate.
  • No major visual or functional regressions.
• Launch Roblox Studio in a windowed nested session.
  • Xwayland should have a window decoration.
  • Studio should cover the entire Xwayland nested session and adapt to size changes, leaving no empty space.
  • Edit mode should have proper camera rotation with the same behaviour of running Studio in a X11 session or natively in Windows. The cursor should also lock while this is done.
  • Childwindows (widgets) should continue to work and display properly.
  • No major visual or functional regressions.
  • (Optional) Ideally, studio's window should have no window decoration, as the session's window has its own.

If there's any missing functionality, figure out what is its release schedule.

Assuming we run into no issues, we should be able to document rootful xwayland, allowing advanced users to experiment and get it working without any changes to Vinegar.

Phase 2 - Integrate rootful mode into Vinegar:

Xwayland rootful is a big scary monster and requires more than what the existing launcher option in Vinegar can do.

Hypothetically, we need to run the following commands:

$ Xwayland -geometry 1024x768 -decorate -host-grab :12 # PLAYER: Create a new Xwayland rootful session running in port 12 (shown as a window).
$ Xwayland -fullscreen -host-grab :12 # STUDIO: Create a new Xwayland rootful session running in port 12 (shown as fullscreen).
$ matchbox-window-manager -display :12 # Run the matchbox WM to resize all X11 windows to match the nested session's size.
$ DISPLAY=":12" %command% # Actually start wine (and therefore Roblox), but on the display port we created the xwayland session with.

We could expand the launcher functionality to make this work, or maybe we can ship Vinegar with a helper program/script which acts as a launcher that transparently prepares the Xwayland session and runs the specified program on it.

Not sure how we'd handle it if functionality was good enough to make it a default, though... Feedback on that is required.

Phase 3 - Packaging this mess:

This is the hardest challenge, in my opinion.

• Source:
  Depending on our implementation, we might need to make changes to the build process.
• Distro packaging:
  This change technically makes Xwayland a dependency, especially as it requires a fairly new version that not every distro is shipping yet. Here's a few ideas:
  • Make the required Xwayland version a weak dependency. If it's not present, simply don't use rootful by default.
  • Don't use rootful by default. Include the option to do so, but expect the user to be smart enough to configure their system.
• Flatpak:
  Is Xwayland present in Vinegar's runtime and is it recent enough? If yes, then we can probably default to rootful there without any second thoughts. If not though...
  • Vinegar would need to break the sandbox to use the system's Xwayland (probably meaning rootful is not a sane default).
  • Make a Xwayland add-on/extension, just like the gamescope add-on.
  • Include Xwayland inside of the Vinegar flatpak itself (sounds like a really stupid and dangerous idea).
• Snap:
  No insights here, requesting comments from the package maintainer. @z-ffqq

Packaging this is very tricky, so I believe flatpak should be our priority. If it's confirmed xwayland rootful meets our requirements, then we should really try to find the sanest way of making it a default there, as that's where most of our users are located. Doing this right could fix our last major wayland limitation by Christmas :P

---

I've left phase 2 and phase 3 intentionally vague and open to discussion. Feedback is highly welcome there, especially once phase 1 is complete.

I will begin my testing shortly and will report back soon.

[lunarlattice0]

10/10 effort on this RFC Jrelvas, nice work

[lunarlattice0]

My TLDR summary:

a regression upstream stops us from having both wayland and xorg available as permissions```

We would have to find a way to either get Wine to run Wayland-only, or figure out if we can run a session inside the flatpak.

[jrelvas-ipc]

Status update:

Unfortunately, even with cursor grabbing, xwayland still does not attempt to lock the cursor unless it's invisible. If you make xwayland have no cursor, then locking works fine... it just has no visible cursor.

But then I remembered something...

This is a FULL x11 session. There's nothing stopping us from, say, running a software cursor program on it.

So that's exactly what I did. I searched for "x11 software cursor" and hit the first link I saw, swcursor. I quickly ran it on DISPLAY :12 and...

oh.shit.webm

Holy fucking SHIT, it works. We have a decent chance of pulling this off.

Far from flawless, but the floodgates are open now.