Support subsurface/subview/widgets?

[madsmtm]

The idea to support "subviews", i.e. a sort of limited Window that can be nested inside of Window, has floated around a few times, so I wanted to open an issue to discuss it.

A Subview would have the following platform equivalent (feel free to edit this):

• iOS: UIView
• macOS: NSView
• Wayland: wl_subsurface
• Web: HTMLCanvasElement (?)
• X11: XID
• Windows: HWND
• Android: android.view.Surface / ANativeWindow

Related: #696. Also discussion in #3311.

[madsmtm]

Personally, I think we should declare this as out of scope of Winit.

At least on macOS and iOS such subviews wouldn't really be useful on their own, we'd have to start allowing creating NSStackView, NSSplitView, NSTextFieldView and so on, and that's a slippery slope, then we're basically becoming GTK.

[kchibisov]

Keep in mind that this is already present in a form of child_window, but the only reason it worked is because of XID and HWND being the same for window. However the API doesn't match what subviews should do, since obviously they can't fullscreen and they have a special way to receive events and resize.

So I guess this issue is more about removing hacky child window stuff and doing proper subview.

Personally, I think we should declare this as out of scope of Winit.

it's already in winit and they are very much required to do complex scene compositing/video players, etc. The same as mixing renderers.

[ids1024]

then we're basically becoming GTK

Having some way to do subsurfaces/subviews is pretty important for creating a general purpose toolkit like GTK on top of winit. So it seems important unless being a backend for a UI toolkit, or a video player is considered out of scope for winit. Or being used to create a video player, etc.

If it's not supported in winit, there would need to be some acceptable way at least to use platform-specific code to implement this on top of winit. Not sure how practical this would be currently on all of these platforms.

Actually using subsurfaces for things like hardware decoded video will inevitably be platform-specific... at least in the absence of libraries that help with that.

Is it possible to represent all of these kind of subviews as a raw-window-handle currently, or would we need changes there too?

[kchibisov]

Basic subviews don't require anything special. We may have option to create special purpose subveiws, but in general the end goal is to have a way to a separate render target within a window to have video players, overlays drawn with software rendering, just transparent surfaces to dim main scene, etc.

And on macOS maybe some views that affect compositing which you can pass via the platform specific attributes.

[madsmtm]

Discussed a bit on Matrix.

Reconsidering, I think my problem is actually that a Winit Window already does a bunch of stuff that I don't want it to do, like handle keyboard input, because fundamentally that's often a bad idea to do for the entire window (unless you're in the limited sphere of a game). And I don't want to add yet another suboptimal solution for things that people really should resolve using system APIs.

But I realize that subviews could actually resolve most of my issues here, especially if we started assigning events to subviews instead of windows (e.g. split some of the events in WindowEvent out to a new SubViewEvent).

But then again, once we do that, we will likely also have to implement some sort of event bubbling, which, ugh, feature creep.

[madsmtm]

Copying the rough notes from today's meeting:

We can route mouse input events to each subsurface, as that's already done automatically on each platform.

Routing keyboard input events needs focus. Maybe we re-implement focusing? Definitely need some sort of platform-specific focusing support.

Some things maybe cannot bubble (e.g. popups, but maybe those are still separate windows?). Has to be user-controllable.

[jgcodes2020]

Throwing my two cents in. (Author of Wayland subsurface PR here)

We're trying to support multiple types of window. All windows can:

• be drawn to
• receive window and pointer events
• etc.

However, each window type has their own features:

• toplevels can be maximized, minimized, fullscreened, etc.
• subviews may not receive keyboard events
• popups may close themselves if you click outside of them

Since we're aiming to use a trait-based API for this, it would make sense to do something like:

pub trait Surface: rwh_06::HasRawWindowHandle/*, etc. */ {
  // surface-specific methods
}
pub trait Toplevel: Surface {
  pub fn set_maximized(maximized: bool);
}
pub trait Subview: Surface {
  pub fn set_position(pos: &Position);
}

[MarijnS95]

For Android, as discussed in Matrix before I left for a month, using multiple Surfaces for compositing independent planes inside one app requires SurfaceControl and SurfaceTransaction bindings in the NDK (experimental branches are up on the repo), but I haven't gotten it to work on a plain NDK app without Java hacks yet: https://issuetracker.google.com/issues/320706287

[axelkar]

Wayland subsurfaces would be really important for power optimization: https://nical.github.io/drafts/gui-gpu-notes.html

[jgcodes2020]

I have done a bit more GUI work on other projects in the meantime, and I've realized macOS's Window/View model might also be a good approach to implementing subsurfaces, as it's easier to map onto other platforms. On desktop platforms, this would look like:

	Window	View
Windows	HWND (WS_OVERLAPPEDWINDOW) + IDCompositionTarget	HWND (WS_POPUP + WS_EX_TOOLWINDOW + DWMWA_CLOAK) + IDCompositionVisual
macOS	NSWindow + NSView	NSView
X11	x11rb::Window	child x11rb::Window with override_redirect(true)
Wayland	wl_surface + xdg_toplevel/xdg_popup	wl_surface + wl_subsurface

WS_CHILD on Windows isn't the best solution since WS_CHILD windows simply override a portion of their parent's window and ignore the compositor, so we would instead have to use DirectComposition since that provides well defined z-order. However, we're left with a dilemma:

• For some applications, a single view is enough; so we could avoid setting up DirectComposition entirely.
• For other applications, we do need multiple views, so we need DirectComposition.

The easiest way to fix this would be to add a hint to WindowAttributes, maybe something like with_single_view, so that any window created with with_single_view(true) wouldn't be able to create subviews.

In addition, platforms are generally inconsistent with regards to clipping views. Wayland and Windows don't, but macOS and X11 do (AFAIK), so that needs to be ironed out.

[axelkar]

wp_viewporter supports cropping subsurfaces and I think it's already used in Winit for scaling. Also, some pointer events need to be handled in Wayland for subsurfaces; how is it for other platforms?

[jgcodes2020]

wl_viewporter supports cropping subsurfaces and I think it's already used in Winit for scaling. Also, some pointer events need to be handled in Wayland for subsurfaces; how is it for other platforms?

On Windows, DirectComposition doesn't delegate inputs, so that will have to be emulated in software. MacOS and X11 do have builtin event delegation and bubbling, but I'm not sure exactly how that works.

[axelkar]

I have done a bit more GUI work on other projects in the meantime, and I've realized macOS's Window/View model might also be a good approach to implementing subsurfaces, as it's easier to map onto other platforms.

@jgcodes2020 you created #3928. Can we still use trait Window: Surface and friends?

Has someone implemented this new API (if it's different)?

[jgcodes2020]

I have done a bit more GUI work on other projects in the meantime, and I've realized macOS's Window/View model might also be a good approach to implementing subsurfaces, as it's easier to map onto other platforms.

@jgcodes2020 you created #3928. Can we still use trait Window: Surface and friends?

Has someone implemented this new API (if it's different)?

I had a PR to implement at least the split between Window and Surface (#3942), though I didn't implement popups/subviews on top of it, as that would require a massive rearchitecting of the underlying code. However, the maintainers have said that they will get to it once they handle the big backend split (#3433).

If I were to implement it now, the resulting API would combine the two designs; using the Surface trait to represent windows, popups, etc. and View to represent subsurfaces, as this seems to correspond best with how most platforms implement the different surface types, though it still needs a lot of design work to actually get right.