GPU-accelerated WPF without WindowsFormsHost

[freezy]

After reading through many issues mentioning the drawbacks of WindowsFormsHost (no transparency, no event bubbling, no borderless windows), I've tried to implement another approach: Do the expensive computing off-screen and copy the result into a (non-GPU-accelerated) SKElement.

As @mattleibow mentioned in #717, the unit tests' WglContext should be able to deliver a GPU-accelerated off-screen context.

My main view with SKElement:

<Window x:Class="WPF.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:wpf="clr-namespace:SkiaSharp.Views.WPF;assembly=SkiaSharp.Views.WPF"
        mc:Ignorable="d"
        Title="MainWindow" Height="450" Width="800">
    <Grid HorizontalAlignment="Stretch" VerticalAlignment="Stretch">
        <wpf:SKElement x:Name="BitmapHost" PaintSurface="OnPaintCanvas" />
    </Grid>
</Window>

Code behind:

public partial class MainWindow : Window
{
	private SKSurface _surface;
	private GRContext _grContext;
	private SKSize _screenCanvasSize;

	public MainWindow()
	{
		InitializeComponent();

		var glContext = new WglContext();
		glContext.MakeCurrent();
	}

	private void OnPaintCanvas(object sender, SKPaintSurfaceEventArgs e)
	{
		OnPaintSurface(e.Surface.Canvas, e.Info.Width, e.Info.Height);
	}

	private void OnPaintSurface(SKCanvas canvas, int width, int height)
	{
		var canvasSize = new SKSize(width, height);

		// check if we need to recreate the off-screen surface
		if (_screenCanvasSize != canvasSize) {
			_surface?.Dispose();
			_grContext?.Dispose();
			_grContext = GRContext.Create(GRBackend.OpenGL);
			_surface = SKSurface.Create(_grContext, true, new SKImageInfo(width, height));
			_screenCanvasSize = canvasSize;
		}

		// draw onto off-screen gl context
		DrawOffscreen(_surface.Canvas, width, height);

		// draw offscreen surface onto screen
		canvas.DrawSurface(_surface, new SKPoint(0f, 0f));
	}

	private void DrawOffscreen(SKCanvas canvas, int width, int height)
	{
		// will be more expensive in the real world
		using (var paint = new SKPaint()) {
			paint.TextSize = 64.0f;
			paint.IsAntialias = true;
			paint.Color = 0xFF4281A4;
			paint.IsStroke = false;
			canvas.DrawText("SkiaSharp", width / 2f, 64.0f, paint);
		}
	}
}

The WglContext class is copied from this repo's test directory.

The problem I'm having is when running this, I'm getting first this:

Managed Debugging Assistant 'CallbackOnCollectedDelegate' : 'A callback was made on a garbage collected delegate of type 'WPF!SkiaSharp.Tests.WNDPROC::Invoke'. This may cause application crashes, corruption and data loss. When passing delegates to unmanaged code, they must be kept alive by the managed application until it is guaranteed that they will never be called.'

This is followed by a System.NullReferenceException without any stack trace. So it's somewhere in a native call, but I can't figure out where. Any hints or suggestions would be appreciated!

I've created a repo to reproduce: freezy/wpf-skia-opengl.

Related: #213, #622, #688

VS bug #776804

[freezy]

Okay, after some more debugging, I figured out what caused the NullReferenceException: The WNDCLASS instance was garbage collected when it shouldn't, moving it to a static property solved it.

However, when closing the window, I'm now getting an System.AccessViolationException, again within some native code:

System.AccessViolationException
  HResult=0x80004003
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

If anyone could shed some light on that, that would be great. I've updated the sample repo, just launch it and close the window to reproduce.

[freezy]

Turns out that Unloaded for disposal is too late, Closing seems to do the trick.

@mattleibow if you think that SkiaSharp would benefit from a WPF view implementing this approach, let me know, otherwise feel free to close!

[mattleibow]

Also related #764 and #755

With regards to drawing offscreen for WPF - this seems like a good way (you only pay for the final draw). I may hook some things up.

I am going to leave this open as a feature request so we can track/prioritize this.

[mattleibow]

From @Mikolaytis in #819:

Hi, I'm using SkiaSharp in my WPF app with WGL rendering as described here.

The FPS of this approach is low (all performance is thrown into copy bytes to WritableBitmap), so I'm in search for another solution and found an perfect example, how to render in WPF D3DImage over OpenGL via SharpDX, Angle.

https://github.com/l3m/wpf-gles

Performance is on another level, but I'm failing to connect this example to SkiaSharp.
Is this possible at all?
Did you ever considered this type of approach for WPF apps?
If this approach will connect to skia - it will be awesome!

[mattleibow]

From @Mikolaytis:

Here we go! Done: https://github.com/Mikolaytis/WpfSkiaAngleSharpDxOpenTK

[mattleibow]

From @Mikolaytis:

Turns out text/line/etc. draw do not render anything, only images are rendering.

[mattleibow]

From @john-cullen:

@Mikolaytis did you ever get text / lines rendering for this?

[mattleibow]

@john-cullen, I can't be sure at this point, but it might be that the stencil buffers aren't set up right. I see this is 0: https://github.com/Mikolaytis/WpfSkiaAngleSharpDxOpenTK/blob/f41433c75701519991eb861aa063653293d1781f/src/WpfGlesDemo/SkiaRenderer.cs#L78

Hopefully @Mikolaytis has got a working solution that we can benefit from. 🤞

[mattleibow]

Just merging some issues and it appears that @freezy has some code in a repo:

@ibgorton in case you're still using a CPU-based surface, I've created a proof of concept using an accelerated off-screen surface without OpenTK or ANGLE that seems to work well. More info and code here: https://github.com/freezy/wpf-skia-opengl.

[ghost]

@mattleibow thanks for responding immediately!

I have a couple of questions. First of all, I apologise if these are obvious, but I've never done any graphics work before now.

How does the approach in this issue / the linked repository differ from using a WriteableBitmap as done here https://github.com/8/SkiaSharp-Wpf-Example (which I got to work on dotnetcore).

Does being accelerated imply that the rendering to the bitmap buffer is calculated on the graphics card instead of the cpu?

Does this translate to noticeably better performance for Skia?

If I'm just issuing API calls to Skia, from a development perspective should the experience be transparent? ie, could I develop using the mechanism I've got to work and later on switch to an accelerated backend without any change in the Skia code? (obviously I'd expect some change in that code that wires up the bitmap to my WPF control).

Basically, I'm evaluating a method to create a high performance chart as there does not seem to be an existing general-purpose implementation that both fits our use-case well enough and has the required performance. SkiaSharp seems to be the nicest way to accomplish that on dotnetcore.

Thank you for your time.

edit: both approaches seem to have about the same performance if I call InvalidateVisual on the bitmaphost in freezy's example from CompositionTarget.Rendering event.

[Mikolaytis]

@mattleibow We are using @freezy approach for a 6 month already. Issue is - we are still using WriteableBitmap to draw canvas on a WPF window. WriteableBitmap is stored in the RAM so - on GPU rendering we are having next pipeline:

1. Rendering surface on GPU
2. Converting surface to bitmap pixels
3. Copying pixels to the Writeablebitmap handle (from GPU to the RAM)
4. WPF will push buffer from RAM to it's texture buffer on GPU
5. Image will be rendered on a WPF window

2-4 steps are terrific performance downgrade.

I want to have next pipeline:

1. Rendering surface on GPU
2. Copying it to the D3DSurface
3. Image will be rendered on a WPF window

I imagine this pipeline should work at least 10x faster than first one.

I've tried a lot of options over a 6 month and did not found a working solution.
The best solution I found I posted here: https://github.com/Mikolaytis/WpfSkiaAngleSharpDxOpenTK
All we need is to try to fix Skia rendering proplems in it.
Changing stencilbuffers options do not help.

[Mikolaytis]

So, the fail of my code was to use Avalonia EGL libs. Today I've built angle myself and everything is working. Wow! https://github.com/Mikolaytis/WpfSkiaAngleSharpDxOpenTK - here is an example with updated libs.
Now I will try to integrate this thing into our project.

[enissimsek]

OpenTK GLWpfControl is (almost) a drop-in replacement for its WinForms counterpart GLControl. So, I used it to port SKGLcontrol to WPF (a SKWpfGLControl, if you will) and it seems to work in my limited testing with GPU support, no flickering etc.

My use-case is fairly limited to just some background, lines and text so please let me know if I am missing anything or if there is a reason that this is not a good solution.

[xiejiang2014]

OpenTK GLWpfControl is (almost) a drop-in replacement for its WinForms counterpart GLControl. So, I used it to port SKGLcontrol to WPF (a SKWpfGLControl, if you will) and it seems to work in my limited testing with GPU support, no flickering etc.

My use-case is fairly limited to just some background, lines and text so please let me know if I am missing anything or if there is a reason that this is not a good solution.

cool！ may i try it？

[enissimsek]

cool！ may i try it？

Here is a small example that compares to SKElement:
SKGLWpfControlExample.zip

[xiejiang2014]

cool！ may i try it？

Here is a small example that compares to SKElement:
SKGLWpfControlExample.zip

The program runs normally.
There are no errors when the computer wakes up or changes the monitor resolution.
It looks like the problem is solved perfectly?
Test with rtx 2070

[dedmen]

@enissimsek Thanks alot! It works just beautifully.
I'm using this with MapsUI and I can finally run in full screen with very smooth fps instead of like.. 0.2fps before.

Here is a before and after comparison video:
https://www.youtube.com/watch?v=_DyyLLqbS34

But it does have some problems, it doesn't really seem to like rendering two dock windows (AvalonDock) it gets confused.

Some elements suddenly stop showing up, swapping back and forth when zooming.
When I hide my grid element (basically just rendering a bunch of lines) it partially comes back, but other things bug out.

Definitely related to GL, if I swap back to SKElement it renders normally again.
I'm using one SKGLWpfControl per window, and it messes up pretty badly.
If I use SKElement for both it works fine.
Hacky workaround, use SKGLWpfControl for the first Map window, and SKElement for all further created Map windows.
Works nicely too, but of course is missing the performance improvement on the second window.

WolfCorps/TacControl@a90164a
Here is my code, didn't make any changes to SKGLWpfControl itself. I assume the issue is inside OpenTK.GLWpfControl

Edit: I thought this might be the problem.
https://github.com/opentk/GLWpfControl/blob/0e657c1033944fbc6b6e7fc53694a7aae139dadc/src/GLWpfControl/DXGLContext.cs#L115
It shares the same context, for both windows, which it seems like it shouldn't.
But even using mainSettings.ContextToUse to select seperate context for both windows, doesn't help.

Changing the code to manually get the correct window handle via Application.Current.Windows, also not better.
Two contexts in two seperate windows will confuse eachother.
Out of ideas at this point, but I also don't really know GL nor any of this code.

[enissimsek]

I don’t have the expertise in this area to offer deeper insight but I knew it is something that needs to be worked around explicitly if you need multiple instances of controls that use GRContext. This may help: [BUG] Weird crosstalk between multiple canvas when using GPU-accelerated WPF with WindowsFormsHost

[SuRGeoNix]

Hey guys, just came across this randomly and might have something that could help you out. It seems that you try to resolve the classic airspace wpf issue. The only solution that I've found is by creating a transparent forewindow and transfer the overlay contents there. I've used the libvlcsharp's code as a base (I've also resolved the issue rendering during the design mode and fullscreen, i will commit my version soon). You can find the libvlcsharp's code here https://github.com/videolan/libvlcsharp/tree/3.x/src/LibVLCSharp.WPF

[xiejiang2014]

@enissimsek Thanks alot! It works just beautifully.
I'm using this with MapsUI and I can finally run in full screen with very smooth fps instead of like.. 0.2fps before.

Here is a before and after comparison video:
https://www.youtube.com/watch?v=_DyyLLqbS34

....................

Hello, I also encountered the same problem. I have tried various attempts and still have not solved it. Do you have any new progress? Thank you.

I tried:
Modify the source code of GLWpfControl, add the MakeCurrent method to it, and call it in SKGLWpfControl.OnPaintSurface.

[NogginBops]

Y'all know that you can open issues in the OpenTK repos and we are likely able to fix the problems you are encountering?
Nothing is going to get fixed if the maintainers don't know about the issues.
I just happened to stumble upon this issue but I would appreciate if you would open an issue in the relevant OpenTK repo and we can likely find a working solution.

[dedmen]

My repro is very dependent on Mapsui/SkiaSharp/AvalonDock, I have no repro with GLWpfControl itself, without SkiaSharp.
So not sure where that belongs, but I made a ticket now opentk/GLWpfControl#58
But I don't know how useful it is without a easy repro without tons of extra stuff.
If anyone has a more simpler repro then please put it in there.

[gmurray81]

Is it just me or is there no way to make GLWPFControl use a transparent background? IMO it doesn't really solve any airspace issues without being able to do that....

[mgood7123]

im not exactly sure this is trying to achieve but if you want a GPU accelerated off-screen surface

then why not create a new GL thread with Gr* api's to create GL context in the new thread, and then obtain a SKSurface and draw to it's Canvas ?

[mgood7123]

or might you be trying to create a Render Thread (in which the thread is an off-screen GL render loop consisting of a draw callback) ?

[gmurray81]

Please consider this for soon @mattleibow would be super useful!

[gmurray81]

I've attempted to add a GL element for WPF that utilizes OpenTK's GL WPF control here: #2317

[gmurray81]

seems to work ok for me, even in cross-talk scenarios (multiple elements on same thread) after I started resetting the Graphics context before painting

[Frostchi]

Has anyone managed to get performant GL rendering? I've tried several options posted on here, and can't seem to get over 15-30 fps and my render feels sluggish.

I get 300+ fps on native WinForms with SKGLControl (for some reason I can't do this using the FormsHost in WPF)

Is WPF just a bad choice for this?

[VRage]

I tried out @freezy solution and it worked good, even over rdp (this is where OpenTk fails).
But if i try the same solution with a newer version of SkiaSharp (2.88) it fails on DrawSurface(): "Exception thrown at 0x00007FFB68F16B4C (libSkiaSharp.dll): 0xC0000005: Access violation reading location 0x0000000000000000." .
Any idea how to fix this? @mattleibow @freezy

[najak3d]

I tried out @freezy solution and it worked good, even over rdp (this is where OpenTk fails). But if i try the same solution with a newer version of SkiaSharp (2.88) it fails on DrawSurface(): "Exception thrown at 0x00007FFB68F16B4C (libSkiaSharp.dll): 0xC0000005: Access violation reading location 0x0000000000000000." . Any idea how to fix this? @mattleibow @freezy

Likewise, I just tried it now with SkiaSharp 2.88.6 - and mine is crashing on the same line of code (DrawSurface), but with this similar error:

System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'

Shown here:

Our project is fine with the "copy back to writeable Bitmap" performance hit - although a pure GPU solution is way better. Our main reason for needing a GPU Surface is because we're using SKSL Shaders that only work on GPU surfaces.

It's working fine for us on Android now (haven't checked iOS yet), but having big issues on WPF.

I've also dabbled with just trying to get a GPU Surface to copy it's contents to a CPU-base SKBitmap - no luck here either - same error as above.

We would really appreciate seeing ANY GL-based WPF Surface Solution working in the main SkiaSharp.Views package.

==
I'm going to try the other solutions from above, to see if either of those will work for us now on latest SkiaSharp 2.88.6.

Ping @mattleibow @freezy