GUI Support

[Earlz]

So, I've been trying for the past few days to get any form of GUI working in my VST host (Reaper) using rust-vst2, and I've faced nothing but difficulties. I'm also pretty new to Rust itself, so my inexperience doesn't help.

Anyway, have you had any success prototyping such a thing, and ideas on what to use? I know the readme says Conrod+SDL, and this is the primary thing I've focused on, but I can't help but wonder if there might be an easier and more stable way. The C++ VST kit that everyone, wdl-ol, includes an OpenGL framework with built in easy support for drawing knobs and such. I'd be curious if doing FFI to that might be easier. Do you have any ideas?

[overdrivenpotato]

So if you look at src/editor.rs, it is up to the plugin to create an editor that implements the Editor trait. One of the things that we have to work with is a window pointer, which can be used by the plugin to do something e.g. initialize an SDL2 context. It seems libraries like wdl-ol abstract over more than just the bare VST api, I'm thinking it might be out of scope for this library to do something like that.

There are a few approaches that we can take though, the main thing I'm interested in is seeing if we can get Piston's 2D graphics library to work with an editor. I don't necessarily want to use OpenGL for drawing as it might not work on all systems and we can use a simpler interface for drawing 2D UI elements. If it's easier though I'm good to get just something running with a UI even if it is OpenGL.

We would also need to have the appropriate button controls like knobs and sliders, I'm thinking these can be done in another crate to separate concerns (or we can use conrod!). It also opens up the possibility of code reuse for something such as an Audio Unit rust implementation which needs UI elements as well.

There's a lot of stuff that needs to be done and I'm OK with trying out a bunch of things, just not sure which direction to head in. If you want to take a stab at it, the very first thing would be to see if you can get a graphics context to work with the pointer given in Editor::open. I'll see if I can help out if you do end up making a public fork/repo.

[Earlz]

Yea, I definitely think it'd fall under a completely different crate, but I feel like it could belong to the same project, or at least be related..

I did some investigating in the wdl-ol code, and I think the way they handle it (very thick codebase, so difficult to be sure) is that they basically have a platform dependent native set of graphics code for each platform. This graphics code handles basically the bare minimum however. It exposes a framebuffer, and allows for the normal event setup (ie, keys, mouse, etc). Then, most programs use their LICE abstraction that works directly on the exposed framebuffer for shapes, glyphs, etc. And finally, the programs that care about OpenGL, basically setup OpenGL to render to an invisible hidden window, and manually blit/copy the framebuffer of the OpenGL window to the VST-host window.

I'm not sure that this is the best way to go, but I'm sure there's a reason why they chose this way rather than directly rendering everything using the window handle.

I've been trying to get Editor::open to work properly with SDL, but having very little success thus far.. I intend to keep chasing this though. It'd be awesome if VST work on Rust was actually preferred over C++, since already this is incredibly easy compared to making a VST plugin in C++.. just the GUI bit isn't there, and eventually the DSP other common audio operation helpers

[overdrivenpotato]

Sorry about the delay, holidays and all 🎉 (Happy new year!)

Alright well it looks like you're right about the wdl-ol implementation, we can definitely do the same thing and implement platform specific graphics code as I am not aware of a rust library that does this. One of my goals would be to have rust-vst2 work with Linux, OS X and Windows though so I would like to support all three platforms properly. I guess the first step is to decide whether we're going down the route of using graphics as a base or whether we should just create a new library from scratch, your thoughts?

[adamski]

You might want to look at how the JUCE C++ framework deals with different platforms. A rust equivalent to JUCE would be amazing, albeit a very ambitious project!

[suhr]

graphics provides primitives to draw, but you still need a backend for it to initialize an sdl context.

I have a different problem though. I want to implement a microtonal (31 tone) sequencer (like Hex). It will have a complex UI, so I rather should use gtk-rs. But I also want it to be a plugin, to use it in my DAW of choice.

So, how do I use gtk in vst? I heard this is not an easy thing to do, but I also know Carla uses Qt and can be loaded as a plugin.

[sklopi]

Did anyone have luck in initializing a graphics backend from the hwnd handle on windows?
I'm currently trying to get this working, but i'm kinda stuck.

[overdrivenpotato]

@suhr I'm not really sure how this should be handled, I do think the best way going forward would be to use some sort of a toolkit but as I don't have experience with either QT or GTK I don't really have any suggestions.

It looks like it boils down to either getting an existing toolkit to use the raw window handle for each platform, or creating a GUI system specific to this library. Personally I'd be interesting in seeing something such as electron being used for the GUI and delegating the implementation/DSP down to rust code, but again I'm not sure where I'd even begin.

[ycros]

I have proof-of-concept working currently on Windows at https://github.com/ycros/vparty using a fork of glutin ycros/glutin@dd0de44 - eventually I'd like to get conrod up and running, but this requires either implementing a whole new conrod backend or getting some changes into piston. I know how to get things running on OSX as well, though things are a little trickier there (your pointer may be a Carbon windowRef or an NSWindow depending on 32bit/64bit/your daw). I've been busy with a new job so I haven't had time to come back to working on this.

Basically, for whatever sort of gui lib you want to end up using, you will need to handle spawning your window as a child window under whatever window handle/pointer/thing you get from your host. There's other open source C++ VST wrappers such as wdl-ol that you can peek at to see how they handle gui initialisation.

[Boscop]

I forked glutin, conrod and the piston window stuff and modified it to support multiple child windows in the same thread (required for VST but by default glutin creates a new thread for each window and doesn't allow child windows) and passing the parent window handle all the way through to where the win32 window creation functions are called, you can find the forks on my page, but I ran into some issues like this:
PistonDevelopers/piston_window#167
(The problem is piston_window uses its own event loop. I either don't get draw events or no input events.)
Right now I'm working on my own lightweight GUI lib on top of glutin, I already have layouting, buttons, knobs, font rendering etc (with batched rendering). but then I got sidetracked working on a game.. I'm getting back to this though.
My plan is to create a VST framework on top of this lib that includes GUI support and common DSP routines that are useful and can be composed to create larger plugins, kind of like JUCE but for Rust.
I currently have a basic starting point, ported some code from Will Pirkle's book to Rust.
If you are interested, we can plan this together on IRC, #rust-music
I think Rust has great potential for audio DSP.

Btw, this is an old screenshot of my GUI lib (this example isn't running in a VST but it works in a VST too):

[Boscop]

Another old screenshot, this VST is not using my GUI lib, just my glutin fork.

[suhr]

My plan is to create a VST framework on top of this lib that includes GUI support and common DSP routines that are useful and can be composed to create larger plugins, kind of like JUCE but for Rust.

Then it should also support LV2 and AU.

[Boscop]

@suhr Yeah, but I don't know anything about those formats... So it would be helpful if we can work together.
Probably we should design an intermediate API for plugin architectures so that each plugin only has to interface that API and can be compiled to all supported backends. I think that's how WDL-OL does it.
Currently, my glutin fork only allows creating child windows on windows. But I abstracted the WindowID into a type, which can be defined in a platform-independent way, so feel free to submit a PR for other platforms. Also I added set_timer() and kill_timer() so that the VST GUI can be drawn at a consistent framerate triggered by the timer, that's how it should be done on other platforms, too. This requires storing a pointer to the plugin in the userdata of the window, this is how I do it on windows.
My glutin fork also supports sending a Timer event instead of using a callback for the timer, but you can't poll events out of nothing, something has to trigger it, and if the mouse is not over your VST window, the host will call idle on the plugin only at like 4 fps. That's why I switched to the timer callback, because my spectrum view needed a higher framerate.

[overdrivenpotato]

@Boscop This looks great, I definitely would be interested in discussing this further. Which IRC network is #rust-music on?

[Boscop]

@overdrivenpotato irc.mozilla.org

[suhr]

cast @poidl

[zyvitski]

I think the idea of using CEF would be a good one. Step one would be to get a CEF window running with its event loop running on a separate thread of course. As far as communication between the window and the VST, there would need to be a queue of events being passed from the window handler into the VST. The queue would need to be available to the processing callback and ideally it would be a lock-free SPSC queue. A big question that needs to be asked is wether this library should be responsible for the actual rendering and layout or if it were to only provide the window and the IPC mechanisms needed to communicate with the plugin. I feel that a good starting point would be the latter option and then work on a standardized set of UI elements and such.

[Boscop]

@overdrivenpotato That screenshot looks cool!
Can bindgen be used for generating bindings to CEF?

[Boscop]

I found another alternative, these guys are using awesonium for their game/editor GUI:
https://www.reddit.com/r/rust_gamedev/comments/5vqlln/shar_one_year_with_rust/
https://youtu.be/OVYQs3KY2EE?t=55
http://www.awesomium.com/
https://github.com/not-fl3/awesomium-rs
I think this is a very promising possibility for VST GUIs.

[adamski]

Personally, I don't think HTML is a good fit for these kinds of apps. Although something like React's JSX for declarative UI could work. Audio apps often require lots of custom UI elements, some of which need to be updated at high rates to represent what's going on in the audio. I might have got the wrong end of the stick here though...

[piedoom]

I agree. I think at a very base level, declarative UI is really nice and accessible (like XAML) but I don't think running a web UI for a VST would be the best idea.

[zyvitski]

Has anyone made any progress on getting a GUI setup? I'd be interested to hear how you dealt with the issue of working with the native window handle that's vst hands you

[Boscop]

@zyvitski Here is a description of what I'm doing in my glutin fork:
rust-windowing/winit#159
I haven't updated my forks in months though (I've been busy with work), the conrod fork should be updated to support the glium backend so that one can render stuff with custom shaders...
I'm working on a kickdrum plugin right now where I plan to use raymarching to render a GUI with correct shadows on the knobs / glass LEDs and using different noise functions to generate textures like wood for ultimate skeuomorphism...

[zyvitski]

@Boscop Are your changes to glutin only setup to work on windows or have you setup a cross platform solution as well? I have both a Mac and Linux setup and would love to get things working across the board.

[zyvitski]

@Boscop I forked a copy of your glutin fork, I will see what I can do to get it going on OS X. Once I have it setup I will issue a pull request. I think It would be a good idea to try and set up a portable timer to replace the windows timer you used ( can just be a wrapper over the windows one with conditional compilation). Once that is up and going I will give it a go on my linux box.

[zyvitski]

Is anyone familiar with how the cocoa api backend for glutin works? I have never dealt nuch with cocoa. I know that the window handle coming in from the host should be a NSView* (I chekced the vst2 sdk source) but I am not sure what the proper method for working with the NSView is? I tried setting it up where the the view created by glutin would be attached to the incoming view but that just resulted in having two windows.

Is the host handing me a new window to work on or a handle to it's window to attach a new window to?

[Boscop]

I think it's a great idea to have cross platform support, but I've never dealt with OS X so unfortunately I can't help much there. But I know it also has a way to store data in a custom pointer associated with a window (which I use to store the plug-in pointer so that the timer callback can access the plug-in.

The host creates a window with or without decorations (depends on the host) and passes the handle to the plug-in. It has to create a child window without decoration in that parent window.

On windows, all child windows have to live in the same thread because all parent windows live in one thread and child windows have to run in the same thread as their parent. Hence the solution with the event hash map.

To bring winit up to speed with the fork it would require to apply the same changes outlined in that post, but in a cross platform manner. So we need these things as cross platform:

• callback timer on window with callback having access to plugin (e.g. store pointer in window user data)
• a way to create child windows such that they run in the same thread (abstracted window handle as hashmap key for event queues)

On all platforms the window handle is a pointer, so we can just have a WindowId or WindowHandle type that is basically a pointer.. (in my forks I called it WindowId)

Once we have the cross platform windowing going we can also create au plugins... :)

[zyvitski]

After spending an entire 8 hour day trying to model your changes to glutin in the osx backend I finally gave up on it and decided that I am just going to opt for the nuclear option on solving this problem once and for all. I have started work on a native port of vstgui-4 which so far is turning out to not be that bad of a task (time consuming yes, but pretty straight forward). Shouldn't take me too long to get the basic building blocks in place to at least get things moving (without widgets and fancy features). As mentioned above I don't wave a windows box, If anyone would like to jump on board once the skeleton is down and cover the windows backend that would be great ( @Boscop, or anyone else interested).

[monomadic]

@zyvitski not sure I agree that it's a super big task, we really just need a better way to generate some kind of window object that is compatible with another library. There's PLENTY we can re-use. If we, for example, made an ACTUAL fork of glutin (vst-glutin? glutin-parentless?), and all just fixed up the current version so that it worked across all platforms, we'd be ok. The glutin objects are the same and can be fed into other connector libraries like piston or whatever sits on top of the regular glutin, and not know the difference. The problem will be divergence of glutin itself after that, so major changes would have to be merged back into the core, but as long as the glutin fork had the SOLE job of supporting childless windows, it would be fine.

[zyvitski]

So here's my game plan: I will open up a new repo and we can get things moving. It's good to know that you can get it going with cocoa.

Like I said, I will init a fresh repo with an empty cargo lib project. Then I will post a link to it here. Gimme around an hour.

[zyvitski]

We can figure out who needs write access and all along the way.

[monomadic]

This is not glutin, above. It's just manually digging into cocoa with cocoa-rs. Have you made any headway yourself? If not it's premature to create any repos just yet. I think @Boscop is probably furthest along, no?

[zyvitski]

@RobSaunders I understand. But that's actually better since we probably shouldn't depend on glutin

[monomadic]

If we don't depend on glutin, or something like it, then that IS a large project :P I just need a button or two for now for testing, that's why I did this.

[zyvitski]

We could depend on it if needed. But we need to brainstorm and figure out what the easiest plan of action will be.

[zyvitski]

We just need to figure out how much of the wheel to re-invent

[zyvitski]

Here we go: https://github.com/zyvitski/vstgui-rs

[zyvitski]

I have added @RobSaunders and @sklopi as contributors. Anybody else who wants access just let me know.

Make sure to accept the contributor invite links when you get them.

[zyvitski]

We can move the conversation here: https://github.com/zyvitski/vstgui-rs/issues/1

[monomadic]

Okay guys I have some good news. I have a winit instance attached to a vst-host-generated NSWindow instance and functioning as expected (except when closing the window, where it disappears, but I will work that out). Here's a screenshot of the event log of the window:

It would be fairly easy to port this to windows. I will post more progress here.

[Boscop]

@RobSaunders Great work, I also think our best bet is to use winit as a base (so that we can use the conrod glium backend on it), and apply the necessary changes to it to create child windows. For windows we can use the same changes that I applied to glutin and integrate your code for OS X.

(Unfortunately I'm quite busy atm with my job and thesis..)

[sklopi]

When i tested I saw that conrod removed the glutin feature in favor of a winit feature.
@Boscop, so your changes for making this possible on windows must be ported to @RobSaunders winit fork?

[monomadic]

from what I can tell from @boscops code, please correct me if I'm wrong, it is for the pre-winit, pre-windowbuilder version of glutin, right?

[Boscop]

pre-winit, post-WindowBuilder, I think from August 2016

[monomadic]

Anyone interested is welcome to join the telegram chat @ https://t.me/joinchat/AAAAAA0l43_P90kCelmGrQ

[monomadic]

Conrod is working now under ableton without crashes, but not bitwig for some reason. Still investigating causes. It's likely a memory management (retention) issue over the objc bridge.

Here's a video.

https://www.youtube.com/watch?v=8XUhJC2sqBI

[Boscop]

Now conrod is working on windows too:
https://sendvid.com/n1dn6gs4
Here you can see the source of that plugin:
https://github.com/Boscop/easyvst/blob/master/examples/conrodgain.rs
There is also a simpler version without GUI.

[monomadic]

Ya, it's now working on all daws on both platforms, myself and boscop are working together to package it up :)

[monomadic]

Well... all daws we have tested so far hahaha

[Syrou]

Just wanted to chip in, i just tried the named blob above, with a clone from https://github.com/Boscop/easyvst/

It does not work under Ableton Live 9 Standard x64 under Windows

[Boscop]

Did you put a font.ttf in the working directory of the Ableton exe?
If yes, what does the log say and how does it fail?

[Syrou]

@Boscop I put it in the same folder as the dll, i'll recheck it it works when placed in ableton.exe folder

[Syrou]

@Boscop Yep, my bad. Placing it in the ableton.exe folder solved it

[Boscop]

Yea, it has to be in the folder of the ableton exe because the exe determines what the current working dir is. You should see a file vst.log in that folder, generated by the plugin.

For a real project the font would be bundled with the plugin of course..

Btw, it is possible to retrieve the path of the dll from within itself at runtime:
https://stackoverflow.com/a/6924332/7488353
This can be used to locate other resource files in the same folder as the dll.

[Boscop]

@Syrou so it's working fine in ableton now? What framerate are you getting on the gui (approx)?

[Syrou]

Is there any way to tell @Boscop? I am not experiencing any lags just by normal usage at the very least

[Boscop]

ok, great