Linux AppImage: Possible issues importing video with certain GPUs

[ferdnyc]

It appears the OpenShot 2.6.1 release AppImage (and all recent daily builds before/after that release) may fail to import certain video formats on systems containing a GPU which has hardware decoding support for the video format.

The issue occurs when the user's existing preferences have a hardware-decoding mode selected. (For instance, from previously running an installed native build of OpenShot.) This setting is not properly ignored by the AppImage builds of OpenShot, which do not currently support hardware video decoding. This results in a valid video file being rejected by OpenShot when importing:

As a workaround, quit all instances of OpenShot, then delete or rename the file $HOME/.openshot_qt/openshot.settings to reset your OpenShot user preferences. The next time OpenShot is launched, that file will be created with default settings, and the AppImage build will successfully import the affected files.

[ferdnyc]

Devs:

The issue occurs because an existing hw-decoder preference setting is still being used by OpenShot, even when running from the AppImage.

Due to the problems users previously reported with hardware decoding in AppImage builds, I added a library-mangler to the project build servers which disables the built-in /usr/lib/x86_64-linux-gnu/vdpau and /usr/lib/x86_64-linux-gnu/dri paths embedded in the AppImage's bundled libvdpau.so.1 and libva.so.2 (respectively) libraries. That's necessary because those paths don't exist on non-Debian-based Linux systems. And on Debian-based systems where they do exist, trying to load the system libraries from our bundled ones was causing crashes.

The problem we now have is that, if libopenshot is told to attempt to use hardware decoding, it will. And if (using my own setup as an example) the hardware-decoding mode is set to 6 (VDPAU), when libavcodec loads our bundled libvdpau.so.1 it will attempt to load libvdpau_nvidia.so.1 to interface with the GPU. When that fails, it will reject the input file instead of falling back to software decoding.

There are two solutions here, as I see it:

1. Anticipating this issue to some extent (and also intending to hide the preferences options for hardware decoders, when running from an AppImage — something I never got around to implementing), I added an option APPIMAGE_BUILD to the libopenshot CMake configuration, which our linux builds have been enabling since OpenShot/libopenshot@8af624fc.
   Unfortunately, what I didn't do is set the build up to #define that symbol when compiling — it's only exposed via the OpenShotVersion.h header. That means that, while it is available informationally to OpenShot (it's set in the Python bindings as openshot.APPIMAGE_BUILD), it doesn't currently affect the libopenshot build itself. But that's easily fixed — in fact, we have the option to do one better:
   libopenshot's hardware-decoding code is all(?) protected by #ifdef USE_HW_ACCEL. Normally that's set by the FFmpeg version, so that hardware-acceleration is enabled when building with FFmpeg version 3.4 or better. But there's no reason we can't use it to disable hardware-acceleration completely in the AppImage builds. By defining USE_HW_ACCEL=0 when APPIMAGE_BUILD is TRUE, we should be able to ensure that the AppImages are built without the ability to attempt hardware decoding, making the preferences setting irrelevant.

2. I happened to notice that the current libvdpau.so.1 bundled into the AppImages supports, in addition to the /usr/lib/x86_64-linux-gnu/vdpau/ path for loadable drivers, a second path ${ORIGIN}/vdpau. That would potentially allow us to bundle the necessary drivers into the AppImage... oh, wait, no. That won't work, because libvdpau_nvidia.so.1 is part of Nvidia's binary driver distribution; even if we had that file we're not (as far as I'm aware) allowed to distribute it. So, yeah, we're back to disabling hwaccel in the AppImage still being the only workable solution. Honestly that's probably still for the best, as when the AppImages could load the local drivers (from /usr/lib/x86_64-linux-gnu/vdpau on systems supporting that path) we were getting numerous reports of crashes. Hence the original mangling.

cc: @jonoomph @JacksonRG @eisneinechse

[eisneinechse]

YES! Disable it in the AppImage.

Short:
It as more than these problems in the AppImage, it also has problems with permissions.

Longer:
I am not the biggest fan anyhow especially on older GPUs as the quality is not great unless one uses a higher setting and then one could just use the software codecs. The Appimage on some systems don't get the permissions to read or write to the GPU via the interface and we would have to do something about it. There are now newer (and better) options to access the GPU, at least on the horizon.

Extended but worth reading:
People also want the modification process of videos as fast as possible that's why already some were suggesting that we use the GPU for those task. But like the GPU encode and decode acceleration the programming interface was different for every GPU type. Now some of the accelerated interfaces are even been dropped.
But we have a standard interface that is gaining traction, Vulkan. There is the Vulkan GPU acceleration that most people know for gaming but the more interesting parts for our purposes are Vulkan Compute, which is not new and can be used to modify the clips, and there is a new interface called Vulkan Video. Vulkan Video is what does the encode and decode and right now there is a speedy development of that at least on Linux, for all GPUs.
Because Vulkan Compute and Vulkan Video are siblings and both work on the GPU this will also make it "easy" to make a acceleration of the modify part and the encode part without having to move the data back and forth between CPU memory and GPU memeory.
It's still a little bit in the future but think we should keep that as an option for the future.
Additional benefit: It's Windows and Linux and there is a layer for Mac OS (not sure how far developed though)

[ferdnyc]

@eisneinechse 👍

Hardware-accelerated de-coding is of precious little value to us anyway, because we have to decode every frame to an RGB image in system memory, (potentially) modify that image, and then display the results "by hand" — the GPU isn't involved in the process hardly at all. The decoding step is the least of the work there. Hardware decoding's benefits depend on the video never coming back out of the GPU, and being displayed directly from video memory as it exits the decoding pipeline. But OpenShot can't possibly work that way.

...Hardware-accelerated EN-coding is nice to have, and can actually represent a genuine speedup for the encoding process. But I don't think it's unfair to say: If you want that, install a native package for your distro. (Or switch to the Flatpak, which may or may not currently have support for hardware encoding, but definitely could have support for hardware encoding much more feasibly than the AppImage can, since Flathub publishes runtimes with GPU driver support already enabled.)

[eisneinechse]

@ferdnyc That is where I can see Vulkan helping us in the future. With Vulkan Video we would be able to load accelerated, then modify (fading, blurr ...) with Vulkan Compute, and then export accelerated with Vulkan Video. Vulkan uses the same memory space a copying would not be necessary. In the future that might be an option!
For now decoding is a nice gimmick but has little potential for a speedup. Like you said, the speed gain would be during export, especially with the newer codecs.
One thing that should also be implemented with the newer codecs, but that is a completely different project, is 2 pass encoding. The newer codecs really like that.

[ferdnyc]

@eisneinechse

That does sound interesting, wow! It'd be awesome if we could develop a fully in-GPU pipeline. It'd mean some major changes to the entire data model for how content moves through OpenShot, but it sounds like the benefits are probably more than worth it.

Would that mean having to implement effects in the YUV color space? Or would the Vulkan pipeline be able to handle round-trip conversion to RGB as well? For some effects like crop and blur it doesn't really matter, obviously, but trying to wrap my brain around YUV ChromaKey is already giving me a headache.

Maybe there's some way to convert the effect into YUV so that it still produces the same results you'd get with RGB? That's a question for color wizards far more powerful than I. (Though, even then, I'm curious whether it could still be parameterized the same way.)

[eisneinechse]

Historically my idea was to have the effects on the GPU but ack then it would have been CUDA and OpenCL, then Vulkan was released with Vulkan Compute and that sounded very interesting as Vulkan is basically on every machine because of the gaming part (no need to install something people don't know), and not too far ago Vulkan Video was released. Now I am right now in a situation where I have not enough time to do anything I want to do (including playing around with RISC-V and it's vector extension) but I try to stay informed and develop ideas. The Vulkan pipeline for decoding, modifying, and encoding (plus display) would be great.
About the colourspace I'm not sure if we will see some extensions to Vulkan for RGB, Vulkan Video is not even in it"s puberty. But moving an effect from RGB to any other colourspace should be a mathematical transformation (like AeffectA-) whuich might even be doable by the computer itself.
The best thing about a Vulkan pipeline would be that no copying has to be done, well and that the whole thing would be platform independent, it would be installed already on any computer used, and it would not be limited to a GPU (or CPU) system.
Vulkan Video so far is only really for h264 as far as I can see with h265, VP9, and AV1 following soon.
I think we should keep our eyes on Vulkan and maybe already preparing for it when making design decisions.

[eisneinechse]

@ferdnyc Just in today:
https://www.phoronix.com/scan.php?page=news_item&px=FFmpeg-Branch-Vulkan-Video
I am certainly keeping that on my list of things to look into.

[stale]

Thank you so much for submitting an issue to help improve OpenShot Video Editor. We are sorry about this, but this particular issue has gone unnoticed for quite some time. To help keep the OpenShot GitHub Issue Tracker organized and focused, we must ensure that every issue is correctly labelled and triaged, to get the proper attention.

This issue will be closed, as it meets the following criteria:

• No activity in the past 180 days
• No one is assigned to this issue

We'd like to ask you to help us out and determine whether this issue should be reopened.

• If this issue is reporting a bug, please can you attempt to reproduce on the latest daily build to help us to understand whether the bug still needs our attention.
• If this issue is proposing a new feature, please can you verify whether the feature proposal is still relevant.

Thanks again for your help!