libvmaf with CUDA -- how to build and use

[DonTequila]

Is there a documentation explaining how to build libvmaf with support for CUDA and how it's been used?

[DonTequila]

Hi all, can somebody please point me to some documentation about building libvmaf with CUDA support if it exists? Thanks!
Also, do I need a NVIDIA card for building it?

[gedoensmax]

You don't need one for building, but you will need a card for executing. Also you need driver stub libraries, which are included in a CUDA docker container.

[gedoensmax]

Could you ldd ffmpeg to see if it really loads the correct libraries ? Whenever that happened it just loaded the wrong .so for me.

[gedoensmax]

Wait did you apply https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230807/29cf249a/attachment.obj to ffmpeg ? See #1232.

[zwayn]

I successfully built it on Linux with meson setup -Denable_cuda=true libvmaf/build libvmaf ninja -C libvmaf/build install

sudo ln -s /opt/cuda/ /usr/local/cuda was also required on my operating system before those two commands, but that's distribution specific. I use Arch Linux.

Standalone vmaf now successfully uses CUDA, by default.

However, vmaf in ffmpeg and aomenc don't... Both are freshly compiled.

I tried that however I still got a compilation error:

`[183/206] Compiling Cuda object src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.o
FAILED: src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.o 
nvcc -Isrc/libcuda_common_vmaf_lib.a.p -Xcompiler=-Wall,-Winvalid-pch,-Wextra -O3 -Xcompiler=-fPIC -Isrc -I/usr/local/cuda/include -I/usr/local/cuda/include -I../src/cuda -I../src/feature -I../src/cuda -I../src/feature/common -I../src -Isrc -I../src/feature/common -I../src/feature -I../src -Isrc -I../include -Iinclude -I../src -Isrc -I../src -Isrc -Isrc/libcuda_common_vmaf_lib.a.p -o src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.o -c ../src/feature/cuda/integer_adm/adm_cm.cu
../src/feature/cuda/integer_adm/adm_cm.cu(52): warning #68-D: integer conversion resulted in a change of sign

../src/feature/cuda/integer_adm/adm_cm.cu(52): warning #68-D: integer conversion resulted in a change of sign

/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
[187/206] Compiling C object test/test_cambi.p/test_cambi.c.o
../test/test_cambi.c: In function ‘get_sample_image’:
../test/test_cambi.c:59:9: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   59 |     int err, count = 0;
      |         ^~~
../test/test_cambi.c: In function ‘get_sample_image_8b’:
../test/test_cambi.c:82:9: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   82 |     int err, count = 0;
      |         ^~~
../test/test_cambi.c: In function ‘get_sample_image_8x8’:
../test/test_cambi.c:94:9: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   94 |     int err, count = 0;
      |         ^~~
[189/206] Compiling C++ object test/test_predict.p/.._src_svm.cpp.o
ninja: build stopped: subcommand failed.`

Any guess ?

[HunterAP23]

I've been trying to build libvmaf with CUDA support into a Windows build that's being cross-compiled on an Ubuntu WSL / Docker container. I was doing this through ffmpeg-windows-build-helpers.

After installing CUDA (I tried 10.2, 11.6, 11.8, and 12.2) and NVCC, along with all the requirements for both libvmaf and the ffmpeg builder tool, I modified the ffmpeg build script to do the following three things:

1. Added the -Denable_cuda=true flag to the libvmaf compile options
2. Added --enable-nonfree --enable-ffnvcodec to the ffmpeg configuration options
3. Changed the script to pull the latest commit of the main branch rather than a specific release

After doing that, I get this issue:

The Meson build system
Version: 1.2.3
Source dir: /ffmpeg-windows-build-helpers/sandbox/win64/vmaf_git/libvmaf
Build dir: /ffmpeg-windows-build-helpers/sandbox/win64/vmaf_git/libvmaf/build
Build type: cross build
Project name: libvmaf
Project version: 2.3.1
C compiler for the host machine: /ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/bin/x86_64-w64-mingw32-gcc (gcc 10.2.0 "x86_64-w64-mingw32-gcc (GCC) 10.2.0")
C linker for the host machine: /ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/bin/x86_64-w64-mingw32-gcc ld.bfd 2.36.1
C++ compiler for the host machine: /ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/bin/x86_64-w64-mingw32-g++ (gcc 10.2.0 "x86_64-w64-mingw32-g++ (GCC) 10.2.0")
C++ linker for the host machine: /ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/bin/x86_64-w64-mingw32-g++ ld.bfd 2.36.1
C compiler for the build machine: cc (gcc 7.5.0 "cc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
C linker for the build machine: cc ld.bfd 2.30
C++ compiler for the build machine: c++ (gcc 7.5.0 "c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
C++ linker for the build machine: c++ ld.bfd 2.30
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: x86_64
Target machine cpu: x86_64
Check usable header "stdatomic.h" : YES
Configuring version.h using configuration
Program nasm found: YES (/usr/bin/nasm)
Configuring config.asm using configuration
Program xxd found: YES (/usr/bin/xxd)
Run-time dependency CUDA found: NO

src/meson.build:162:22: ERROR: Dependency lookup for cuda with method 'system' failed: Couldn't find requested CUDA module 'cudart_static'

A full log can be found at /ffmpeg-windows-build-helpers/sandbox/win64/vmaf_git/libvmaf/build/meson-logs/meson-log.txt

I fixed the above issue by copying the libcudart_static.a file that gets added to my system when installing Cuda 11.6 through the cuda-toolkit-11-6 package. For me this file is in both /usr/local/cuda-11.6/targets/x86_64-linux/lib and /usr/lib/x86_64-linux-gnu, and I copied this file to the ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/x86_64-w64-mingw32/lib directory.

After doing this, the cross_compile_ffmpeg.sh script to change what normally gets written to meson-cross.mingw.txt by changing the following section:

[binaries]
c = '${cross_prefix}gcc'
cpp = '${cross_prefix}g++'
ld = '${cross_prefix}ld'
ar = '${cross_prefix}ar'
strip = '${cross_prefix}strip'
pkgconfig = '${cross_prefix}pkg-config'
nm = '${cross_prefix}nm'
windres = '${cross_prefix}windres'

and adding cuda = '/usr/bin/nvcc' to the bottom.

Afterwards attempting to run the compile script again, I'm met with a different error, this one from libvmaf itself:

[2/19] Linking target test/test_picture.exe
FAILED: test/test_picture.exe
/home/hunterap/ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/bin/x86_64-w64-mingw32-g++  -o test/test_picture.exe test/test_picture.exe.p/test.c.obj test/test_picture.exe.p/test_picture.c.obj test/test_picture.exe.p/.._src_picture.c.obj test/test_picture.exe.p/.._src_mem.c.obj test/test_picture.exe.p/.._src_ref.c.obj test/test_picture.exe.p/.._src_thread_pool.c.obj -Wl,--allow-shlib-undefined -Wl,-O1 -pthread -Wl,--start-group -lcuda /home/hunterap/ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/lib/../lib/libcudart_static.a -L/usr/local/cuda/lib/x64 -Wl,--subsystem,console -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group
/home/hunterap/ffmpeg-windows-build-helpers/sandbox/cross_compilers/mingw-w64-x86_64/lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld: cannot find -lcuda
collect2: error: ld returned 1 exit status
[3/19] Compiling Cuda object src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.obj
FAILED: src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.obj
/usr/bin/nvcc -Isrc/libcuda_common_vmaf_lib.a.p -Xcompiler=-Wall,-Winvalid-pch,-Wextra -O3 -Isrc -I/usr/local/cuda/include -I/usr/local/cuda/include -I../src/cuda -I../src/feature -I../src/cuda -I../src/feature/common -I../src -Isrc -I../src/feature/common -I../src/feature -I../src -Isrc -I../include -Iinclude -I../src -Isrc -I../src -Isrc -Isrc/libcuda_common_vmaf_lib.a.p -o src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.obj -c ../src/feature/cuda/integer_adm/adm_cm.cu
../src/feature/cuda/integer_adm/adm_cm.cu(52): warning #68-D: integer conversion resulted in a change of sign

../src/feature/cuda/integer_adm/adm_cm.cu(52): warning #68-D: integer conversion resulted in a change of sign

/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
ninja: build stopped: subcommand failed.

Not sure what the -lcuda option does specifically, but it seems to look for CUDA even though it was able to find it earlier.

I've relayed this info to both the ffmpeg-windows-build-helpers and the media-autobuild_suite repos. The former can't do much since they can't release builds with nonfree features. I'll check later today if the changes I listed above work with media-autobuild_suite

[Xosrov]

I successfully built it on Linux with meson setup -Denable_cuda=true libvmaf/build libvmaf ninja -C libvmaf/build install
sudo ln -s /opt/cuda/ /usr/local/cuda was also required on my operating system before those two commands, but that's distribution specific. I use Arch Linux.
Standalone vmaf now successfully uses CUDA, by default.
However, vmaf in ffmpeg and aomenc don't... Both are freshly compiled.

I tried that however I still got a compilation error:

`[183/206] Compiling Cuda object src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.o
FAILED: src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.o 
nvcc -Isrc/libcuda_common_vmaf_lib.a.p -Xcompiler=-Wall,-Winvalid-pch,-Wextra -O3 -Xcompiler=-fPIC -Isrc -I/usr/local/cuda/include -I/usr/local/cuda/include -I../src/cuda -I../src/feature -I../src/cuda -I../src/feature/common -I../src -Isrc -I../src/feature/common -I../src/feature -I../src -Isrc -I../include -Iinclude -I../src -Isrc -I../src -Isrc -Isrc/libcuda_common_vmaf_lib.a.p -o src/libcuda_common_vmaf_lib.a.p/feature_cuda_integer_adm_adm_cm.cu.o -c ../src/feature/cuda/integer_adm/adm_cm.cu
../src/feature/cuda/integer_adm/adm_cm.cu(52): warning #68-D: integer conversion resulted in a change of sign

../src/feature/cuda/integer_adm/adm_cm.cu(52): warning #68-D: integer conversion resulted in a change of sign

/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |                                                                                                                                                 ^ 
/usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
  530 |         operator=(_Functor&& __f)
      |                                                                                                                                                  ^ 
/usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
[187/206] Compiling C object test/test_cambi.p/test_cambi.c.o
../test/test_cambi.c: In function ‘get_sample_image’:
../test/test_cambi.c:59:9: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   59 |     int err, count = 0;
      |         ^~~
../test/test_cambi.c: In function ‘get_sample_image_8b’:
../test/test_cambi.c:82:9: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   82 |     int err, count = 0;
      |         ^~~
../test/test_cambi.c: In function ‘get_sample_image_8x8’:
../test/test_cambi.c:94:9: warning: variable ‘err’ set but not used [-Wunused-but-set-variable]
   94 |     int err, count = 0;
      |         ^~~
[189/206] Compiling C++ object test/test_predict.p/.._src_svm.cpp.o
ninja: build stopped: subcommand failed.`

Any guess ?

I have the same problem, Ubuntu 22.04, CUDA 12.3. It seems this is a problem with gcc and nvcc compaibility (see here)

[gedoensmax]

@Xosrov the Dockerfile in #1330 builds on 22.04 and 12.3 without any problems on my side. Could you try this ? I know that it might not be the best solution for your case, but at least a sample to go by ?

[alessandroberna]

@damian101 I think i had your same issue. I compiled both vmaf and FFmpeg on Arch from the latest git version, however the filter libvmaf_cuda was missing from the freshly built ffmpeg.

It turns out that ninja -C libvmaf/build install installs to /usr/local/ (so /usr/local/include, /usr/local/lib and /usr/local/bin), ffmpeg on the other hand seems to look for libraries inside /usr/include and /usr/lib first.
It's thus very likely that you were compiling ffmpeg with the version of libvmaf that's included on the arch repos, which is compiled without -Denable_cuda=true.

Before compiling FFmpeg, take a look at the output of the configure script, you should see libvmaf_cuda listed under Enabled filters:, if that's not the case you either forgot to enable the necessary configurations or it's not using the right library.