Newer version of nvidia-container-toolkit

[oscarthorn]

Hi!
Any suggestions about how I can update the nvidia-container-toolkit package to a newer version? I tried just changing the SRCREV to a newer commit (tried 1.10 and 1.14) and while that builds fine, when actually running I get various errors like:
nvidia-container-runtime: symbol lookup error: nvidia-container-runtime: undefined symbol: cuDeviceGet. I suppose it is being built against the wrong version of cuda somehow?

The reason I'm wanting to do this is newer versions have better podman support (there is for example a bug in 1.9 fixed in 1.10), also jetpack 5.1.2/l4t 35.4.1comes with nvidia-container-toolkit 1.11-rc1 as of now. And builds are available for tegra all the way up to 1.14.3 from the https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list repo.

Am happy to submitt a PR (either for 1.11-rc1 to match jetpack or 1.14.3 for the latest stable) if someone can point me in the right direction to getting this working.

Cheers!

[madisongh]

I think 1.11 would be the appropriate version out of the Github repo to line up with what's in R35.4.1. Looking at it, one thing I'm not clear about is that it appears to be referencing two libnvidia-container libraries - libnvidia-container.so.1 from the main branch and libnvidia-container.so.0 from the jetson branch. We currently only build the latter, so we'll need a second recipe to build the former as well. That's my guess, at least. The other dependency to look at is the one on libtirpc - looks like they updated to a newer version, so we may be able to drop the recipe we were carrying for the older version.

[oscarthorn]

Ah, I see, where do you see these references? I'll take a look at getting the right libraries in.

I think they are trying to (or have done rather) make it so that the standard arm64 builds are tegra compatible. So maybe they need both for that reason.

[madisongh]

Just grep for LIBNVIDIA_CONTAINER0 in the nvidia-container-toolkit repo at the v1.11.0 tag. And I suspect you're right about the reason, I just haven't studied it enough to understand exactly how it's supposed to work.

[oscarthorn]

So did some digging, and the first commit where you can't run nvidia-container-runtime at all, is this one: NVIDIA/nvidia-container-toolkit@9d2363e but some previous commits will have had the problem in certain execution paths. I think the actual problem (for us) is introduced in this commit NVIDIA/nvidia-container-toolkit@a672713.

What is going on is that the wrapper is using cgo to call c code from go. Specifically you can remove everything (excluding the C block) and just have:

package cuda

import (
	"fmt"
)

/*
#cgo LDFLAGS: -Wl,--unresolved-symbols=ignore-in-object-files

#ifdef _WIN32
#define CUDAAPI __stdcall
#else
#define CUDAAPI
#endif

typedef enum cudaError_enum {
	CUDA_SUCCESS = 0
} CUresult;

CUresult CUDAAPI cuDriverGetVersion(int *driverVersion);
*/
import "C"

// cuda stores a reference the cuda dynamic library

// Version returns the CUDA version of the driver as a string or an error if this
// cannot be determined.
func Version() (string, error) {
	var version C.int
	if result := C.cuDriverGetVersion(&version); result != C.CUDA_SUCCESS {
		return "", fmt.Errorf("failed to get CUDA version: result=%v", result)
	}

	return "", nil
}

And still get the error. So I, not knowing much go, suppose that just having C.cuDriverGetVersion even though the function is not called (I removed pretty much all code so there is nothing that would call it) causes it to want to lookup that symbol. Which it cant find for whatever reason.

Maybe someone here knows golang better?

Edit: Also I copied the nvidia-container-runtime binary from a standard orin dev kit and it starts fine. So maybe some build time flags missing?

[madisongh]

Did you update the recipe to add DEPENDS = "cuda-driver" so the driver library is present for linking against? You might want to double-check with readelf or one of the go tools that the executable references libcuda.so.1 as a dependency, too.

[madisongh]

See PR #1449 for the updated recipes. For now, I've left all our CSV pathname translations in the Jetson-specific libnvidia-container0 library, and the runtime is configured to use that ("legacy" mode) instead of the newer CSV processing that they've added directly to the Go code in nvidia-container-toolkit. Once we can get the toolkit code updated to handle our Ubuntu-to-non-Ubuntu path name hacks, we could switch to "csv" mode and drop the the libnvidia-container stuff altogether.

[oscarthorn]

Hi! @madisongh Thanks for the update! Sorry for not responding to this, I ended up moving to container-toolkit 1.14.3, since it turns out that was needed for podman to work well for me. I was just about to submit a PR with that update (even though it's not relevant until jetpack 6) which caused me to se you had already fixed the problem.

I'm curious though was it DEPENDS = "cuda-driver" that fixed the issue above? Because what I found was that the default go recipe in poky/meta adds a bunch of security flags, some of which require that all symbols be defined at startup which the cuda ones were not since it is dynamically loaded during runtime. So I ended doing my on do_build without said flags. But if DEPENDS = "cuda-driver" fixes it then that is a lot clearer.

[madisongh]

It was a bit more complicated than that, although I did take care of cleaning up some other issues along with the update. Fixing the linking problem for the go code was handled here.