How to allow Nvidia container to access Yocto base image packages/libraries

[ervgan]

Hello,

I am running a Nvidia container on top of a Yocto image and I was wondering how we can allow the Nvidia container to access packages installed in the Yocto image (for example cmake, glog, etc). Do I have to mount the root file system (or the path where the packages are located) in the Nvidia container? I'm sure it's something pretty straightforward but I havent been able to find a similar post.

Thanks!

[dwalkes]

Typically you'd build the packages you need in the docker container inside the docker container itself and leave them out of the Yocto image. It's possible to volume mount directories from Yocto into a docker container but you may run into issues running applications as they are likely using different runtime library dependencies in the docker container vs Yocto, hence the motivation for containerization.

[ervgan]

Thanks! that makes sense. On a related matter, I have been trying to run some TensorRT samples in the TensorRT container on top of the Yocto base image with CUDA, TensorRT and Nvidia container runtime installed. I keep running into the following issue for multiple Nvidia libraries:

/sample_mnist: error while loading shared libraries: libnvdla_compiler.so: cannot open shared object file: No such file or directory

The only way was for me to mount /usr/lib/ of the Yocto base image into the TensorRT container which works, from what you are saying, I am guessing there should be better way to achieve this? These libraries are low-level drivers that should be mounted by Nvidia runtime but it is apparently not doing so. Any idea what might be happening?

I am running the docker with the option --runtime Nvidia.

[dwalkes]

Which branch are you on? Which image are you using? The libnvdla_compiler.so file should be passed through to the container using the l4t.csv file setting here which is placed in the rootfs here.
The libnvdla_compiler so file should be installed to the rootfs here by the tegra-libraries-core package.
This should help give you some pointers as to how this will be passed through to the container.

[ervgan]

Oh I actually see that on the Dunfell branch, there is no l4t.csv file and the two commands:
install -d ${D}${sysconfdir}/nvidia-container-runtime/host-files-for-container.d install -m 0644 ${WORKDIR}/l4t.csv ${D}${sysconfdir}/nvidia-container-runtime/host-files-for-container.d are missing from the tegra-configs_32.7.2.bb file. But the tegra-libraries-core does install it to the rootfs. Is this the desired behavior for the Dunfell branch?

[madisongh]

This goes back to what I mentioned here about the difference between the R32.x (JetPack 4) containers vs. R35.x (JetPack 5) containers. With the older container interface, a lot more libraries need to be passed through from the host OS, so we used a different technique for configuring that - each recipe that exposes files into the container generates a CSV file using this bbclass.

[ervgan]

I see this clarifies it a bit. However, I am still confused about two points:

• Using this different technique for older versions should solve the issue of passing shared Nvidia libraries to the container in the dunfell branch so I should not see the error
  /sample_mnist: error while loading shared libraries: libnvdla_compiler.so: cannot open shared object file: No such file or directory, then is it something else that is causing the issue? is there no other way other than mounting the shared libraries into the container if I encounter this issue? maybe create the csv file manually?

• unrelated question, when I create a new yocto image by specifying the recipes I want like this for example;
  IMAGE_BASE_INSTALL += " \ tensorrt-plugins \ tensorrt-samples \ tensorrt-trtexec \ cudnn \ "
  I know that all these recipes will be used to build my image, but how do I know which other recipes are used in my image? for example, I am assuming the recipe-bsp recipes from meta-tegra are all included by default in any yotco build?

Thanks a lot!

[madisongh]

For item 1, take a look at the files in /etc/nvidia-container-runtime/host-files-for-container.d/. If that library isn't mentioned in one of them, then maybe something went wrong with the build. The CSV should be in a package generated by the recipe that installs that library.

You can also enable debug logging in the container by uncommenting the debug = line in the [nvidia-container-runtime] section of the /etc/nvidiacontainer-runtime/config.toml file. Then fire up the container, and you should fine log messages about all of the passthrough mappings in the the log file configured on that line.

For item 2, you should find a .manifest file for the image you built in the image deploy directory. That lists all of the packages installed in the image. You can also trawl through the log files (in particular, the log.do_rootfs file) in the build tree for your image recipe to get more info on the package processing.

[ervgan]

Thanks a lot for your help!