Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cudaPackages.cuda_compat: automatically discover libnvrm* on jetsons #273389

Closed
wants to merge 1 commit into from
Closed

cudaPackages.cuda_compat: automatically discover libnvrm* on jetsons #273389

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 43 additions & 4 deletions pkgs/development/cuda-modules/cuda/overrides.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
{cudaVersion, lib}:
{
cudaVersion,
lib,
stdenv,
}:
let
inherit (lib) attrsets lists strings;
# cudaVersionOlder : Version -> Boolean
Expand Down Expand Up @@ -43,9 +47,44 @@ attrsets.filterAttrs (attr: _: (builtins.hasAttr attr prev)) {
);

cuda_compat = prev.cuda_compat.overrideAttrs (
prevAttrs: {
env.autoPatchelfIgnoreMissingDeps =
prevAttrs.env.autoPatchelfIgnoreMissingDeps + " libnvrm_gpu.so libnvrm_mem.so libnvdla_runtime.so";
prevAttrs: rec {
env.autoPatchelfIgnoreMissingDeps = builtins.concatStringsSep " " (
[
prevAttrs.env.autoPatchelfIgnoreMissingDeps
"libnvrm_gpu.so"
"libnvrm_mem.so"
"libnvdla_runtime.so"
]
++ libcudaExtraNeeded
);

# Append the jetpack's libnvrm_{gpu,mem}.so impure location with the lowest priority.
# This way we trade safety for the out-of-the box experience on jetsons.
appendRunpaths = [
"${stdenv.cc.cc.lib}/lib"
] ++ prevAttrs.appendRunpaths ++ ["/usr/lib/aarch64-linux-gnu/tegra/"];

libcudaExtraNeeded = [
"libnvos.so"
"libnvsocsys.so"
"libnvrm_sync.so"
"libnvos.so"
"libnvsciipc.so"
"libnvsocsys.so"
"libnvrm_chip.so"
"libnvrm_host1x.so"
"libstdc++.so"
];
Comment on lines +67 to +77
Copy link
Contributor Author

@SomeoneSerge SomeoneSerge Dec 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quoting the message from matrix:

I'm wondering if we could somehow move this to nixglhost (e.g. as LD_PRELOAD) or maybe we could write and link a library that would scan /usr/lib/aarch-blahblaah/tegra and dlopen stuff in the correct order.

...presumably, this list might change at any time, i.e. our current nixpkgs revision might not be compatible with the future jetpacks


# Remove once autoPatchelfHook supports __structuredAttrs
preFixup = ''
appendRunpaths="''${appendRunpaths[@]}"

for needed in "''${libcudaExtraNeeded[@]}" ; do
patchelf "$out/compat/libcuda.so" --add-needed "$needed"
done
'';

# `cuda_compat` only works on aarch64-linux, and only when building for Jetson devices.
brokenConditions = prevAttrs.brokenConditions // {
"Trying to use cuda_compat on aarch64-linux targeting non-Jetson devices" =
Expand Down