libglvnd, ocl-icd, vulkan-loader: Add /run/opengl-driver(-32) to RUNPATH. #60985

ambrop72 · 2019-05-05T12:15:16Z

Previously we were relying on LD_LIBRARY_PATH to discover vendor driver libraries (libGL, ligGLX, libEGL, OpenCL and Vulkan). This has the problem that setuid programs (in particular VirtualBox) ignore LD_LIBRARY_PATH. Fix it by setting RUNPATH in various dispatch libraries to look in /run/opengl-driver(-32).

Setting this on the VirtualBoxVM binary (which would be simpler) would not solve the problem, the RUNPATH must be on the library which performs the dlopen(). The dlopen man page claims that the RUNPATH from the executable is used but that is incorrect, only RUNPATH from the module performing dlopen() is used.

Note that setting RPATH on the executable would work, but patchelf manipulates RUNPATH and not RPATH (RPATH is a legacy feature).

Fixes #22760
Fixes #18457.

Motivation for this change

3D acceleration in VirtualBox does not work with NVidia drivers when hardening is enabled for VirtualBox (the default) (#22760)

Things done

Tested this on 19.03, with this change there is no error when starting a VM with 3D acceleration enabled and the VM log shows that OpenGL was initialized successfully.

flokli · 2019-05-05T13:46:06Z

does this mean we can drop https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/hardware/opengl.nix#L148-L150 too?

ambrop72 · 2019-05-05T14:13:31Z

I think so, hope there isn't something else that uses it.

flokli · 2019-05-05T14:41:15Z

@abbradar How does that sound to you?

ambrop72 · 2019-05-06T16:22:07Z

Actually I think LD_LIBRARY_PATH may still be needed for libcuda, which does not have a dispatch library (AFAIK). So for now I am hesitant to remove it. For the future we could think up some solution maybe.

ambrop72 · 2019-05-06T16:29:17Z

For CUDA I suppose we could apply the same RUNPATH correction to any code that dlopen's libcuda, but I am not at all familiar with CUDA things. Do apps normally dlopen libcuda directly, or are they linked to it, or is there a standard helper library that takes care of this?

ambrop72 · 2019-05-06T17:07:27Z

Forgot about VDPAU though, where we have libvdpau as the dispatch library. From what I can tell, it does not need LD_LIBRARY_PATH, because we configure it with --with-module-dir=${libGL_driver.driverLink}/lib/vdpau and the code tries to dlopen <module_dir>/lib/vdpau/libvdpau_<vendor>.sh. So it should keep working if we get rid of LD_LIBRARY_PATH.

I will remove that LD_LIBRARY_PATH on my system and see if things break.

ambrop72 · 2019-05-07T18:40:15Z

Added patching of some CUDA things.

With these patches, I have then removed that LD_LIBRARY_PATH settings from opengl.nix and do not see any breakage. KDE Plasma, Chromium, Blender, and Steam+games still work fine on my system with NVIDIA driver.

I suggest we apply the changes in this PR and also remove that LD_LIBRARY_PATH, then we can fix any breakages that people report by patching RUNPATH the same way. I would be really great to get rid of that LD_LIBRARY_PATH (it's a hack and it slows down program startup).

flokli · 2019-05-17T23:02:44Z

I gave this a test run on my system (19.03 with these two commits cherry-picked). Also removed adding to LD_LIBRARY_PATH from opengl.nix - no breakages observed - inclined to merge this :-)

@ambrop72, can you add the commit removing from LD_LIBRARY_PATH (plus some changelog entry) to this PR?

It should be safe to backport (only) 73f34cb to 19.03, too. cc @samueldr @lheckemann

ambrop72 · 2019-05-20T17:39:57Z

Sorry forgot about this. Well now I know that maintainers can push to the PR branch by default :)
I agree to backport the first commit only, to fix the VirtualBox problem.

abbradar

Awesome idea! Regarding complete drop of LD_LIBRARY_PATH: I'm not sure now if that would break anything else; I'll try to remember what I encountered when I tried to remove it long time ago but don't count on this much. Bottom line is that there were some problems but maybe they are not relevant anymore.

pkgs/applications/misc/blender/default.nix

abbradar · 2019-05-20T21:13:14Z

Nah, my idea is to make it non-propagating and only add it to libglvnd and friends. I just feel we'll have more derivations we'd need to fix so we want to factor out the code. EDIT: Sorry, I thought I'd answer in thread if I reply by mail :D

…

On May 20, 2019 11:50:22 PM GMT+03:00, Ambroz Bizjak ***@***.***> wrote: ambrop72 commented on this pull request. > @@ -80,6 +81,16 @@ stdenv.mkDerivation rec { --prefix PYTHONPATH : ${pythonPackages.numpy}/${python.sitePackages} ''; + # Set RUNPATH so that libcuda and libnvrtc in /run/opengl-driver(-32)/lib can be Is it really a good idea to add that RPATH entry to all binaries (once you include the hook)? I remember being bothered by some other hooks, like rpath stripping, not being configurable in that respect. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #60985 (comment)

-- Nikolay.

flokli · 2019-05-21T10:08:13Z

@abbradar I'm fine with dropping the commit removing LD_LIBRARY_PATH for now, to not break too much at once :-)

Do you want to give the RUNPATH hook a try?

abbradar · 2019-05-21T10:51:48Z

pkgs/applications/misc/blender/default.nix

+    for program in $out/bin/blender $out/bin/.blender-wrapped; do
+      isELF "$program" || continue
+      origRpath=$(patchelf --print-rpath "$program")
+      patchelf --set-rpath "$origRpath:${libGL_driver.driverLink}/lib" "$program"


Maybe we want to put driverLink path first instead? This way if a package depends on Mesa or nvidia drivers directly it would still load system-wide drivers, not the ones it was linked with -- that's the behavior we arguably expect. It's especially important with nvidia because its driver libraries require exactly the same kernel module version and provide stable ABI.

There is only one libGL that can be linked to, that is the one from libglvnd. Neither the mesa_drivers nor nvidia-x11 provide a libGL that can be linked to, only libGL_vendor that is dlopen-ed from libglvnd. I do not see any case where this RPATH adjustment would need to override some default library. Adding the RPATH at the end should reduce the performance impact to normal library loading.

abbradar · 2019-05-21T11:14:31Z

pkgs/development/libraries/vulkan-loader/default.nix

@@ -17,7 +17,7 @@ stdenv.mkDerivation rec {
    sha256 = "0zhrwj1gi90x2w8gaaaw5h4b969a8gfy244kn0drrplhhb1nqz3b";
  };

-  nativeBuildInputs = [ pkgconfig ];
+  nativeBuildInputs = [ pkgconfig patchelf ];


patchelf is included in stdenv for Linux by default so no need to add it here.

abbradar · 2019-05-21T11:21:31Z

@flokli I implemented a setup hook and added it everywhere. I only manually tested libglvnd for now.

abbradar · 2019-05-21T11:32:46Z

@flokli Regarding LD_LIBRARY_PATH: I'm very much in favor of dropping it but have a gut feeling it would break something ;) I'll run a rebuild with this on my machine later. If it works okay too we could merge and fix breakages as they appear. @vcunat maybe you remember any problems w.r.t. replacing LD_LIBRARY_PATH with RUNPATH? We discussed LD_LIBRARY_PATH and OpenGL before.

abbradar · 2019-05-21T15:22:59Z

I didn't mean libGL, rather any drivers themselves. Nvidia has this problem as I mentioned if one has nvidia_x11 in their RUNPATH for some reason, but also mesa_drivers. I think there were several cases where packages were linked to mesa_drivers or nvidia_x11 but I'll recheck this.

…

On May 21, 2019 6:17:54 PM GMT+03:00, Ambroz Bizjak ***@***.***> wrote: ambrop72 commented on this pull request. > @@ -80,6 +81,16 @@ stdenv.mkDerivation rec { --prefix PYTHONPATH : ${pythonPackages.numpy}/${python.sitePackages} ''; + # Set RUNPATH so that libcuda and libnvrtc in /run/opengl-driver(-32)/lib can be + # found. See the explanation in libglvnd. + postFixup = optionalString cudaSupport '' + for program in $out/bin/blender $out/bin/.blender-wrapped; do + isELF "$program" || continue + origRpath=$(patchelf --print-rpath "$program") + patchelf --set-rpath "$origRpath:${libGL_driver.driverLink}/lib" "$program" There is only one libGL that can be linked to, that is the one from `libglvnd`. Neither the `mesa_drivers` nor `nvidia-x11` provide a libGL that can be linked to, only `libGL_vendor` that is dlopen-ed from `libglvnd`. I do not see any case where this RPATH adjustment would need to override some default library. Adding the RPATH at the end should reduce the performance impact to normal library loading. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #60985 (comment)

-- Nikolay.

ambrop72 · 2019-05-21T15:25:51Z

I didn't mean libGL, rather any drivers themselves. Nvidia has this problem as I mentioned if one has nvidia_x11 in their RUNPATH for some reason, but also mesa_drivers. I think there were several cases where packages were linked to mesa_drivers or nvidia_x11 but I'll recheck this.

Well I didn't see any issue with appending RUNPATH. For libglvnd and other dispatch libraries the only thing that matters is that dlopen will find libraries like libGL_vendor and there is only going to be one package that has those, due to the name.

abbradar · 2019-05-21T15:29:19Z

Issues begin when one tries to run a nix-env installed package with a different version of Nvidia or mesa_drivers than current system. Surely this is a flaky setting but we don't want to break it more than necessary. The same problem can happen if one has e.g. Nvidia beta drivers installed and runs a package which has your usual nvidia_x11 in its RUNPATH.

…

On May 21, 2019 6:26:03 PM GMT+03:00, Ambroz Bizjak ***@***.***> wrote: > I didn't mean libGL, rather any drivers themselves. Nvidia has this problem as I mentioned if one has nvidia_x11 in their RUNPATH for some reason, but also mesa_drivers. I think there were several cases where packages were linked to mesa_drivers or nvidia_x11 but I'll recheck this. Well I didn't see any issue with appending RUNPATH. For libglvnd and other dispatch libraries the only thing that matters is that `dlopen` will find libraries like `libGL_vendor` and there is only going to be one package that has those, due to the name. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #60985 (comment)

-- Nikolay.

ambrop72 · 2019-05-21T15:42:49Z

Yeah, I'll build and check a few things as well.

ambrop72 · 2019-05-21T16:27:46Z

Tested that all packages with these R(UN)PATH adjustments build and get the correct R(UN)PATH (using readelf -d): libglvnd, ocl-icd, vulkan-loader, cudatoolkit, cudnn, nccl, blender(cudaSupport=true). Also cudatoolkit correctly gets RPATH instead of RUNPATH.

I will now rebase locally on top of channels/nixos-unstable and rebuild and test with my system (including the LD_LIBRARY_PATH removal).

ambrop72 · 2019-05-22T07:28:18Z

While testing this I got hit by SDDM breakage in nixos-unstable (not caused by this, happens also on vanilla nixos-unstable, #61840). So I used LightDM instead, but with LightDM one still gets something in LD_LIBRARY_PATH (see comment here).

I will try removing that xserver LD_LIBRARY_PATH and fixing resulting breakage, but cannot do it for all drivers. But I don't think we should delay merging this (minus the LD_LIBRARY_PATH removal which can be done layer once more testing was done without it).

flokli · 2019-05-22T08:30:57Z

@ambrop72 agreed. Could you squash things together a bit and drop 2f056af?

This hook allows to add NixOS driver libraries path to given ELF objects' RUNPATH. We use it instead of settings RUNPATH manually everywhere. It must be invoked in postFixup so that RUNPATH stripping does not remove the path. It puts the path first instead of last so that system-wide drivers are always preferred.

Previously we were relying on LD_LIBRARY_PATH to discover driver libraries (libGL, ligGLX, libEGL, OpenCL and Vulkan). This has the problem that setuid programs (in particular VirtualBox) ignore LD_LIBRARY_PATH. Fix it by setting RUNPATH in various dispatch libraries. This is not needed for libvdpau because it is already configured to look for libraries in the driver paths. Fixes NixOS#22760.

This is to avoid relying on LD_LIBRARY_PATH for finding the CUDA driver libraries.

ambrop72 · 2019-05-22T16:37:51Z

Rearranged commits, dropped the LD_LIBRARY_PATH removal. I think this is ready.

flokli · 2019-05-22T21:35:56Z

Thanks!

matthewbauer · 2019-05-23T03:20:55Z

pkgs/build-support/add-opengl-runpath/setup-hook.sh

+# actually sets RUNPATH not RPATH, which applies only to dependencies of the binary
+# it set on (including for dlopen), so the RUNPATH must indeed be set on these
+# libraries and would not work if set only on executables.
+addOpenGLRunpath() {


This should be a no-op on macOS. We can link to the OpenGL framework there I think.

This should be a no-op, or we should link to the OpenGL framework?

nixos-discourse · 2019-05-28T19:40:49Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/software-rendering-using-mesa-not-possible/3023/8

arianvp · 2019-06-04T10:53:11Z

pkgs/development/libraries/vulkan-loader/default.nix

+  # Set RUNPATH so that driver libraries in /run/opengl-driver(-32)/lib can be found.
+  # See the explanation in addOpenGLRunpath.
+  postFixup = ''
+    addOpenGLRunpath $out/lib/libvulkan.so


This should not be needed.

Vulkan Loader doesn't need the openGLRunpath hardcoded because it will look up libraries not in the opengl path but in whatever the ICD json points to. So vulkan will never look in /run/opengl-driver/lib,
it will instead look in /nix/store/y99l7729c0157a52y9qfzi3zg4f94b57-mesa-noglu-18.3.4-drivers/lib/libvulkan_intel.so because the ICD in /run/opengl-driver/share/vulkan/icd.d says so

cat /run/opengl-driver/share/vulkan/icd.d/intel_icd.x86_64.json { "ICD": { "api_version": "1.1.90", "library_path": "/nix/store/y99l7729c0157a52y9qfzi3zg4f94b57-mesa-noglu-18.3.4-drivers/lib/libvulkan_intel.so" }, "file_format_version": "1.0.0" }

It is currently needed for NVidia because the library_path is only a name:

$ cat /run/opengl-driver-32/share/vulkan/icd.d/nvidia.json { "file_format_version" : "1.0.0", "ICD": { "library_path": "libGLX_nvidia.so", "api_version" : "1.1.99" } }

I think we should patch it and then remove the runpath.

I agree. any other drivers that would need similar fixups? ;-)

Opened #62870 to set absolute paths in NVidia ICD files.
Also related:

nvidia-x11: Set 32-bit library paths for 32-bit libraries. #62859: Fix that will enable actually dropping RUNPATH in dispatch libraries and LD_LIBRARY_PATH in opengl.nix.

vulkan-loader: Always include /run/opengl-driver(-32)/share in search path. #62869: Will enable also dropping XDG_DATA_DIRS in opengl.nix.

nixos-discourse · 2019-11-01T12:24:54Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/building-with-cuda-on-nixpkgs/4462/8

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 501-1000 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels May 5, 2019

flokli requested review from abbradar and Mic92 May 5, 2019 14:41

FRidh mentioned this pull request May 7, 2019

document design behind /run/opengl-driver #6802

Open

ofborg bot requested a review from cillianderoiste May 7, 2019 19:08

ofborg bot added 10.rebuild-darwin: 1001-2500 and removed 10.rebuild-darwin: 501-1000 labels May 7, 2019

This was referenced May 20, 2019

drop LD_LIBRARY_PATH for opengl programs #11434

Closed

VirtualBox nixos guest missing 3D acceleration #18457

Closed

ofborg bot added 6.topic: nixos 8.has: module (update) labels May 20, 2019

abbradar reviewed May 20, 2019

View reviewed changes

pkgs/applications/misc/blender/default.nix Show resolved Hide resolved

abbradar reviewed May 21, 2019

View reviewed changes

ambrop72 force-pushed the virtualbox-nvidia-accel-fix branch from 0ef193d to 547c6b6 Compare May 21, 2019 16:17

ofborg bot added the 8.has: package (new) label May 21, 2019

ambrop72 mentioned this pull request May 21, 2019

Update cgminer #25039

Closed

abbradar and others added 3 commits May 22, 2019 18:19

Add driver library path to some packages to find CUDA libraries.

28a0918

This is to avoid relying on LD_LIBRARY_PATH for finding the CUDA driver libraries.

ambrop72 force-pushed the virtualbox-nvidia-accel-fix branch from 547c6b6 to 28a0918 Compare May 22, 2019 16:36

ofborg bot removed 6.topic: nixos 8.has: module (update) labels May 22, 2019

flokli merged commit 2ed6903 into NixOS:master May 22, 2019

marsam mentioned this pull request May 23, 2019

libglvnd broken on darwin #61921

Closed

matthewbauer reviewed May 23, 2019

View reviewed changes

matthewbauer mentioned this pull request May 28, 2019

Support libGL in non-NixOS Nixpkgs #62169

Open

ambrop72 mentioned this pull request May 29, 2019

opengl: provide swrast as fallback when no GL drivers are found #62177

Closed

10 tasks

arianvp reviewed Jun 4, 2019

View reviewed changes

ambrop72 mentioned this pull request Jun 8, 2019

nvidia-x11: Put absolute library paths into ICD config files. #62870

Merged

10 tasks

ambrop72 mentioned this pull request Jul 16, 2019

libglvnd, ocl-icd, vulkan-loader: Remove addOpenGLRunpath where not needed. #64945

Merged

10 tasks

flokli mentioned this pull request Aug 10, 2019

VirtualBox: Virtual machines fail to start. #65564

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libglvnd, ocl-icd, vulkan-loader: Add /run/opengl-driver(-32) to RUNPATH. #60985

libglvnd, ocl-icd, vulkan-loader: Add /run/opengl-driver(-32) to RUNPATH. #60985

ambrop72 commented May 5, 2019 •

edited by flokli

flokli commented May 5, 2019

ambrop72 commented May 5, 2019

flokli commented May 5, 2019

ambrop72 commented May 6, 2019 •

edited

ambrop72 commented May 6, 2019

ambrop72 commented May 6, 2019

ambrop72 commented May 7, 2019

flokli commented May 17, 2019

ambrop72 commented May 20, 2019

abbradar left a comment

abbradar commented May 20, 2019 via email •

edited

flokli commented May 21, 2019

abbradar May 21, 2019

ambrop72 May 21, 2019

abbradar May 21, 2019

abbradar commented May 21, 2019

abbradar commented May 21, 2019

abbradar commented May 21, 2019 via email

ambrop72 commented May 21, 2019

abbradar commented May 21, 2019 via email

ambrop72 commented May 21, 2019

ambrop72 commented May 21, 2019

ambrop72 commented May 22, 2019

flokli commented May 22, 2019

ambrop72 commented May 22, 2019

flokli commented May 22, 2019

matthewbauer May 23, 2019 •

edited

flokli May 23, 2019

nixos-discourse commented May 28, 2019

arianvp Jun 4, 2019

ambrop72 Jun 4, 2019

flokli Jun 5, 2019

ambrop72 Jun 8, 2019

nixos-discourse commented Nov 1, 2019

libglvnd, ocl-icd, vulkan-loader: Add /run/opengl-driver(-32) to RUNPATH. #60985

libglvnd, ocl-icd, vulkan-loader: Add /run/opengl-driver(-32) to RUNPATH. #60985

Conversation

ambrop72 commented May 5, 2019 • edited by flokli

Motivation for this change

Things done

flokli commented May 5, 2019

ambrop72 commented May 5, 2019

flokli commented May 5, 2019

ambrop72 commented May 6, 2019 • edited

ambrop72 commented May 6, 2019

ambrop72 commented May 6, 2019

ambrop72 commented May 7, 2019

flokli commented May 17, 2019

ambrop72 commented May 20, 2019

abbradar left a comment

Choose a reason for hiding this comment

abbradar commented May 20, 2019 via email • edited

flokli commented May 21, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abbradar commented May 21, 2019

abbradar commented May 21, 2019

abbradar commented May 21, 2019 via email

ambrop72 commented May 21, 2019

abbradar commented May 21, 2019 via email

ambrop72 commented May 21, 2019

ambrop72 commented May 21, 2019

ambrop72 commented May 22, 2019

flokli commented May 22, 2019

ambrop72 commented May 22, 2019

flokli commented May 22, 2019

matthewbauer May 23, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixos-discourse commented May 28, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixos-discourse commented Nov 1, 2019

ambrop72 commented May 5, 2019 •

edited by flokli

ambrop72 commented May 6, 2019 •

edited

abbradar commented May 20, 2019 via email •

edited

matthewbauer May 23, 2019 •

edited