@abbradar See #67791 and #67791 (comment)

@abbradar things in nixpkgs have been using addOpenGLRunpath to patch the runpath on executables and shared libraries, e.g. I submitted a PR for fix CUDA in mpv: #67790

@ivan This is similar to OpenGL drivers before libglvnd became a thing, we used LD_LIBRARY_PATH instead of patching all OpenGL software.

I'm not sure if we want to start using addOpenGLRunpath now but that's a (different) architectural decision. Personally I think LD_LIBRARY_PATH is less error-prone as it's very simple to accidentally link to nvidia_x11 instead of using addOpenGLRunpath in a package, and this is wrong and breaks when packages come from different version of nixpkgs.

This was done in 370d3af for #11434, but I don't fully understand the rationale.

"Currently Mesa and NVidia drivers don't set setLdLibraryPath because they work with libglvnd and do not override libraries" - I believe this is referring to GL calls going through libglvnd and thus not needing LD_LIBRARY_PATH for just GL functionality.

I believe #11434 (comment) references some specific problem that this change solved ("any program that links to one of these libraries will load the one from the AMD drivers. This can result in horrible breakage") but I don't know the details, maybe @ambrop72 can clarify?

The idea is that now for dispatch libraries we either:

• use vendor configuration files which specify full paths to libraries;
• use setOpenGLRunpath for edge cases (no configuration file, gets library name via GLX etc.).

That means that for vast majority of cases LD_LIBRARY_PATH is not needed anymore because we localized all library search to those dispatch libraries. The problems, however, start when there's no dispatch library for something (i.e. CUDA) - cudatoolkit is only one point of entry into it. IMO for these cases we should fall back to the old approach, which is LD_LIBRARY_PATH - the world of library end-users is potentially immense and checking and patching all these packages would be much bigger task than fixing just the dispatchers.

Regarding the comment - LD_LIBRARY_PATH can be harmful when the driver itself is broken (in this example API drivers) and provides replacements for system libraries. This is out of scope - NVIDIA drivers are clear in that regard.

@ambrop72 I see now that you already patched some packages to find CUDA, what do you think of my suggestion instead?

My argument is that it's less error-prone that way (especially if application links to libcuda directly, in which case it's very difficult to notice that it's broken), makes CUDA applications easier to package and that our old policy was to avoid patching all the world, opting for LD_LIBRARY_PATH when we cannot fix this by controlling only dispatcher libraries.

I believe #11434 (comment) references some specific problem that this change solved ("any program that links to one of these libraries will load the one from the AMD drivers. This can result in horrible breakage") but I don't know the details, maybe @ambrop72 can clarify?

Whenever a driver ships with their own version of some library, e.g. llvm, which uses the same sonames as system libraries, if an application both links to a nixpkgs driver and uses the driver (possibly very indirectly), only one of the libraries will end up used, one thing will inevitably use the wrong version. This cannot be solved due to the design of Linux dynamic linking. However, if those driver-specific libraries end up in LD_LIBRARY_PATH, then every program that links to such a conflicting library is affected, not just programs that actually use the driver.

I understand the arguments about breaking things, but I think the simple solution of sticking with LD_LIBRARY_PATH is just too invasive, as it has potential to break every program, including those that do not use, or affect startup time. Therefore I still think this (or #67791) should go to 19.09 only and we fix the breakages on master. By the way could we have a meta bug to keep track of such breakages? I can assist fixing them.

Also, if we keep the LD_LIBRARY_PATH for 19.09 and backport addOpenGLRunpath fixes to 19.09, then the "older applications not working without LD_LIBRARY_PATH" problem will not be after the upgrade to the next release, where LD_LIBRARY_PATH finally disappears, assuming one is not using applications older than 19.09.

Regarding the comment - LD_LIBRARY_PATH can be harmful when the driver itself is broken (in this example API drivers) and provides replacements for system libraries. This is out of scope - NVIDIA drivers are clear in that regard.

Sorry I missed this, yes the NVidia drivers seem fine. But I still wouldn't want to undo the work that went into getting rid of LD_LIBRARY_PATH because of one case (CUDA) and which only affects a fraction of people.

I think the sledgehammer approach is inconsistent with Nix/NixOS philosophy which is focused on purity. Of course we can't be pure here, so the current solution for such problems is to fix individual packages to look in impure places (/run/opengl-runpath). But doing something that affects everything, whether it needs to or not, is the complete opposite of the ideal.

By the way I filed a PR to fix Houdini (#67958). Generally I will be glad to fix other programs that I am told about.

My problem is specifically with CUDA: addOpenGLRunpath is a burden to use for all CUDA applications

Please note that for big applications, setting LD_LIBRARY_PATH in a wrapper (which in many cases already exists) is a simpler approach than patchelf.

I suggest that this is merged only to 19.09 and without reverting the CUDA fixes. This would fix the immediate 19.09 issues and leave the most freedom to decide how to proceed - by still having the fixes in 19.09, if the LD_LIBRARY_PATH is not reintroduced in master and therefore is not present in the next release after 19.09, applications in user environments will not break after an upgrade doe to disappearance of LD_LIBRARY_PATH.

I'm still not sure we should do this by patching packages but let's try and see what problems arise. The package I was trying to fix now was tensorflow, also tensorflow-bin has invalid RUNPATH (nvidia-x11) and all other sorts of ML packages are broken now - we need to fix them.

I don't think we should merge this for 19.09, rather either merge to post-19.09 unstable or reject. For 19.09 we'd rather leave LD_LIBRARY_PATH always enabled for the time being, there is #67791 for that.

I don't think we should merge this for 19.09, rather either merge to post-19.09 unstable or reject. For 19.09 we'd rather leave LD_LIBRARY_PATH always enabled for the time being, there is #67791 for that.

Yeah that's better because it prevents breakage of packages from user environments after upgrade to 19.09 in cases of using open-source drivers.

I fixed this bug for tensorflow at least. Let's close this for now.