Update Vulkan, SPIR-V, glslang and shaderc #94049

CajuM · 2020-07-28T01:52:10Z

Motivation for this change

Keeping master up to date with recent developments in upstream projects

Things done

Ralith

Looks good, thanks! Note that this subsumes #93604.

jonringer · 2020-08-02T02:54:54Z

closes #94247

jonringer · 2020-08-03T16:29:17Z

the shaderc bump is causing some regressions, I now get:

builder for '/nix/store/86lrf5m4vy9r4hbwm6r95ljv6w0pwi4d-libplacebo-2.72.0.drv' failed with exit code 1; last 10 log lines:
  build flags: -j128 -l128
  [42/44] Compiling C++ object 'src/25a6634@@placebo@sha/glsl_glslang.cc.o's_gen.c.o'
  FAILED: src/25a6634@@placebo@sha/glsl_glslang.cc.o
  g++ -Isrc/25a6634@@placebo@sha -Isrc -I../src -I../src/include -I../src/include/dummy -I../subprojects/xtalloc/include -I../subprojects/bstr/include -I/nix/store/81iy20n5rhvcm18ha289anygpgq5qyjc-lcms2-2.11-dev/include -I/nix/store/riw48ysh0rhkf2fck7m1iiany4rif2mv-vulkan-headers-1.2.141.0/include -I/nix/store/psda85jw38cxzqp6fa59vrdipz67h6bk-epoxy-1.5.4-dev/include -I/usr/include/glslang -I/nix/store/my4bdsgi308bq216gkhfwnwnk35yhszn-libplacebo-2.72.0/include/glslang -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -std=c++11 -Wundef -Wshadow -Wparentheses -Wpointer-arith -fvisibility=hidden -fPIC -pthread -MD -MQ 'src/25a6634@@placebo@sha/glsl_glslang.cc.o' -MF 'src/25a6634@@placebo@sha/glsl_glslang.cc.o.d' -o 'src/25a6634@@placebo@sha/glsl_glslang.cc.o' -c ../src/glsl/glslang.cc
  ../src/glsl/glslang.cc:229:1: error: cannot convert '<brace-enclosed initializer list>' to 'int' in initialization
    229 | };
        | ^

jonringer · 2020-08-03T16:41:09Z

I created an upstream issue https://code.videolan.org/videolan/libplacebo/-/issues/108

CajuM · 2020-08-03T17:09:43Z

I see two solutions:

we drop the https://code.videolan.org/videolan/libplacebo/-/commit/82e3be1839379791b58e98eb049415b42888d5b0.patch from libplacebo
we update shaderc or glslang to newer versions: the header which is causing us trouble is in glslang, not shaderc. But for some reason shaderc: 2019.1 -> 2020.1 #93604 broke on shaderc, maybe because libplacebo got the header from shaderc's glslang dependency?

CajuM · 2020-08-03T17:13:54Z

I have also noticed that the shaderc-dev derivation has bundled the glslang headers, shouldn't they be in a separate derivation?
Also, do we have a requirement for overriding dependencies for glslang and vulkan-validation-layers? When I made this PR I just copied this pattern. But now I'm beginning to wonder if we can just use the same versions everywhere.

jonringer · 2020-08-03T19:16:23Z

@CajuM shaderc is re-exporting all of the glslang headers it was built with. Easiest solution I found is just make shaderc the last dependency in buildInputs for libplacebo

$ git diff pkgs/
diff --git a/pkgs/development/libraries/libplacebo/default.nix b/pkgs/development/libraries/libplacebo/default.nix
index dc3e5e30c74..15b52c400df 100644
--- a/pkgs/development/libraries/libplacebo/default.nix
+++ b/pkgs/development/libraries/libplacebo/default.nix
@@ -47,10 +47,10 @@ stdenv.mkDerivation rec {
   buildInputs = [
     vulkan-headers
     vulkan-loader
-    shaderc
     glslang
     lcms2
     epoxy
+    shaderc
   ];

   mesonFlags = [

jonringer · 2020-08-03T19:19:10Z

Not sure if this is intended behavior, and probably exists in the current package:

/nix/store/5prb4s19gv9l4xm71zwkad8yh6x5rz4v-shaderc-2020.2-dev/include/glslang/
├── build_info.h
├── HLSL
│   ├── hlslAttributes.h
│   ├── hlslGrammar.h
│   ├── hlslOpMap.h
│   ├── hlslParseables.h
│   ├── hlslParseHelper.h
│   ├── hlslScanContext.h
│   ├── hlslTokens.h
│   └── hlslTokenStream.h
├── Include
│   ├── arrays.h
│   ├── BaseTypes.h
│   ├── Common.h
│   ├── ConstantUnion.h
│   ├── glslang_c_interface.h
│   ├── glslang_c_shader_types.h
│   ├── InfoSink.h
│   ├── InitializeGlobals.h
│   ├── intermediate.h
│   ├── PoolAlloc.h
│   ├── ResourceLimits.h
│   ├── ShHandle.h
│   └── Types.h
├── MachineIndependent
│   ├── attribute.h
│   ├── glslang_tab.cpp.h
│   ├── gl_types.h
│   ├── Initialize.h
│   ├── iomapper.h
│   ├── LiveTraverser.h
│   ├── localintermediate.h
│   ├── ParseHelper.h
│   ├── parseVersions.h
│   ├── preprocessor
│   │   ├── PpContext.h
│   │   └── PpTokens.h
│   ├── propagateNoContraction.h
│   ├── reflection.h
│   ├── RemoveTree.h
│   ├── ScanContext.h
│   ├── Scan.h
│   ├── SymbolTable.h
│   └── Versions.h
├── Public
│   └── ShaderLang.h
└── SPIRV
    ├── bitutils.h
    ├── disassemble.h
    ├── doc.h
    ├── GlslangToSpv.h
    ├── GLSL.ext.AMD.h
    ├── GLSL.ext.EXT.h
    ├── GLSL.ext.KHR.h
    ├── GLSL.ext.NV.h
    ├── GLSL.std.450.h
    ├── hex_float.h
    ├── Logger.h
    ├── NonSemanticDebugPrintf.h
    ├── spirv.hpp
    ├── SpvBuilder.h
    ├── spvIR.h
    ├── SPVRemapper.h
    └── SpvTools.h

6 directories, 58 files

Ralith · 2020-08-05T18:05:11Z

The situation here's a bit weird because various vulkan tooling packages all have different hardcoded revision dependencies on glslang, none of which are stable releases. If a package depends on both shaderc and glslang (why?) then it should probably try to use the same version of glslang that shaderc does. Maybe the explicit glslang dep should be removed from libplacebo entirely?

CajuM · 2020-09-03T14:28:21Z

I made one more patch, on top of this PR, but I'm not sure if it's ok because I removed the custom versions for glslang, everything works now. I also updated vulkan tooling to newer versions, except shaderc which requires an unreleased version of glslang. I also had some problems with vkd3d and vkquake, but managed to fix them. Also about glslang, there are several workarounds available made by openSUSE, Fedora and Gentoo. Gentoo patched shaderc to support the older version of glslang. Fedora and openSUSE used unreleased versions in rawhide and factory
https://gist.github.com/CajuM/86b867aa73eb85ab997cca73c8b9416b

pkgs/development/tools/vulkan-validation-layers/default.nix

pkgs/tools/graphics/vulkan-tools/default.nix

doronbehar · 2020-09-24T12:54:57Z

pkgs/top-level/all-packages.nix

@@ -15824,24 +15824,24 @@ in
        src = fetchFromGitHub {


Again, I know you are merely attempting to update a few packages, but I think it ought to be taken care of while at it:

I understand why Google needs overrides as explained here. But could you explain what's with the overrides here? Ideally, these overrides should be in a separate file, and automatically generated via an passthru.updateScript. If that's not possible, it would be nice to get these details out of all-packages.nix.

there's an easier way to do it, the patch I posted on gist removes the custom buildInputs versions all-together. So we'd be using one stable version for each of the updated packages in the entire repo. I didn't push it to this branch because I wasn't sure if it would be accepted. One problem I see is that we'd have to wait a bit to update the whole set of packages due to incompatible versions. Another might be that we'd be introducing regressions by not using the provided dependency versions for glslang and shaderc.

The general issue is that upstream only advertises support for specific revisions, and every package that depends on glslang/spirv-tools seems to have its own unique vetted revision. Using a different revision than the one supported by upstream seems like asking for trouble. No opinion on the proper place/mechanism to encode the information, though; I can't keep up with the latest nixpkgs tricks.

This is what I was thinking of:

diff --git i/pkgs/development/compilers/glslang/default.nix w/pkgs/development/compilers/glslang/default.nix index dee3f488ecc..135eb94dd70 100644 --- i/pkgs/development/compilers/glslang/default.nix +++ w/pkgs/development/compilers/glslang/default.nix @@ -3,9 +3,27 @@ , cmake , jq , python3 -, spirv-headers -, spirv-tools +, spirv-headers' +, spirv-tools' }: +let + spirv-tools = spirv-tools'.overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "SPIRV-Tools"; + rev = "fd8e130510a6b002b28eee5885a9505040a9bdc9"; + sha256 = "00b7xgyrcb2qq63pp3cnw5q1xqx2d9rfn65lai6n6r89s1vh3vg6"; + }; + }); + spirv-headers = spirv-headers'.overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "SPIRV-Headers"; + rev = "f8bf11a0253a32375c32cad92c841237b96696c0"; + sha256 = "1znwjy02dl9rshqzl87rqsv9mfczw7gvwfhcirbl81idahgp4p6l"; + }; + }); +in stdenv.mkDerivation rec { pname = "glslang"; diff --git i/pkgs/development/tools/vulkan-validation-layers/default.nix w/pkgs/development/tools/vulkan-validation-layers/default.nix index 0d3c227291f..0bd9dd0a662 100644 --- i/pkgs/development/tools/vulkan-validation-layers/default.nix +++ w/pkgs/development/tools/vulkan-validation-layers/default.nix @@ -1,6 +1,46 @@ -{ stdenv, fetchFromGitHub, cmake, writeText, python3 -, vulkan-headers, vulkan-loader, glslang, spirv-headers -, pkgconfig, xlibsWrapper, libxcb, libXrandr, wayland }: +{ stdenv +, fetchFromGitHub +, cmake +, writeText +, python3 +, spirv-headers +, spirv-tools +, vulkan-headers +, vulkan-loader +, glslang' +, pkgconfig +, xlibsWrapper +, libxcb +, libXrandr +, wayland +}: +let + glslang = (glslang'.override { + spirv-tools' = spirv-tools.overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "SPIRV-Tools"; + rev = "e128ab0d624ce7beb08eb9656bb260c597a46d0a"; + sha256 = "0jj8zrl3dh9fq71jc8msx3f3ifb2vjcb37nl0w4sa8sdhfff74pv"; + }; + }); + spirv-headers' = spirv-headers.overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "SPIRV-Headers"; + rev = "ac638f1815425403e946d0ab78bac71d2bdbf3be"; + sha256 = "1lkhs7pxcrfkmiizcxl0w5ajx6swwjv7w3iq586ipgh571fc75gx"; + }; + }); + }).overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "glslang"; + rev = "e00d27c6d65b7d3e72506a311d7f053da4051295"; + sha256 = "00lzvzk613gpm1vsdxffmx52z3c52ijwvzk4sfhh95p71kdydhgv"; + }; + }); +in stdenv.mkDerivation rec { pname = "vulkan-validation-layers"; @@ -21,7 +61,7 @@ stdenv.mkDerivation rec { buildInputs = [ glslang - spirv-headers + glslang.spirv-headers vulkan-headers vulkan-loader libxcb diff --git i/pkgs/top-level/all-packages.nix w/pkgs/top-level/all-packages.nix index f4b0ae7dc31..fbf6f625e0c 100644 --- i/pkgs/top-level/all-packages.nix +++ w/pkgs/top-level/all-packages.nix @@ -9095,22 +9095,8 @@ in dotnetPackages = recurseIntoAttrs (callPackage ./dotnet-packages.nix {}); glslang = callPackage ../development/compilers/glslang { - spirv-tools = spirv-tools.overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "SPIRV-Tools"; - rev = "fd8e130510a6b002b28eee5885a9505040a9bdc9"; - sha256 = "00b7xgyrcb2qq63pp3cnw5q1xqx2d9rfn65lai6n6r89s1vh3vg6"; - }; - }); - spirv-headers = spirv-headers.overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "SPIRV-Headers"; - rev = "f8bf11a0253a32375c32cad92c841237b96696c0"; - sha256 = "1znwjy02dl9rshqzl87rqsv9mfczw7gvwfhcirbl81idahgp4p6l"; - }; - }); + spirv-headers' = spirv-headers; + spirv-tools' = spirv-tools; }; go_bootstrap = if stdenv.isAarch64 then @@ -15819,31 +15805,7 @@ in vulkan-loader = callPackage ../development/libraries/vulkan-loader { }; vulkan-tools = callPackage ../tools/graphics/vulkan-tools { }; vulkan-validation-layers = callPackage ../development/tools/vulkan-validation-layers { - glslang = (glslang.override { - spirv-tools = spirv-tools.overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "SPIRV-Tools"; - rev = "e128ab0d624ce7beb08eb9656bb260c597a46d0a"; - sha256 = "0jj8zrl3dh9fq71jc8msx3f3ifb2vjcb37nl0w4sa8sdhfff74pv"; - }; - }); - spirv-headers = spirv-tools.overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "SPIRV-Headers"; - rev = "ac638f1815425403e946d0ab78bac71d2bdbf3be"; - sha256 = "1lkhs7pxcrfkmiizcxl0w5ajx6swwjv7w3iq586ipgh571fc75gx"; - }; - }); - }).overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "glslang"; - rev = "e00d27c6d65b7d3e72506a311d7f053da4051295"; - sha256 = "00lzvzk613gpm1vsdxffmx52z3c52ijwvzk4sfhh95p71kdydhgv"; - }; - }); + glslang' = glslang; }; vtkWithQt5 = vtk.override { qtLib = qt514; };

The diff doesn't change the inputs of glslang at all, from the version of the current PR, but it does change them for vulkan-validation-layers. I think it's unavoidable if the typo is to be fixed.

how about?

diff --git a/pkgs/development/compilers/glslang/default.nix b/pkgs/development/compilers/glslang/default.nix index dee3f488ecc..183d0f4df02 100644 --- a/pkgs/development/compilers/glslang/default.nix +++ b/pkgs/development/compilers/glslang/default.nix @@ -5,7 +5,28 @@ , python3 , spirv-headers , spirv-tools +, argSpirv-tools ? null +, argSpirv-headers ? null }: +let + localSpirv-tools = ({ argSpirv-tools ? spirv-tools.overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "SPIRV-Tools"; + rev = "fd8e130510a6b002b28eee5885a9505040a9bdc9"; + sha256 = "00b7xgyrcb2qq63pp3cnw5q1xqx2d9rfn65lai6n6r89s1vh3vg6"; + }; + }) }: argSpirv-tools) { argSpirv-tools = argSpirv-tools; }; + + localSpirv-headers = ({ argSpirv-headers ? spirv-headers.overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "SPIRV-Headers"; + rev = "f8bf11a0253a32375c32cad92c841237b96696c0"; + sha256 = "1znwjy02dl9rshqzl87rqsv9mfczw7gvwfhcirbl81idahgp4p6l"; + }; + }) }: argSpirv-headers) { argSpirv-headers = argSpirv-headers; }; +in stdenv.mkDerivation rec { pname = "glslang"; @@ -20,22 +41,22 @@ stdenv.mkDerivation rec { # These get set at all-packages, keep onto them for child drvs passthru = { - inherit spirv-tools spirv-headers; + inherit localSpirv-tools localSpirv-headers; }; nativeBuildInputs = [ cmake python3 bison jq ]; enableParallelBuilding = true; postPatch = '' - cp --no-preserve=mode -r "${spirv-tools.src}" External/spirv-tools - ln -s "${spirv-headers.src}" External/spirv-tools/external/spirv-headers + cp --no-preserve=mode -r "${localSpirv-tools.src}" External/spirv-tools + ln -s "${localSpirv-headers.src}" External/spirv-tools/external/spirv-headers ''; # Ensure spirv-headers and spirv-tools match exactly to what is expected preConfigure = '' HEADERS_COMMIT=$(jq -r < known_good.json '.commits|map(select(.name=="spirv-tools/external/spirv-headers"))[0].commit') TOOLS_COMMIT=$(jq -r < known_good.json '.commits|map(select(.name=="spirv-tools"))[0].commit') - if [ "$HEADERS_COMMIT" != "${spirv-headers.src.rev}" ] || [ "$TOOLS_COMMIT" != "${spirv-tools.src.rev}" ]; then + if [ "$HEADERS_COMMIT" != "${localSpirv-headers.src.rev}" ] || [ "$TOOLS_COMMIT" != "${localSpirv-tools.src.rev}" ]; then echo "ERROR: spirv-tools commits do not match expected versions: expected tools at $TOOLS_COMMIT, headers at $HEADERS_COMMIT"; exit 1; fi diff --git a/pkgs/development/tools/vulkan-validation-layers/default.nix b/pkgs/development/tools/vulkan-validation-layers/default.nix index 0d3c227291f..9c8490f7277 100644 --- a/pkgs/development/tools/vulkan-validation-layers/default.nix +++ b/pkgs/development/tools/vulkan-validation-layers/default.nix @@ -1,6 +1,46 @@ -{ stdenv, fetchFromGitHub, cmake, writeText, python3 -, vulkan-headers, vulkan-loader, glslang, spirv-headers -, pkgconfig, xlibsWrapper, libxcb, libXrandr, wayland }: +{ stdenv +, fetchFromGitHub +, cmake +, writeText +, python3 +, spirv-headers +, spirv-tools +, vulkan-headers +, vulkan-loader +, glslang +, pkgconfig +, xlibsWrapper +, libxcb +, libXrandr +, wayland +}: +let + localGlslang = (glslang.override { + argSpirv-tools = spirv-tools.overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "SPIRV-Tools"; + rev = "e128ab0d624ce7beb08eb9656bb260c597a46d0a"; + sha256 = "0jj8zrl3dh9fq71jc8msx3f3ifb2vjcb37nl0w4sa8sdhfff74pv"; + }; + }); + argSpirv-headers = spirv-headers.overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "SPIRV-Headers"; + rev = "ac638f1815425403e946d0ab78bac71d2bdbf3be"; + sha256 = "1lkhs7pxcrfkmiizcxl0w5ajx6swwjv7w3iq586ipgh571fc75gx"; + }; + }); + }).overrideAttrs (_: { + src = fetchFromGitHub { + owner = "KhronosGroup"; + repo = "glslang"; + rev = "e00d27c6d65b7d3e72506a311d7f053da4051295"; + sha256 = "00lzvzk613gpm1vsdxffmx52z3c52ijwvzk4sfhh95p71kdydhgv"; + }; + }); +in stdenv.mkDerivation rec { pname = "vulkan-validation-layers"; @@ -20,8 +60,8 @@ stdenv.mkDerivation rec { ]; buildInputs = [ - glslang - spirv-headers + localGlslang + localGlslang.localSpirv-headers vulkan-headers vulkan-loader libxcb @@ -32,7 +72,7 @@ stdenv.mkDerivation rec { enableParallelBuilding = true; cmakeFlags = [ - "-DGLSLANG_INSTALL_DIR=${glslang}" + "-DGLSLANG_INSTALL_DIR=${localGlslang}" ]; # Help vulkan-loader find the validation layers diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix index f4b0ae7dc31..7ee745abe9f 100644 --- a/pkgs/top-level/all-packages.nix +++ b/pkgs/top-level/all-packages.nix @@ -9094,24 +9094,7 @@ in dotnetPackages = recurseIntoAttrs (callPackage ./dotnet-packages.nix {}); - glslang = callPackage ../development/compilers/glslang { - spirv-tools = spirv-tools.overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "SPIRV-Tools"; - rev = "fd8e130510a6b002b28eee5885a9505040a9bdc9"; - sha256 = "00b7xgyrcb2qq63pp3cnw5q1xqx2d9rfn65lai6n6r89s1vh3vg6"; - }; - }); - spirv-headers = spirv-headers.overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "SPIRV-Headers"; - rev = "f8bf11a0253a32375c32cad92c841237b96696c0"; - sha256 = "1znwjy02dl9rshqzl87rqsv9mfczw7gvwfhcirbl81idahgp4p6l"; - }; - }); - }; + glslang = callPackage ../development/compilers/glslang { }; go_bootstrap = if stdenv.isAarch64 then srcOnly { @@ -15818,33 +15801,7 @@ in vulkan-headers = callPackage ../development/libraries/vulkan-headers { }; vulkan-loader = callPackage ../development/libraries/vulkan-loader { }; vulkan-tools = callPackage ../tools/graphics/vulkan-tools { }; - vulkan-validation-layers = callPackage ../development/tools/vulkan-validation-layers { - glslang = (glslang.override { - spirv-tools = spirv-tools.overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "SPIRV-Tools"; - rev = "e128ab0d624ce7beb08eb9656bb260c597a46d0a"; - sha256 = "0jj8zrl3dh9fq71jc8msx3f3ifb2vjcb37nl0w4sa8sdhfff74pv"; - }; - }); - spirv-headers = spirv-tools.overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "SPIRV-Headers"; - rev = "ac638f1815425403e946d0ab78bac71d2bdbf3be"; - sha256 = "1lkhs7pxcrfkmiizcxl0w5ajx6swwjv7w3iq586ipgh571fc75gx"; - }; - }); - }).overrideAttrs (_: { - src = fetchFromGitHub { - owner = "KhronosGroup"; - repo = "glslang"; - rev = "e00d27c6d65b7d3e72506a311d7f053da4051295"; - sha256 = "00lzvzk613gpm1vsdxffmx52z3c52ijwvzk4sfhh95p71kdydhgv"; - }; - }); - }; + vulkan-validation-layers = callPackage ../development/tools/vulkan-validation-layers { }; vtkWithQt5 = vtk.override { qtLib = qt514; };

@CajuM sorry for the long wait. This diff is just as good. Please commit it, and please mention near every override why it is needed.

pkgs/top-level/all-packages.nix

jonringer · 2020-10-13T23:39:06Z

Please follow CONTRIBUTING.md and manual#submitting-changes-making-patches and squash the fix-up commits.

git rebase -i makes this easy todo. I created a small video demonstrating it's use here. A more indepth text tutorial can be found here

doronbehar

Otherwise looks good to me.

doronbehar · 2020-10-14T19:27:24Z

pkgs/development/compilers/glslang/default.nix

@@ -20,22 +49,22 @@ stdenv.mkDerivation rec {

  # These get set at all-packages, keep onto them for child drvs
  passthru = {
-    inherit spirv-tools spirv-headers;
+    inherit localSpirv-tools localSpirv-headers;


Small nitpick - this is a change in the "API" of it - use spirv-tools = localSpirv-tools, etc.

doronbehar · 2020-10-14T19:28:44Z

pkgs/development/tools/vulkan-validation-layers/default.nix

+# vulkan-validation-layers requires a custom glslang version, while glslang requires
+# custom versions for spirv-tools and spirv-headers
+# The vulkan-validation-layers dependency is taken from:
+# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json
+# While those for glslang are taken from:
+# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json


Suggested change

# vulkan-validation-layers requires a custom glslang version, while glslang requires

# custom versions for spirv-tools and spirv-headers

# The vulkan-validation-layers dependency is taken from:

# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json

# While those for glslang are taken from:

# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json

# vulkan-validation-layers requires a custom glslang version, while glslang requires

# custom versions for spirv-tools and spirv-headers. The git hashes required for all

# of these deps is documented upstream here:

# https://github.com/KhronosGroup/Vulkan-ValidationLayers/blob/master/scripts/known_good.json

doronbehar · 2020-10-14T19:29:35Z

pkgs/development/tools/vulkan-validation-layers/default.nix

-    glslang
-    spirv-headers
+    localGlslang
+    localGlslang.localSpirv-headers


If you change back the API, as I think you should, don't forget this place. Note it shouldn't trigger rebuilds, but please check it all.

SuperSandro2000 · 2020-10-17T08:45:03Z

the shaderc bump is causing some regressions, I now get:

builder for '/nix/store/86lrf5m4vy9r4hbwm6r95ljv6w0pwi4d-libplacebo-2.72.0.drv' failed with exit code 1; last 10 log lines:
  build flags: -j128 -l128
  [42/44] Compiling C++ object 'src/25a6634@@placebo@sha/glsl_glslang.cc.o's_gen.c.o'
  FAILED: src/25a6634@@placebo@sha/glsl_glslang.cc.o
  g++ -Isrc/25a6634@@placebo@sha -Isrc -I../src -I../src/include -I../src/include/dummy -I../subprojects/xtalloc/include -I../subprojects/bstr/include -I/nix/store/81iy20n5rhvcm18ha289anygpgq5qyjc-lcms2-2.11-dev/include -I/nix/store/riw48ysh0rhkf2fck7m1iiany4rif2mv-vulkan-headers-1.2.141.0/include -I/nix/store/psda85jw38cxzqp6fa59vrdipz67h6bk-epoxy-1.5.4-dev/include -I/usr/include/glslang -I/nix/store/my4bdsgi308bq216gkhfwnwnk35yhszn-libplacebo-2.72.0/include/glslang -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -std=c++11 -Wundef -Wshadow -Wparentheses -Wpointer-arith -fvisibility=hidden -fPIC -pthread -MD -MQ 'src/25a6634@@placebo@sha/glsl_glslang.cc.o' -MF 'src/25a6634@@placebo@sha/glsl_glslang.cc.o.d' -o 'src/25a6634@@placebo@sha/glsl_glslang.cc.o' -c ../src/glsl/glslang.cc
  ../src/glsl/glslang.cc:229:1: error: cannot convert '<brace-enclosed initializer list>' to 'int' in initialization
    229 | };
        | ^

I still get this error. Reverting the shaderc bump fixes it for me.

doronbehar · 2020-10-17T09:00:17Z

😬 Then this means vlc and mpv are broken now. I wish all of maintainers of those packages would have used actual releases.. Using latest libplacebo from master doesn't fix this issue.. I'm working on a local override for libplacebo to use the version(s) that were previously used before this PR.

CajuM · 2020-10-17T09:12:03Z

grimacing Then this means vlc and mpv are broken now. I wish all of maintainers of those packages would have used actual releases.. Using latest libplacebo from master doesn't fix this issue.. I'm working on a local override for libplacebo to use the version(s) that were previously used before this PR.

It would seem that removing the https://code.videolan.org/videolan/libplacebo/-/commit/523056828ab86c2f17ea65f432424d48b6fdd389.patch patch fixes things

doronbehar · 2020-10-17T11:06:17Z

All of these packages are a mess (I'm not sure who is to blame). How come Arch Linux doesn't need such overrides? Perhaps they do need and use

Maybe we shouldn't give to much attention to the known_good.json file - just because they have verified these versions work together, doesn't mean that any other combination of versions won't work. For instance Arch Linux even disables shaderc vendoring, somewhat similarly to us, but with the same glslang and spirv-tools versions as the normal packages:

https://github.com/archlinux/svntogit-packages/blob/2900077c8c054d2c9604780d3f189e889271ec6d/trunk/PKGBUILD#L18-L29

Since there are no shaderc maintainers, I'm reluctant into doing this.

SuperSandro2000 · 2020-10-17T12:03:53Z

grimacing Then this means vlc and mpv are broken now. I wish all of maintainers of those packages would have used actual releases.. Using latest libplacebo from master doesn't fix this issue.. I'm working on a local override for libplacebo to use the version(s) that were previously used before this PR.

It would seem that removing the code.videolan.org/videolan/libplacebo/-/commit/523056828ab86c2f17ea65f432424d48b6fdd389.patch patch fixes things

I removed https://code.videolan.org/videolan/libplacebo/-/commit/82e3be1839379791b58e98eb049415b42888d5b0.patch and it builds for me. Reverting shaderc also works #100818. Not sure what the right way is. Open for suggestions.

doronbehar · 2020-10-17T13:15:29Z

It would seem that removing the code.videolan.org/videolan/libplacebo/-/commit/523056828ab86c2f17ea65f432424d48b6fdd389.patch patch fixes things

I removed code.videolan.org/videolan/libplacebo/-/commit/82e3be1839379791b58e98eb049415b42888d5b0.patch and it builds for me.

I don't know what you did, but this sounds like a reasonable idea in comparison to #100818. I get:

  patching file src/glsl/glslang.cc
  Hunk #1 FAILED at 211.
  1 out of 1 hunk FAILED -- saving rejects to file src/glsl/glslang.cc.rej

With this change:

diff --git i/pkgs/development/libraries/libplacebo/default.nix w/pkgs/development/libraries/libplacebo/default.nix
index dc3e5e30c74..91de1f70fa7 100644
--- i/pkgs/development/libraries/libplacebo/default.nix
+++ w/pkgs/development/libraries/libplacebo/default.nix
@@ -27,10 +27,6 @@ stdenv.mkDerivation rec {
 
   patches = [
     # to work with latest glslang, remove on release >2.72.0
-    (fetchpatch {
-      url = "https://code.videolan.org/videolan/libplacebo/-/commit/523056828ab86c2f17ea65f432424d48b6fdd389.patch";
-      sha256 = "051vhd0l3yad1fzn5zayi08kqs9an9j8p7m63kgqyfv1ksnydpcs";
-    })
     (fetchpatch {
       url = "https://code.videolan.org/videolan/libplacebo/-/commit/82e3be1839379791b58e98eb049415b42888d5b0.patch";
       sha256 = "0nklj9gfiwkbbj6wfm1ck33hphaxrvzb84c4h2nfj88bapnlm90l";

CajuM · 2020-10-17T13:18:46Z

My bad, it was 82e3be1839379791b58e98eb049415b42888d5b0.patch

From some reason, if we take this patch out, libplacebo builds fine, and mpv is again usable.

doronbehar · 2020-10-17T13:50:05Z

Taking that patch out makes mpv and vlc build and work. Hence I prefer #100825 over #100818 .

doronbehar · 2020-10-17T13:58:25Z

Out of scope for the issue with libplacebo: All projects of KhronosGroup should not have local overrides, as according to my tests they are compatible with each other, as you'd expect since it's the same group that develops all of these tools - (glslang has local overrides).

To improve the situation of all ugly overrides, I think it'd be best to introduce an attribute set such as KhronosGroupPackages where it'd have it's own callPackage that will insure compatible versions are used in scope.

It should be also worth nagging upstream that their build process is not ideal for downstream package maintainers, considering the fact spirv-tools for instance needs to vendor it's libraries, and is uncapable of using distribution's builds directly.

(cherry picked from commit 0826f2e)

This reverts commit b2c8af3.

acowley · 2021-02-10T20:15:31Z

Does anyone have any new thoughts on how to handle these upgrades? I packaged the tint shader compiler, but it needed newer spirv-tools. I'd be happy to PR tint to nixpkgs, but I don't want to break glslang dependents.

SuperSandro2000 · 2021-02-11T01:01:00Z

We can either add multiple versions of glslang (maybe 11.1 or 8.X?) or you add a local overwrite in your package if only the version changes and nothing else.

doronbehar · 2021-02-15T14:28:33Z

We can either add multiple versions of glslang (maybe 11.1 or 8.X?)

That sounds great to me - to remove all overrides inside packages' expressions, and have as many variants as needed of the same packages all declared in all-packages.nix.

Ralith · 2021-02-15T18:38:58Z

Prominent consumers use distinct, specific, non-release revisions that must be updated in lockstep with the consumers. Moving that into a separate file sounds painful.

ofborg bot requested a review from Ralith July 28, 2020 02:00

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 101-500 labels Jul 28, 2020

Ralith approved these changes Jul 28, 2020

View reviewed changes

Ralith mentioned this pull request Jul 31, 2020

vulkan-headers: 1.2.131.1 -> 1.2.148 #94247

Closed

doronbehar requested changes Sep 11, 2020

View reviewed changes

pkgs/development/tools/vulkan-validation-layers/default.nix Outdated Show resolved Hide resolved

pkgs/tools/graphics/vulkan-tools/default.nix Show resolved Hide resolved

pkgs/tools/graphics/vulkan-tools/default.nix Show resolved Hide resolved

Câju Mihai-Drosi added 8 commits September 24, 2020 08:19

vulkan-headers: 1.2.131.1 -> 1.2.141.0

abc0d91

vulkan-loader: 1.2.131.2 -> 1.2.141.0

4884f33

glslang: 8.13.3559 -> 8.13.3743

6c81540

spirv-headers: 1.5.1 -> 1.5.3

42133cb

spirv-tools: 2019.4 -> 2020.2

9efc9c8

vulkan-validation-layers: 1.2.131.2 -> 1.2.141.0

926152f

shaderc: 2019.1 -> 2020.2

fab03d3

vulkan-tools: 1.2.131.1 -> 1.2.141.0

bda92bf

CajuM force-pushed the vulkan-up branch from fed263d to bda92bf Compare September 24, 2020 05:20

ofborg bot requested a review from Ralith September 24, 2020 05:30

CajuM requested a review from doronbehar September 24, 2020 10:33

doronbehar reviewed Sep 24, 2020

View reviewed changes

doronbehar reviewed Sep 27, 2020

View reviewed changes

pkgs/top-level/all-packages.nix Outdated Show resolved Hide resolved

CajuM force-pushed the vulkan-up branch from abd65b1 to 8734f82 Compare October 14, 2020 12:48

CajuM requested a review from doronbehar October 14, 2020 13:21

doronbehar requested changes Oct 14, 2020

View reviewed changes

Move glslang and vulkan-validation-layers overrides out of all-packages

00946dc

CajuM force-pushed the vulkan-up branch from 8734f82 to 00946dc Compare October 14, 2020 22:02

doronbehar merged commit 0826f2e into NixOS:master Oct 15, 2020

CajuM deleted the vulkan-up branch October 15, 2020 12:20

SuperSandro2000 mentioned this pull request Oct 17, 2020

shaderc: 2019.1 -> 2020.1 #93604

Closed

10 tasks

SuperSandro2000 mentioned this pull request Oct 17, 2020

libplacebo: fix building with shaderc 2020.2 #100818

Merged

10 tasks

doronbehar added a commit to doronbehar/nixpkgs that referenced this pull request Oct 17, 2020

libplacebo: Take out glslang patch not needed after NixOS#94049

d8b5d9a

From some reason, if we take this patch out, libplacebo builds fine, and mpv is again usable.

doronbehar mentioned this pull request Oct 17, 2020

libplacebo: Take out glslang patch not needed after #94049 #100825

Closed

10 tasks

dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Nov 6, 2020

Merge pull request NixOS#94049 from CajuM/vulkan-up

b2c8af3

(cherry picked from commit 0826f2e)

dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Nov 6, 2020

Revert "Merge pull request NixOS#94049 from CajuM/vulkan-up" (rebuilds)

9f16017

This reverts commit b2c8af3.

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Vulkan, SPIR-V, glslang and shaderc #94049

Update Vulkan, SPIR-V, glslang and shaderc #94049

CajuM commented Jul 28, 2020 •

edited

Ralith left a comment

jonringer commented Aug 2, 2020

jonringer commented Aug 3, 2020

jonringer commented Aug 3, 2020

CajuM commented Aug 3, 2020 •

edited

CajuM commented Aug 3, 2020 •

edited

jonringer commented Aug 3, 2020

jonringer commented Aug 3, 2020

Ralith commented Aug 5, 2020

CajuM commented Sep 3, 2020

doronbehar Sep 24, 2020

CajuM Sep 25, 2020

Ralith Sep 27, 2020

doronbehar Sep 27, 2020 •

edited

CajuM Sep 27, 2020 •

edited

doronbehar Oct 13, 2020

jonringer commented Oct 13, 2020

doronbehar left a comment

doronbehar Oct 14, 2020

doronbehar Oct 14, 2020

doronbehar Oct 14, 2020

SuperSandro2000 commented Oct 17, 2020

doronbehar commented Oct 17, 2020

CajuM commented Oct 17, 2020

doronbehar commented Oct 17, 2020

SuperSandro2000 commented Oct 17, 2020

doronbehar commented Oct 17, 2020

CajuM commented Oct 17, 2020

doronbehar commented Oct 17, 2020

doronbehar commented Oct 17, 2020

acowley commented Feb 10, 2021

SuperSandro2000 commented Feb 11, 2021 •

edited

doronbehar commented Feb 15, 2021

Ralith commented Feb 15, 2021

Update Vulkan, SPIR-V, glslang and shaderc #94049

Update Vulkan, SPIR-V, glslang and shaderc #94049

Conversation

CajuM commented Jul 28, 2020 • edited

Motivation for this change

Things done

Ralith left a comment

Choose a reason for hiding this comment

jonringer commented Aug 2, 2020

jonringer commented Aug 3, 2020

jonringer commented Aug 3, 2020

CajuM commented Aug 3, 2020 • edited

CajuM commented Aug 3, 2020 • edited

jonringer commented Aug 3, 2020

jonringer commented Aug 3, 2020

Ralith commented Aug 5, 2020

CajuM commented Sep 3, 2020

doronbehar Sep 24, 2020

Choose a reason for hiding this comment

CajuM Sep 25, 2020

Choose a reason for hiding this comment

Ralith Sep 27, 2020

Choose a reason for hiding this comment

doronbehar Sep 27, 2020 • edited

Choose a reason for hiding this comment

CajuM Sep 27, 2020 • edited

Choose a reason for hiding this comment

doronbehar Oct 13, 2020

Choose a reason for hiding this comment

jonringer commented Oct 13, 2020

doronbehar left a comment

Choose a reason for hiding this comment

doronbehar Oct 14, 2020

Choose a reason for hiding this comment

doronbehar Oct 14, 2020

Choose a reason for hiding this comment

doronbehar Oct 14, 2020

Choose a reason for hiding this comment

SuperSandro2000 commented Oct 17, 2020

doronbehar commented Oct 17, 2020

CajuM commented Oct 17, 2020

doronbehar commented Oct 17, 2020

SuperSandro2000 commented Oct 17, 2020

doronbehar commented Oct 17, 2020

CajuM commented Oct 17, 2020

doronbehar commented Oct 17, 2020

doronbehar commented Oct 17, 2020

acowley commented Feb 10, 2021

SuperSandro2000 commented Feb 11, 2021 • edited

doronbehar commented Feb 15, 2021

Ralith commented Feb 15, 2021

CajuM commented Jul 28, 2020 •

edited

CajuM commented Aug 3, 2020 •

edited

CajuM commented Aug 3, 2020 •

edited

doronbehar Sep 27, 2020 •

edited

CajuM Sep 27, 2020 •

edited

SuperSandro2000 commented Feb 11, 2021 •

edited