mesa: restore OpenCL support. #82729

athas · 2020-03-16T14:49:23Z

Motivation for this change

This is a first shot at addressing #62933. In practice, OpenCL support should probably be hidden behind an option, because it takes up a significant chunk of space (requires a full LLVM at run-time).

Currently I put all the OpenCL stuff in the drivers output, because at least mesa.icd needs to be available via /run/opengl-driver.

Things done

killercup

If someone can tell me how to best test this, I'll give it a go!

Edit: Also cmd+enter triggering the "approve" button is not the wisest choice, github

pkgs/development/libraries/mesa/default.nix

rnhmjoj · 2020-03-17T13:04:12Z

@athas As you said, this should be behind a guard like enableOpenCL.

I would suggest to create a package mesa-opencl = mesa.override { enableOpenCL = true; } or maybe an output like mesa.opencl, if possible.
Then just adding it to opengl.extraPackages should install the ICD and the driver.

If someone can tell me how to best test this, I'll give it a go!

If you build mesa from you nixpkgs checkout, you can install it in you system by using the (internal) opengl.package option. Alternatively, add it to opengl.extraPackages if you only want to test openCL.

To test wheter the openCL driver can be loaded you can use opengl-info.
I have a small program that uses an OpenCL kernel to compute the Hurst exponent laying around, I could give you that but I'm sure there are better tests.

athas · 2020-03-17T13:10:37Z

I had trouble using a mesa.opencl output, because I got circular dependencies between the drivers and opencl output. I think libOpenCL.so (which is big) has to live in drivers, because that is where mesa.icd must be in order to be accessible via /run/opengl-driver/etc/OpenCL/mesa.icd. At least, addressing this is beyond my current skill set.

For the same reason, it's also a little tricky to debug, since the ICD loader only seems to pick up stuff in /run/opengl-driver/etc/OpenCL.

killercup · 2020-03-17T14:27:12Z

you can install it in you system by using the (internal) opengl.package option. Alternatively, add it to opengl.extraPackages if you only want to test openCL.

Thanks for the tips, @rnhmjoj! This works for me with a 5700XT:

{
hardware.opengl = let
    opencl_pr = import (builtins.fetchTarball {
        name = "opencl_pr";
        url = "https://github.com/athas/nixpkgs/archive/f92a2a9b69eba9909d25ffaab6ded4d6f0f4efad.tar.gz";
    }) { };
in {
    enable = true;
    driSupport32Bit = true;
    package = opencl_pr.mesa.drivers;
    extraPackages = [ opencl_pr.mesa ];
};
}

clinfo output looks good, but it also contains

=== CL_PROGRAM_BUILD_LOG ===
fatal error: cannot open file '/nix/store/l81x3yfwahahhxmfk4wbg2ki025v451p-libclc-2019-06-09/lib/clc/gfx1010-amdgcn-mesa-mesa3d.bc': No such file or directory
  Preferred work group size multiple              <getWGsizes:1200: create kernel : error -46>

Not sure if I should worry about that.

athas · 2020-03-17T14:48:18Z

I think the libclc is just too old for that GPU. I'll make a PR about bumping libclc.

athas · 2020-03-17T14:57:09Z

Oh, apparently Mesa OpenCL doesn't support Navi cards at all...

killercup · 2020-03-17T15:33:25Z

Oh, that's too bad! This is what clinfo tell me now, but I don't really have a clue what everything means.

$ clinfo                                                                                                                                                                                                                                                   
Number of platforms                               1
  Platform Name                                   Clover
  Platform Vendor                                 Mesa
  Platform Version                                OpenCL 1.1 Mesa 19.3.3
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd
  Platform Extensions function suffix             MESA

  Platform Name                                   Clover
Number of devices                                 1
  Device Name                                     AMD NAVI10 (DRM 3.36.0, 5.5.9, LLVM 9.0.1)
  Device Vendor                                   AMD
  Device Vendor ID                                0x1002
  Device Version                                  OpenCL 1.1 Mesa 19.3.3
  Driver Version                                  19.3.3
  Device OpenCL C Version                         OpenCL C 1.1 
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Max compute units                               40
  Max clock frequency                             2100MHz
  Max work item dimensions                        3
  Max work item sizes                             256x256x256
  Max work group size                             256
=== CL_PROGRAM_BUILD_LOG ===
fatal error: cannot open file '/nix/store/l81x3yfwahahhxmfk4wbg2ki025v451p-libclc-2019-06-09/lib/clc/gfx1010-amdgcn-mesa-mesa3d.bc': No such file or directory
  Preferred work group size multiple              <getWGsizes:1200: create kernel : error -46>
  Preferred / native vector sizes                 
    char                                                16 / 16      
    short                                                8 / 8       
    int                                                  4 / 4       
    long                                                 2 / 2       
    half                                                 8 / 8        (cl_khr_fp16)
    float                                                4 / 4       
    double                                               2 / 2        (cl_khr_fp64)
  Half-precision Floating-point support           (cl_khr_fp16)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Address bits                                    64, Little-Endian
  Global memory size                              8589934592 (8GiB)
  Error Correction support                        No
  Max memory allocation                           6871947673 (6.4GiB)
  Unified memory for Host and Device              No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       32768 bits (4096 bytes)
  Global Memory cache type                        None
  Image support                                   No
  Local memory type                               Local
  Local memory size                               32768 (32KiB)
  Max number of constant args                     16
  Max constant buffer size                        2147483647 (2GiB)
  Max size of kernel argument                     1024
  Queue properties                                
    Out-of-order execution                        No
    Profiling                                     Yes
  Profiling timer resolution                      0ns
  Execution capabilities                          
    Run OpenCL kernels                            Yes
    Run native kernels                            No
  Device Extensions                               cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp64 cl_khr_fp16

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  Clover
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [MESA]
  clCreateContext(NULL, ...) [default]            Success [MESA]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 Clover
    Device Name                                   AMD NAVI10 (DRM 3.36.0, 5.5.9, LLVM 9.0.1)
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 Clover
    Device Name                                   AMD NAVI10 (DRM 3.36.0, 5.5.9, LLVM 9.0.1)
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 Clover
    Device Name                                   AMD NAVI10 (DRM 3.36.0, 5.5.9, LLVM 9.0.1)

ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.2.10
  ICD loader Profile                              OpenCL 2.1

athas · 2020-03-17T18:39:24Z

I think that means that all the infrastructure works. The OpenCL ICD loader can find the Mesa ICD and talk to the hardware. It just can't compile any kernels, because some GPU-specific library files are missing.

Here is a small OpenCL test program that compiles and launches a kernel. Compile with

$ nix-shell -p ocl-icd -p opencl-headers --run 'gcc rot13.c -o rot13 -lOpenCL'

and then run with

$ ./rot13

I don't expect it'll work on your machine. It also doesn't work on my Vega 64, as the kernel launch hangs forever. That's an improvement, though: Mesa OpenCL used to reboot my computer, even on non-NixOS distributions.

The general buggyness of Mesa OpenCL is going to be an obstacle to testing this...

rnhmjoj · 2020-03-17T19:47:49Z

Oh, that's too bad! This is what clinfo tell me now, but I don't really have a clue what everything means.

That looks good. In case the driver is not found you would simply get:

ERROR: clGetPlatformIDs(-1001)

killercup · 2020-03-17T21:15:02Z

@athas yep, doesn't start, "fatal error: cannot open file '/nix/store/l81x3yfwahahhxmfk4wbg2ki025v451p-libclc-2019-06-09/lib/clc/gfx1010-amdgcn-mesa-mesa3d.bc': No such file or directory".

davidak · 2020-05-16T22:06:37Z

I added the code from #82729 (comment) to my configuration.nix and now OpenCL works!

[davidak@gaming:~]$ nix run nixpkgs.clinfo -c clinfo
Number of platforms                               1
  Platform Name                                   Clover
  Platform Vendor                                 Mesa
  Platform Version                                OpenCL 1.1 Mesa 19.3.3
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd
  Platform Extensions function suffix             MESA

  Platform Name                                   Clover
Number of devices                                 1
  Device Name                                     AMD Radeon (TM) RX 480 Graphics (POLARIS10, DRM 3.35.0, 5.4.41, LLVM 9.0.1)

Even BOINC detects the GPU now:

davidak

works, changes looks OK as far as i can judge

rnhmjoj · 2020-05-17T10:37:01Z

@vcunat can you take a look at this PR?

rnhmjoj

The change looks good but openCL support should be optional to avoid a large closure size increase.
For convenience, a mesa-opencl package could be added so that users don't have to rebuild mesa.

cab404 · 2020-05-20T22:59:55Z

I tried it, but blender doesn't report my videocard (540X) as a usable device. Is that normal?

davidak · 2020-05-29T03:02:18Z

@cab404 i also don't see my AMD Radeon RX 480 as supported. so might be blender specific

mohe2015 · 2020-11-13T21:56:01Z

@athas May you rebase this please?

athas · 2020-11-13T22:16:33Z

I'll try over the weekend, but I won't have the spare life cycles to actually test it.

mohe2015 · 2020-11-13T22:23:48Z

Cool @athas. https://github.com/mohe2015/nixpkgs/tree/mesa-opencl may or may not be a correct rebase (nonsquashed, only the first three commits as I did some further tests). Feel free to use it (without me as an author is fine as it's just a rebase)

I'm building now but it seems like it's almost a full rebuild...

SuperSandro2000 · 2020-11-23T23:55:21Z

Can you please fix the merge conflict and target staging with the mass rebuild?

timhae · 2020-12-04T12:35:01Z

I also added the code from the comment to my configuration.nix and was able to compile the program mentioned in the other comment, the output is (as expected) Hello, World!. Thanks a lot for your effort and please let me know if I can do any additional testing to help you.

athas · 2020-12-05T09:15:43Z

I have rebased on top of master. Haven't tried running it, but by inspection of the files, it looks OK.

nixos-discourse · 2020-12-06T16:55:18Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/openvx-opencl-setup-for-amd-cpu-and-gpu/10383/1

pkgs/development/libraries/mesa/default.nix

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

SuperSandro2000 · 2020-12-08T04:08:15Z

Please target staging and squash the commits.

shpinog · 2021-03-03T14:18:20Z

I also used the workaround above, but it doesn't work now. Unfortunately, rocm does not work on many platforms, as it requires hardware support for the processor.

SuperSandro2000 · 2021-03-03T17:21:12Z

pkgs/development/libraries/mesa/default.nix

+    fi
+
+    # We construct our own .icd file that contains an absolute path.
+    rm -rf $out/etc/OpenCL


Suggested change

rm -rf $out/etc/OpenCL

rm -r $out/etc/OpenCL

nixos-discourse · 2021-03-04T19:57:56Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-get-opecl-working-on-amdgpu/11786/2

davidak · 2021-04-04T13:24:06Z

I also used this workaround, since may last year and noticed now that it does not work anymore.

My config.: https://codeberg.org/davidak/nixos-config/src/commit/3d15f77fbe04177c546c0a15026ec1f3240d50ac/modules/amd/default.nix

I don't see any OpenCL device:

[davidak@gaming:~/code/nixpkgs]$ nix run nixpkgs.clinfo -c clinfo
Number of platforms                               0

system: "x86_64-linux"
host os: Linux 5.11.2, NixOS, 21.05pre274338.102eb68ceec (Okapi)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.10
channels(root): "home-manager-20.09, nixos-21.05pre279456.04a2b269d89, nixos-hardware, nixos-unstable-21.05pre279456.04a2b269d89"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maybe this needs an update?

kvtb · 2021-08-31T22:22:26Z

Could we merge it under enableOpenCL conditional, so those who concern the closure size could opt-out?
It is the only showstopper, isn't it?

sbruder · 2021-09-01T13:48:55Z

I rebased the PR (and the included patch against mesa) onto the latest master and added a switch for making OpenCL support optional (as @rnhmjoj suggested): https://github.com/sbruder/nixpkgs/commits/mesa-opencl and tested the PR cherry-picked onto 21.05 with the following config:

{ pkgs, ... }:
{
  hardware.opengl =
    let
      pkgsMesaClover = import
        (pkgs.fetchFromGitHub {
          owner = "sbruder";
          repo = "nixpkgs";
          rev = "41d33334ec2d07da8f5bd0f3749949241c42266b";
          sha256 = "sha256-sA/kwZ8kXuRC/sLjja2ISpyTnlEwZjifyYeiz9hFWRE=";
        })
        { inherit (pkgs) system; };
    in
    {
      package = pkgsMesaClover.mesa-opencl.drivers;
      extraPackages = [
        pkgsMesaClover.mesa-opencl
      ];
    };
}

After a rebuild clinfo detects my RX 480 (gist with the output of clinfo and comparison to its output with ROCm), though it only shows support for OpenCL 1.1, not 1.2 as it should support (though I don’t know if upstream does not support it on my hardware or if this is due to how nixpkgs packages it).

Some programs that work include @athas’ rot13 example, hashcat (with --force) and waifu2x-converter-cpp. Others that don’t work include Blender (which is a known problem and Blender will drop OpenCL support in the near future) and the NNEDI3CL VapourSynth plugin (fails with Boost.Compute: kernel compilation failed (-43)).

Besides that, some programs that do work show warnings like this, which probably are not problematic, but seems like it’s not fully working yet:

'+fp32-denormals' is not a recognized feature for this target (ignoring feature)
'+fp64-fp16-denormals' is not a recognized feature for this target (ignoring feature)
'+fp32-denormals' is not a recognized feature for this target (ignoring feature)
'+fp64-fp16-denormals' is not a recognized feature for this target (ignoring feature)
'+fp32-denormals' is not a recognized feature for this target (ignoring feature)
'+fp64-fp16-denormals' is not a recognized feature for this target (ignoring feature)
'+fp32-denormals' is not a recognized feature for this target (ignoring feature)
'+fp64-fp16-denormals' is not a recognized feature for this target (ignoring feature)

rnhmjoj · 2021-09-01T15:51:40Z

@sbruder can you make a new pull request (against staging) with your patch?

primeos · 2021-09-18T21:49:34Z

I'll go ahead and close this in favor of #136402 (will be merged soon).

Thanks @athas et al.

killercup approved these changes Mar 17, 2020

View reviewed changes

pkgs/development/libraries/mesa/default.nix Show resolved Hide resolved

athas force-pushed the mesa-opencl branch from 3aaf3ea to f92a2a9 Compare March 17, 2020 12:46

ofborg bot requested a review from vcunat March 17, 2020 12:55

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 1001-2500 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Mar 17, 2020

FRidh added this to Needs review in Staging Apr 3, 2020

davidak approved these changes May 16, 2020

View reviewed changes

This was referenced May 16, 2020

geekbench5 don't find compute devices #87990

Closed

amdgpu-pro no longer works on unstable #33619

Closed

rnhmjoj requested changes May 17, 2020

View reviewed changes

FRidh moved this from Needs review to WIP in Staging Jun 19, 2020

FRidh added this to the 20.09 milestone Jun 21, 2020

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

ryantm added the 2.status: merge conflict label Oct 3, 2020

mesa: restore OpenCL support.

618da44

athas force-pushed the mesa-opencl branch from f92a2a9 to 618da44 Compare December 5, 2020 09:15

ofborg bot removed the 2.status: merge conflict label Dec 5, 2020

ofborg bot requested a review from primeos December 5, 2020 09:25

SuperSandro2000 reviewed Dec 6, 2020

View reviewed changes

pkgs/development/libraries/mesa/default.nix Outdated Show resolved Hide resolved

pkgs/development/libraries/mesa/default.nix Outdated Show resolved Hide resolved

athas and others added 2 commits December 6, 2020 23:34

Update pkgs/development/libraries/mesa/default.nix

bdc8fe0

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

Update pkgs/development/libraries/mesa/default.nix

a6fb169

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

FRidh modified the milestones: 20.09, 21.03 Dec 20, 2020

SuperSandro2000 reviewed Mar 3, 2021

View reviewed changes

SuperSandro2000 added the 2.status: merge conflict label Apr 5, 2021

sbruder mentioned this pull request Sep 1, 2021

mesa: restore OpenCL support. #136402

Merged

12 tasks

primeos closed this Sep 18, 2021

Staging automation moved this from WIP to Done Sep 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mesa: restore OpenCL support. #82729

mesa: restore OpenCL support. #82729

athas commented Mar 16, 2020

killercup left a comment •

edited

rnhmjoj commented Mar 17, 2020

athas commented Mar 17, 2020

killercup commented Mar 17, 2020

athas commented Mar 17, 2020

athas commented Mar 17, 2020

killercup commented Mar 17, 2020

athas commented Mar 17, 2020

rnhmjoj commented Mar 17, 2020

killercup commented Mar 17, 2020

davidak commented May 16, 2020

davidak left a comment

rnhmjoj commented May 17, 2020

rnhmjoj left a comment

cab404 commented May 20, 2020

davidak commented May 29, 2020

mohe2015 commented Nov 13, 2020

athas commented Nov 13, 2020

mohe2015 commented Nov 13, 2020 •

edited

SuperSandro2000 commented Nov 23, 2020

timhae commented Dec 4, 2020

athas commented Dec 5, 2020

nixos-discourse commented Dec 6, 2020

SuperSandro2000 commented Dec 8, 2020

shpinog commented Mar 3, 2021 •

edited

SuperSandro2000 Mar 3, 2021

nixos-discourse commented Mar 4, 2021

davidak commented Apr 4, 2021

kvtb commented Aug 31, 2021 •

edited

sbruder commented Sep 1, 2021

rnhmjoj commented Sep 1, 2021

primeos commented Sep 18, 2021

mesa: restore OpenCL support. #82729

mesa: restore OpenCL support. #82729

Conversation

athas commented Mar 16, 2020

Motivation for this change

Things done

killercup left a comment • edited

Choose a reason for hiding this comment

rnhmjoj commented Mar 17, 2020

athas commented Mar 17, 2020

killercup commented Mar 17, 2020

athas commented Mar 17, 2020

athas commented Mar 17, 2020

killercup commented Mar 17, 2020

athas commented Mar 17, 2020

rnhmjoj commented Mar 17, 2020

killercup commented Mar 17, 2020

davidak commented May 16, 2020

davidak left a comment

Choose a reason for hiding this comment

rnhmjoj commented May 17, 2020

rnhmjoj left a comment

Choose a reason for hiding this comment

cab404 commented May 20, 2020

davidak commented May 29, 2020

mohe2015 commented Nov 13, 2020

athas commented Nov 13, 2020

mohe2015 commented Nov 13, 2020 • edited

SuperSandro2000 commented Nov 23, 2020

timhae commented Dec 4, 2020

athas commented Dec 5, 2020

nixos-discourse commented Dec 6, 2020

SuperSandro2000 commented Dec 8, 2020

shpinog commented Mar 3, 2021 • edited

SuperSandro2000 Mar 3, 2021

Choose a reason for hiding this comment

nixos-discourse commented Mar 4, 2021

davidak commented Apr 4, 2021

kvtb commented Aug 31, 2021 • edited

sbruder commented Sep 1, 2021

rnhmjoj commented Sep 1, 2021

primeos commented Sep 18, 2021

killercup left a comment •

edited

mohe2015 commented Nov 13, 2020 •

edited

shpinog commented Mar 3, 2021 •

edited

kvtb commented Aug 31, 2021 •

edited