Closes #85637

This fixes the build of scipy.

Merged to unbreak packages on master. Further fixes can be made in a follow-up PR.

FYI, I think the nixos-unstable channel also needs this fix. The build I just ran against that channel failed.

@matthewbauer looks like Python docs is missing a section. link to overlays is broken on MKL section

@matthewbauer looks like Python docs is missing a section. link to overlays is broken on MKL section

It's a reference to another section in manual. It does render properly in the full html, but is a little awkward on its own.

@matthewbauer am I missing something or is there no way to have an application rely on MKL, while rest of system is on whatever a user wants? Previously, I could write an application that does:

let
  pythonOverrides = python-self: python-super: {
    # openblas does not work https://github.com/AlexEMG/DeepLabCut/issues/637#issuecomment-606893058
    numpy = python-super.numpy.override { blas = mkl; };
  };

  python = python3.override {
    packageOverrides = pythonOverrides;
  };
  py = python.pkgs;
in
py.toPythonApplication [...]

There are scientific applications that only work with MKL, or only get correct results with OpenBLAS for example. How should I specify this as a package maintainer?

@matthewbauer I have the same question as @tbenst. Due to #104026 I need to force R use OpenBlas while keeping the rest of the system default to MKL. After few unsuccessful attempts I ended up with this which appears to be working:

 R = super.R.override {
    blas = super.blas.override { blasProvider = super.openblasCompat; };
    lapack = super.lapack.override { lapackProvider = super.openblasCompat; };
 };

Is that the right approach?

Is that the right approach?

LGTM

you can do nix-store --query --tree $(nix-build) to verify which version was included into the closure

Yes, looks good. I really wish #104026 is fixed somehow. R from nixpkgs is the only R I know that works incorrectly with MKL. I am thinking to create my own derivation for R which uses MKL directly.

Yes, looks good. I really wish #104026 is fixed somehow. R from nixpkgs is the only R I know that works incorrectly with MKL. I am thinking to create my own derivation for R which uses MKL directly.

might also be nice to just change the R packaging so that it can work with the blas/lapack providers more nicely. Looking at https://stackoverflow.com/questions/14996697/compile-r-with-mkl-with-mulithreads-support , may expose some passthru linking options, and just have R consume those additional configure flags conditionally

from R/default.nix

      --with-blas="-L${blas}/lib -lblas"

should be something like

      --with-blas="-L${blas}/lib -lblas ${blas.configureFlags}"

in mkl/default.nix

  passthru = {
    tests = ...;
    configureFlags = "-lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core "
  };

in alternative/blas/default.nix

  passthru = {
    inherit isILP64;
    provider = blasProvider;
    implementation = blasImplementation;
    configureFlags = blasProvider.configureFlags or "";
  };

If we did go this route, probably would want to do the same for lapack. And I would probably make the flags an array, as string are kind of "messy".

@jonringer I copied R/default.nix and tried applying the changes directly. It didn't work at first but then I decided to just replace the two lines --with-blas/--with-lapack with what Intel recommended in https://software.intel.com/content/www/us/en/develop/articles/using-intel-mkl-with-r.html (I've added mkl to the arguments of the derivation):

--with-blas="-L${mkl}/lib -Wl,--no-as-needed -lmkl_gf_lp64 -Wl,--start-group -lmkl_gnu_thread  -lmkl_core  -Wl,--end-group -fopenmp  -ldl -lpthread -lm"
--with-lapack

and the resulting R finally showed correct results for my test (#104026). Note that there is no =... after --with-lapack. Do you think it is worth adding RwithMKL or something like that to nixpkgs until this generic framework is fixed?