Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

openblas, mkl: also export unversioned libraries for linux #87134

Merged
merged 2 commits into from May 8, 2020
Merged

openblas, mkl: also export unversioned libraries for linux #87134

merged 2 commits into from May 8, 2020

Conversation

jonringer
Copy link
Contributor

@jonringer jonringer commented May 6, 2020 •
edited

Motivation for this change

While working on #87118 , I noticed that the tests weren't able to find -llapack, even though it was available through openblas. Turns out that it was explicitly searching for just liblapack.so and not liblapack.so.3.

for linux, this will now export both the normal .so and the versioned .so.3

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@jonringer jonringer requested review from bhipple and ttuegel May 6, 2020 19:56
@jonringer jonringer mentioned this pull request May 6, 2020
Merged
10 tasks
@bhipple bhipple requested a review from matthewbauer May 7, 2020 01:10
bhipple
bhipple approved these changes May 7, 2020
View reviewed changes
Copy link
Contributor

@bhipple bhipple left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a good change to me!

@drewrisinger
Copy link
Contributor

drewrisinger commented May 7, 2020 •
edited

cc @matthewbauer, figured he should be looped in b/c he spent a lot of time looking at the LAPACK/BLAS system recently.
EDIT: whoops, missed the pre-existing review request, didn't read closely.

@matthewbauer
Copy link
Member

So -lblas and -llapack should resolve to blas.so.3 and lapack.so.3 if blas.soandlapack.so` are unavailable. But some other things might expect it to be there.

matthewbauer
matthewbauer requested changes May 7, 2020
View reviewed changes
Copy link
Member

@matthewbauer matthewbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost forgot that we also need this for mkl as well!

@bhipple
mkl: include symlinks to versioned libblas.so.3 names
296f5b0 
Some build systems look for this specifically.
@bhipple bhipple force-pushed the openblas-export-unversioned branch from 386b5ab to 296f5b0 Compare May 8, 2020 01:43
@bhipple
Copy link
Contributor

bhipple commented May 8, 2020

Updated with mkl support as well. I also used a let binding to avoid excessively long strings for the shlib file extension lookup.

LGTM now:

$ ls -l $(nix-build -A openblas)/lib/lib*
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libblas.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libblas.so.3 -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libcblas.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libcblas.so.3 -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/liblapacke.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/liblapacke.so.3 -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/liblapack.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/liblapack.so.3 -> libopenblasp-r0.3.9.so
-r-xr-xr-x  3 root root 27850664 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libopenblas.so -> libopenblasp-r0.3.9.so
lrwxrwxrwx 23 root root       22 Dec 31  1969 /nix/store/45bkmkl800030xy3s913sxsmv7s64652-openblas-0.3.9/lib/libopenblas.so.0 -> libopenblasp-r0.3.9.so

$ ls -l $(nix-build -A mkl)/lib/lib*
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libblas.so -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libblas.so.3 -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libcblas.so -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libcblas.so.3 -> libmkl_rt.so
-r-xr-xr-x  8 root root   697489 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libiomp5_db.so
-r-xr-xr-x  8 root root  2315028 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libiomp5.so
-r-xr-xr-x  8 root root   192373 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libiompstubs5.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/liblapacke.so -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/liblapacke.so.3 -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/liblapack.so -> libmkl_rt.so
lrwxrwxrwx 29 root root       12 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/liblapack.so.3 -> libmkl_rt.so
-r-xr-xr-x  8 root root 47925310 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_avx2.so
-r-xr-xr-x  8 root root 65553609 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_avx512_mic.so
-r-xr-xr-x  8 root root 60359069 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_avx512.so
-r-xr-xr-x  8 root root 51230378 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_avx.so
-r-xr-xr-x  8 root root   499559 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_intelmpi_ilp64.so
-r-xr-xr-x  8 root root   302050 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_intelmpi_lp64.so
-r-xr-xr-x  8 root root   504799 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_openmpi_ilp64.so
-r-xr-xr-x  8 root root   307258 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_openmpi_lp64.so
-r-xr-xr-x  8 root root   499880 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_sgimpt_ilp64.so
-r-xr-xr-x  8 root root   302339 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_blacs_sgimpt_lp64.so
-r-xr-xr-x  8 root root   169344 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_cdft_core.so
-r-xr-xr-x  8 root root 71510820 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_core.so
-r-xr-xr-x  8 root root 40895803 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_def.so
-r-xr-xr-x  8 root root 10507840 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_gf_ilp64.so
-r-xr-xr-x  8 root root 11218247 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_gf_lp64.so
-r-xr-xr-x  8 root root 25398547 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_gnu_thread.so
-r-xr-xr-x  8 root root 10512146 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_intel_ilp64.so
-r-xr-xr-x  8 root root 11222553 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_intel_lp64.so
-r-xr-xr-x  8 root root 40645032 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_intel_thread.so
-r-xr-xr-x  8 root root 48789719 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_mc3.so
-r-xr-xr-x  8 root root 47256466 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_mc.so
-r-xr-xr-x  8 root root  6592837 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_rt.so
-r-xr-xr-x  8 root root  7725985 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_scalapack_ilp64.so
-r-xr-xr-x  8 root root  7735678 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_scalapack_lp64.so
-r-xr-xr-x  8 root root 22190068 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_sequential.so
-r-xr-xr-x  8 root root 12556013 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_avx2.so
-r-xr-xr-x  8 root root 13763906 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_avx512_mic.so
-r-xr-xr-x  8 root root 12386491 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_avx512.so
-r-xr-xr-x  8 root root 12616795 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_avx.so
-r-xr-xr-x  8 root root  6375214 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_cmpt.so
-r-xr-xr-x  8 root root  6748578 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_def.so
-r-xr-xr-x  8 root root 11355391 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_mc2.so
-r-xr-xr-x  8 root root 11433131 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_mc3.so
-r-xr-xr-x  8 root root 11386740 Dec 31  1969 /nix/store/7s1k6wazbvynzf5hfpr4fli7l6z4w379-mkl-2020.1.217/lib/libmkl_vml_mc.so

@bhipple
Copy link
Contributor

bhipple commented May 8, 2020

@GrahamcOfBorg build openblas

@bhipple bhipple changed the title openblas: also export unversioned libraries for linux openblas, mkl: also export unversioned libraries for linux May 8, 2020
@ofborg ofborg bot requested a review from bhipple May 8, 2020 01:52
@FRidh FRidh requested a review from matthewbauer May 8, 2020 04:51
@FRidh FRidh added this to WIP in Staging via automation May 8, 2020
@FRidh FRidh moved this from WIP to Needs review in Staging May 8, 2020
@FRidh FRidh merged commit 7066420 into NixOS:staging May 8, 2020
Staging automation moved this from Needs review to Done May 8, 2020
@jonringer jonringer deleted the openblas-export-unversioned branch May 8, 2020 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ttuegel ttuegel Awaiting requested review from ttuegel

@bhipple bhipple Awaiting requested review from bhipple

@matthewbauer matthewbauer Awaiting requested review from matthewbauer

Assignees
No one assigned
Labels
10.rebuild-darwin: 0 10.rebuild-linux: 501+ 10.rebuild-linux: 1001-2500
Projects
Staging
  
Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@jonringer @drewrisinger @matthewbauer @bhipple @FRidh