[WIP] sage: Add darwin support #45364

jbaum98 · 2018-08-20T03:56:18Z

Motivation for this change

I want sage to build on darwin. Lot's of packages just need their platforms updated.

Right now, I'm having trouble getting sagelib to build, failing with an error

clang++ -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-2.7/build/cythonized/sage/libs/linbox/linbox.o -L/nix/store/5w9j23h0qsar813vdvsi1dyhg73iczlh-givaro-4.0.4/lib -L/nix/store/l3xnp9rniv92n69m2mls4ww0s14jqln0-linbox-1.5.2/lib -L/private/tmp/nix-build-python2.7-sagelib-8.3.drv-1/sage-src-8.3/lib -L/nix/store/xkksa1sdiddm4m7hd2mx87acvbs3rzjd-python-2.7.15/lib -llinbox -llinboxsage -lblas -lgivaro -lgmp -lgmpxx -lstdc++ -o build/lib.macosx-10.10-x86_64-2.7/sage/libs/linbox/linbox.so
ld: warning: directory not found for option '-L/private/tmp/nix-build-python2.7-sagelib-8.3.drv-1/sage-src-8.3/lib'
ld: library not found for -lblas
clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation)

I know it's blas related, but there's a lot of blas stuff going on in pkgs/applications/science/math/sage/sagelib.nix. @timokau any thoughts?

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

timokau · 2018-08-20T08:28:55Z

The equivalent log line from the linux build is

g++ -pthread -shared -lgcc_s build/temp.linux-x86_64-2.7/build/cythonized/sage/libs/linbox/linbox.o -L/nix/store/5pbx09wbqrmfplri7jgzq4gcpnbcd0nm-givaro-4.0.4/lib -L/nix/store/m4lqi72qdfnkr2vyydl8hr2x1fwhmyhk-linbox-1.5.2/lib -L/build/sage-src-8.3/lib -L/nix/store/f01rhaxdfip9c49k4mhfjgfrwl5lcamn-python-2.7.15/lib -llinbox -llinboxsage -lgivaro -lgmp -lgmpxx -lopenblas -lstdc++ -lpython2.7 -o build/lib.linux-x86_64-2.7/sage/libs/linbox/linbox.so

I'm honestly a bit surprised how that works because the openblas lib dir is not passed to the linker. Anyways I think the important difference is -lopenblas on linux vs -lblas on darwin. Anyways the args passed to gcc are generated in sage's env.py starting at line 313. That is done by calling pkgconfig. Since you changed linboxes blas from openblas to blas, that causes the difference. So you probably need to add plain blas as a dependency for sage as well.

timokau · 2018-08-20T08:30:13Z

pkgs/applications/science/math/giac/default.nix

@@ -91,7 +91,8 @@ stdenv.mkDerivation rec {
    license = licenses.gpl3Plus;
    ## xcas is buildable on darwin but there are specific instructions I could
    ## not test
-    platforms = platforms.linux;
+    ## ^^ not sure what this means


@symphorien do you remember what you meant by that?

Yeah that was a note that I wanted to discuss that comment above about xcas

OSX is supported as per https://www-fourier.ujf-grenoble.fr/~parisse/giac_compile.html
but the compilation process is said to be different: quote:

Notes for compilation on Mac OS X. For recent versions of giac (>=0.9.x) on Mac OS X.6 run the script mkosx.

So I thought the derivation would not work as such on osx and would require more work.

If xcas works as is on darwin, you can just remove the comment.

jbaum98 · 2018-08-20T15:08:05Z

There are like three or four dependencies with openblas in the name, a few of which seem to have to do with pkg-config. I should change them all to blas? Or do I just have to change the input to sage in all-packages and it will propagate by itslef?

timokau · 2018-08-20T15:42:14Z

Those provide different pkg-config files for openblas, see #45257. If you want to switch to blas completely, you probably would have to provide equivalents of those for blas. For now I'd just try to add blas in addition to openblas. Its not pretty but it might work (its only needed to compile the cython files interfacing with linbox).

timokau · 2018-08-20T15:43:27Z

Also I think openblas and atlas are the only two officially supported blas implementations by sage. So not sure if plain blas would work.

timokau · 2018-10-03T20:10:52Z

@jbaum98 are you still interested in this?

jbaum98 · 2018-10-04T00:41:42Z

I am, I haven't had much time to work on it though. What was the decision on the blas thing?

timokau · 2018-10-04T21:58:07Z

What decision?

timokau · 2018-10-04T21:59:56Z

Regarding the blas thing I would recommend to try if the plain sage package (which currently doesn't run any tests) also fails the fflas_ffpack tests. If not, compare differences in build. If yes, just disable checks on darwin for now (it is reported upstream and nobody has noticed yet so it cannot be too serious).

mmahut · 2019-08-25T18:28:41Z

Are there any updates on this pull request, please?

jbaum98 · 2019-08-25T18:38:04Z

Sorry, this fell off my radar. I may be able to take another crack at it in a few weeks, but I started this because I needed to use sage, and I don't anymore.

timokau · 2019-08-25T21:12:53Z

While full darwin support for sage would be nice of course, isn't the part in this PR basically ready?

vcunat · 2019-08-25T21:14:54Z

Yes, assuming those packages (still) work reasonably on darwin...

timokau · 2019-08-25T22:03:30Z

@jbaum98 are you up to a rebase and a re-test?

jbaum98 · 2019-08-26T01:25:07Z

For sure. I won't be able to do it for another week or two because I don't have a Mac machine available right now.

jbaum98 · 2019-09-06T16:10:56Z

I did a rebase and now it's stuck building gap because libtool is using an absolute path:

/usr/bin/libtool -o libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o
xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'.

This particular issue seems to have a straightforward fix, but I can't say if I'll have time to work on this for a bit. I feel like this isn't the issue I was stuck on last time, but to be honest I don't remember that well.

vcunat · 2019-09-08T07:46:56Z

libgap was an unreachable file, so I removed it in 252a30e.

@GrahamcOfBorg build gap giac lrcalc nauty
@GrahamcOfBorg build pynac singular ecl arb
@GrahamcOfBorg build fplll iml mpfi brial
@GrahamcOfBorg build cliquer m4ri m4rie rankwidth
@GrahamcOfBorg build rubiks sympow python3Packages.cvxopt

vcunat · 2019-09-08T09:04:00Z

The merge conflict was blocking the builds, I presume, so let me try again:

@GrahamcOfBorg build gap giac lrcalc nauty
@GrahamcOfBorg build pynac singular ecl arb
@GrahamcOfBorg build fplll iml mpfi brial
@GrahamcOfBorg build cliquer m4ri m4rie rankwidth
@GrahamcOfBorg build rubiks sympow python3Packages.cvxopt

vcunat · 2019-09-08T09:43:21Z

If I read Borg right, the only build failures are gap and sympow.

vcunat amended the commit a bit; see the PR for details/discussion.

timokau · 2019-09-20T20:51:06Z

Right, libgap is outdated.

If I read Borg right, the only build failures are gap and sympow.

In that case should we just leave out the gap and sympow parts for now?

vcunat · 2019-09-21T07:39:35Z

Yes, that's what I did last weekend; Hydra: https://hydra.nixos.org/eval/1542673#tabs-new

vcunat amended the commit a bit; see the PR for details/discussion. (cherry picked from commit 991c0e1)

timokau · 2019-09-21T08:58:04Z

Nice, thanks @vcunat and @jbaum98 :)

timokau · 2019-09-25T10:05:07Z

By the way I'm still interested in getting sage to run on darwin. As far as I'm aware, there is currently no good way for a mac user to install sage without building it from source. So this could help a lot of people.

If anyone wants to take a shot, feel free to reach out either by opening a new nixpkgs issue or messaging me on IRC/matrix.

jbaum98 requested a review from FRidh as a code owner August 20, 2018 03:56

GrahamcOfBorg added 6.topic: darwin Running or building packages on Darwin 6.topic: python 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 0 labels Aug 20, 2018

timokau reviewed Aug 20, 2018

View reviewed changes

ryantm added the 2.status: work-in-progress label Feb 24, 2019

jbaum98 force-pushed the sage_darwin branch from 7f104a0 to 0c71062 Compare September 6, 2019 16:10

ofborg bot requested review from timokau, 7c6f434c and symphorien September 6, 2019 16:25

ofborg bot added 10.rebuild-linux: 1-10 and removed 10.rebuild-linux: 0 labels Sep 6, 2019

Mark lots of packages as buildable on darwin

e015108

jbaum98 force-pushed the sage_darwin branch from 0c71062 to e015108 Compare September 6, 2019 21:31

fixup! Mark lots of packages as buildable on darwin

748a201

vcunat added a commit that referenced this pull request Sep 15, 2019

treewide: mark packages as buildable on darwin (PR #45364)

991c0e1

vcunat amended the commit a bit; see the PR for details/discussion.

vcunat closed this Sep 21, 2019

vcunat added a commit that referenced this pull request Sep 21, 2019

treewide: mark packages as buildable on darwin (PR #45364)

2b72c13

vcunat amended the commit a bit; see the PR for details/discussion. (cherry picked from commit 991c0e1)

symphorien mentioned this pull request Nov 25, 2019

giac-with-xcas: fix build #74068

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] sage: Add darwin support #45364

[WIP] sage: Add darwin support #45364

jbaum98 commented Aug 20, 2018

timokau commented Aug 20, 2018

timokau Aug 20, 2018

jbaum98 Aug 20, 2018

symphorien Aug 20, 2018

symphorien Sep 6, 2019 •

edited

jbaum98 commented Aug 20, 2018

timokau commented Aug 20, 2018

timokau commented Aug 20, 2018

timokau commented Oct 3, 2018

jbaum98 commented Oct 4, 2018

timokau commented Oct 4, 2018

timokau commented Oct 4, 2018

mmahut commented Aug 25, 2019

jbaum98 commented Aug 25, 2019

timokau commented Aug 25, 2019

vcunat commented Aug 25, 2019

timokau commented Aug 25, 2019

jbaum98 commented Aug 26, 2019

jbaum98 commented Sep 6, 2019

vcunat commented Sep 8, 2019

vcunat commented Sep 8, 2019

vcunat commented Sep 8, 2019

timokau commented Sep 20, 2019

vcunat commented Sep 21, 2019

timokau commented Sep 21, 2019

timokau commented Sep 25, 2019

[WIP] sage: Add darwin support #45364

[WIP] sage: Add darwin support #45364

Conversation

jbaum98 commented Aug 20, 2018

Motivation for this change

Things done

timokau commented Aug 20, 2018

timokau Aug 20, 2018

Choose a reason for hiding this comment

jbaum98 Aug 20, 2018

Choose a reason for hiding this comment

symphorien Aug 20, 2018

Choose a reason for hiding this comment

symphorien Sep 6, 2019 • edited

Choose a reason for hiding this comment

jbaum98 commented Aug 20, 2018

timokau commented Aug 20, 2018

timokau commented Aug 20, 2018

timokau commented Oct 3, 2018

jbaum98 commented Oct 4, 2018

timokau commented Oct 4, 2018

timokau commented Oct 4, 2018

mmahut commented Aug 25, 2019

jbaum98 commented Aug 25, 2019

timokau commented Aug 25, 2019

vcunat commented Aug 25, 2019

timokau commented Aug 25, 2019

jbaum98 commented Aug 26, 2019

jbaum98 commented Sep 6, 2019

vcunat commented Sep 8, 2019

vcunat commented Sep 8, 2019

vcunat commented Sep 8, 2019

timokau commented Sep 20, 2019

vcunat commented Sep 21, 2019

timokau commented Sep 21, 2019

timokau commented Sep 25, 2019

symphorien Sep 6, 2019 •

edited