[WIP] julia 0.7 #41499
[WIP] julia 0.7 #41499
Conversation
|
Julia part seems fine to me. I have not read LLVM patches; maybe @dtzWill @lovek323 @Ralith @Ericson2314 @matthewbauer might have some opinion on them. |
| sha256 = "0s6dik373yvpj8xxmps3wasr6s5wrzbccpfw7vzlclbgxlpyjq0i"; | ||
| }; | ||
|
|
||
| rmathVersion = "0.1"; |
There was a problem hiding this comment.
Rmath isn't needed anymore for Julia 0.7.
| , libunwind, readline, utf8proc, zlib | ||
| , llvm, libffi, ncurses | ||
| # standard library dependencies | ||
| , curl, fftwSinglePrec, fftw, gmp, libgit2, mpfr, openlibm, openspecfun, pcre2 |
There was a problem hiding this comment.
openspecfun isn't needed anymore for Julia 0.7.
| march = { "x86_64" = "x86-64"; "i686" = "pentium4"; }."${arch}" | ||
| or (throw "unsupported architecture: ${arch}"); | ||
| # Julia requires Pentium 4 (SSE2) or better | ||
| cpuTarget = { "x86_64" = "x86-64"; "i686" = "pentium4"; }."${arch}" |
There was a problem hiding this comment.
You don't actually need to set CPU target if it's identical to MARCH. However, it would be a good idea to set it to something more fancy so that several images are built, to enable the use of recent instructions where possible (this matters a lot for scientific work). Upstream uses this:
- x86:
'JULIA_CPU_TARGET="pentium4;sandybridge,-xsaveopt,clone_all" ' - x86_64:
'JULIA_CPU_TARGET="generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)'
There was a problem hiding this comment.
Yeah, but Julia 0.7 is much more robust now, it will always use SSE2 and it will throw an error if compiler options are incorrect. Since the JULIA_CPU_TARGET values I posted come from upstream, they should be safe to use.
There was a problem hiding this comment.
I think it was way before 0.7, though, if there are improvements. we could try being optimistic.
There was a problem hiding this comment.
So like this?
cpuTarget = { "x86_64" = "generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)";
"i686" = "pentium4;sandybridge,-xsaveopt,clone_all"; }."${arch}"(Compiling)
|
|
||
| "USE_SYSTEM_BLAS=1" | ||
| "USE_BLAS64=${if openblas.blas64 then "1" else "0"}" | ||
| "LIBBLAS=-lopenblas" |
There was a problem hiding this comment.
Do you know is this is the threaded OpenBLAS? I'm asking because I remember it wasn't completely obvious which library to use when I created the Fedora package.
There was a problem hiding this comment.
It's compiled with
''PREFIX="''$(out)"''
"NUM_THREADS=64"
"INTERFACE64=${if blas64 then "1" else "0"}"
"NO_STATIC=1"
There was a problem hiding this comment.
I'm not sure but USE_THREAD=1 might be needed too. It's easy to check: if you run a linear algebra operation on a large matrix, more than 1 CPU will be used if threads are enabled.
FWIW, the Fedora package builds three variants of OpenBLAS: one without threads, one with threads, one with OpenMP. https://src.fedoraproject.org/rpms/openblas/blob/master/f/openblas.spec
There was a problem hiding this comment.
@. rand(10000,10000)^11 * rand(10000,10000)
100% on all 4 CPUs
| "USE_SYSTEM_PCRE=1" | ||
| "PCRE_CONFIG=${pcre2.dev}/bin/pcre2-config" | ||
| "PCRE_INCL_PATH=${pcre2.dev}/include/pcre2.h" | ||
| "USE_SYSTEM_READLINE=1" |
|
@nalimilan Thanks, compiling. I applied all of those except the cpu thing for now |
|
Things done:
Compiled with tests and run binaries Passing version info (waiting for answer on c548814) |
|
Thanks for tackling this! The Linux version builds and starts fine, on osx 10.13.5 I got the following: If this is still expected please feel free to ping me once/if you need osx testers. |
|
@rened thanks for trying it out! (un)fortunately I do not own a mac so I cannot test it. However I'm just applying julia patches, can you try and verify llvm-6.0 alone (ie unpatched) does work? Thanks |
|
@nico202 as far as I can see that worked, yes |
|
@nico202 building julia now with the review changes included, let's see |
|
It shouln't change since I've not touched llvm. I've looked at the patches but nothing seems to relate to python/sysctl. You could try disabling all the pachets (from file all-packags.nix line 6682 on), compile llmv6, if it works enable one more and check again. |
|
will do! |
|
@nico202 I simply had to disable The build worked and julia starts! |
|
Good to know, thanks! I removed the override. I replaced patches with fetchpatch that is the suggested method (it's ugly to have all of them in all-packages.nix, any suggestion?) I also updated julia-git to actually be the latest git version as of now. It's compiling fine too edit: removed version 0.5 |
|
Great! Testing all three versions on osx and Ubuntu |
|
(the git version is probably failing. 0.6 should be upgraded to 0.6.3 in theory but it's not my prio right now) |
|
The hash of one of the patches seems to be wrong: |
|
Hitting the next hash in 87d233c: |
|
Suggestion, shouldn't we have |
|
sure. Done and fixed hashes. nix-env -f . -i julia # -> julia 0.6.2 I'll re-build all of them on NixOS (will take a lot), If you can try again on mac+ubuntu we'll have a complete test and I could then squash everything |
|
All three builds succeed on darwin and Ubuntu - thanks again for your efforts, this is great! |
|
0.7.0-beta is out, I'm building, tomorrow I'll push the new commit |
|
@nico202, any progress? beta2 is out, rc1 should come today/tomorrow - shall we rerun the tests and commit? (cc @flicaflow) |
|
I upgraded to beta2, I can push it. I'll leave for vacation tomorrow so I wont update to rc1 before 2 weeks |
|
No idea of what's happening. There's here the alpha -> beta upgrade, https://github.com/nico202/nixpkgs/tree/julia07, not tested after the rebase on master though. |
|
I did definitely not merge this 😕 |
|
@nico202 I will test it, but could you please use |
|
@dotlambda I think there was a branch reset, and Github has some detection of manual merges, and their code detects every situation where you push a commit that is a descendant of all commits currently in the PR branch. If PR branch is reset to master, GitHub considers any push to master a merge of the PR. |
Motivation for this change
Julia 0.7 is going to be relased soon (R). Since it will bring Pkg3, it will be easier to be able to integrate the package management in nixos. With this attempt I hope to bring julia 0.7-alpha to a good PR state so that upgrading to the released version will be easy. So please tell me how to fix the PR to a decent state, then I'll squash things waiting for the day 0.7 will be officially released
Things done
I upgraded the used llvm to 6.0 (suggested version). I'm applying llvm patches as here, execept one that is needed for mingw32, and 2 that are breaking llvm unit tests. If anybody knows why is welcome.
In julia tests are working, except for 2 that requires network connection (but network presence is checked with getipaddr() that seems to work during tests, even if there's no network). The only other failing test is Distributed but I have to figure out why (it depends on Socket maybe is related to the network).
In this PR i'd also like to drop old julia versions (except for 0.6.2, that must be upgraded to 0.6.3).
I think that patches should be fetched with fetchPatch, but I want a feedback before start messing (compiling it takes like forever so I prefer to do changes only when I'm sure to do the right thing)
sandboxinnix.confon non-NixOS)nix-shell -p nox --run "nox-review wip"./result/bin/)