Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Julia: reinit at latest #98043

Closed
wants to merge 2 commits into from
Closed

Conversation

doronbehar
Copy link
Contributor

Make build pass

Motivation for this change

Close #93333 and Fix #91930 .

Things done

Using ofborg to test the builds.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@doronbehar
Copy link
Contributor Author

@GrahamcOfBorg build julia

@doronbehar
Copy link
Contributor Author

Build timeout ... I guess there's no choice other then failing locally.

@7c6f434c
Copy link
Member

So the plan is not to have LTS?

@garrison
Copy link
Member

So the plan is not to have LTS?

I think maybe removing it was unintentional? @doronbehar said elsewhere:

I'm experimenting a more broad change to the situation at #98043 . There, only the derivations julia_lts and julia will be left, and hopefully they will build at some point, with most of our deps used instead of upstream's.

@7c6f434c
Copy link
Member

I worded my question based on the assumption that between words and code, the latter is the authoritative source — maybe I have just underestimated the WIP part, of course.

(But it is a question either way, not a statement, of course)

@garrison
Copy link
Member

In any case, I think that julia 1.0 ought to be included, and I would also suggest that it be called julia_10 or julia_10_lts rather than julia_lts, as people may wish to continue using it even after a new LTS release has been declared, and ideally one package will not upgrade to another without a human explicitly changing a derivation. Says the blog post on Julia's Release Process:

Fortunately, even [a new LTS branch] does not force people using Julia 1.0.x to upgrade: they can keep using the last 1.0.x version and packages that are compatible with it. At that point it will be the most stable, thoroughly tested, patched version of Julia in existence, so it will be safe to keep using indefinitely if one doesn't need newer features.

@doronbehar
Copy link
Contributor Author

Oh please folks, I didn't intent to discard the LTS, it's just that it's expression wasn't ready yet and I wanted to get CI happy until 1.5.1 was working, though now it doesn't matter as I'm testing the build locally.

@doronbehar
Copy link
Contributor Author

Also, I wouldn't lend high hopes in the work here. I'm getting weird "file not found" errors during build, when trying to use as many external deps as possible. I'm using direnv and lorri and this .envrc and shell.nix put in where the tarball was extracted:

# .envrc
eval "$(lorri direnv)"
PATH=$PATH:$CMAKE_BIN
out=$PWD/nix-out
# for the "out command":
PATH=$PATH:$PWD/nix-out-cmd-bin

Where the file $PWD/nix-out-cmd-bin/out has:

#!/bin/sh
echo $out

And shell.nix has:

with import /home/doron/repos/nixpkgs {};

mkShell {
  inherit (julia) buildInputs nativeBuildInputs makeFlags;
  CMAKE_BIN="${cmake}/bin";
}

I'm launching make with eval make ${makeFlags[@]}. The persistent error is:

cat: /home/doron/downloads/julia-1.5.1/usr/manifest/suitesparse-wrapper: No such file or directory
WARNING: using mismatched version for :
  want

Trying building with their suitesparse or ours, and using their blas and lapack didn't help.

libssh2
curl
libgit2
patchelf
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

patchelf should go into nativeBuildInputs. And why is it needed in a build from source?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lucus16
Copy link
Contributor

Lucus16 commented Sep 19, 2020

I fixed 1.0.4 over at https://github.com/Lucus16/nixpkgs/tree/fix-julia

I don't know how you want to fit it in so I'll leave that to you.

@doronbehar
Copy link
Contributor Author

@Lucus16 It would be great to have a working expression for 1.0.4 in this PR. You can open a PR against my branch pkg/julia/latest which this PR originates from.

@Lucus16
Copy link
Contributor

Lucus16 commented Sep 21, 2020

@doronbehar Here you go: doronbehar#1

@tbenst
Copy link
Contributor

tbenst commented Sep 29, 2020

idk if this is helpful or not, but the build does succeed using the derivation from 1.3 (lightly modified), but 22 tests Fail (although 3853097 tests pass :): https://gist.github.com/tbenst/495998f0b7e1bfbac1c75ac2155ac528

Edit: should have specified that this is on 20.09

@doronbehar
Copy link
Contributor Author

I've incorporated Guix' method for Julia where the patches are applied on our derivations via overrides, and USE_SYSTEM_ variables are set to 1. I got the sha256 of all patches and outdated versions Julia supposedly needs, but I haven't give it a full try. The overrided gmp, libuv and libunwind build, but I haven't checked if llvm builds, and if Julia builds with them.

@FlorianFranzen
Copy link
Contributor

@doronbehar Let me know if you need a hand. I would really like to have Julia back in nixpkgs.

Comment on lines +103 to +110
for i in backtrace cmdlineargs; do
mv test/$i.jl{,.off}
touch test/$i.jl
done
rm stdlib/Sockets/test/runtests.jl && touch stdlib/Sockets/test/runtests.jl
rm stdlib/Distributed/test/runtests.jl && touch stdlib/Distributed/test/runtests.jl
sed -e 's/Invalid Content-Type:/invalid Content-Type:/g' -i ./stdlib/LibGit2/test/libgit2.jl
sed -e 's/Failed to resolve /failed to resolve /g' -i ./stdlib/LibGit2/test/libgit2.jl
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we're removing a good number of tests here. Why is that necessary? Are these known failures that we have existing issues for? If so it would be good to mention them here.

rev = version;
sha256 = "040l7f1hk7xyza11sry5cj4fhw05na949865axqqhxnifdvnmfji";
};
# Tests fail with this branch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an upstream issue for this?

@doronbehar
Copy link
Contributor Author

@doronbehar Let me know if you need a hand. I would really like to have Julia back in nixpkgs.

You can help by try and building it as is. I got lazy to do it on my notebook laptop, as it needs building a custom version of llvm_7. If the llvm build succeeds, I'd also appreciate if you'd setup cachix and share the llvm build with me.

cp "${dsfmt}" "./deps/srccache/dsfmt-${dsfmtVersion}.tar.gz"
cp "${rmath-julia}" "./deps/srccache/Rmath-julia-${rmathVersion}.tar.gz"
cp "${libuv}" "./deps/srccache/libuv-${libuvVersion}.tar.gz"
cp "${virtualenv}" "./deps/srccache/virtualenv-${virtualenvVersion}.tar.gz"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need virtualenv? I can't think of any good reason. In the past it was used in the build system I think but that is long ago.

@matklad
Copy link
Member

matklad commented Oct 28, 2020

I've tried this today rebased on top of 5d0e2de, and got the following build error:

01:38:51|~/config|master⚡*
λ nix-env -f /home/matklad/projects/nixpkgs -iA julia
installing 'julia-1.5.2'
these derivations will be built:
  /nix/store/w8z1i4130m7jhqnllzgpfbpbspq91k87-julia-1.5.2.drv
building '/nix/store/w8z1i4130m7jhqnllzgpfbpbspq91k87-julia-1.5.2.drv'...
unpacking sources
unpacking source archive /nix/store/01m42pwxzzd188jbbixvjz2ywpf6hsbq-julia-1.5.2-full.tar.gz
source root is julia-1.5.2
setting SOURCE_DATE_EPOCH to timestamp 1600924982 of file julia-1.5.2/deps/srccache/objconv.zip
patching sources
configuring
no configure script, doing nothing
building
build flags: SHELL=/nix/store/k8p54jg8ipvnfz435mayf5bnqhw4qqap-bash-4.4-p23/bin/bash USE_BINARYBUILDER=0 PREFIX=\$\(out\) prefix=\$\(out\) USE_SYSTEM_LIBUNWIND=1 USE_SYSTEM_PCRE=1 PCRE_CONFIG=/nix/store/vcxly22i1csi2xbjyjvsn6vk19cf5lnw-pcre2-10.35-dev/bin/pcre2-config PCRE_INCL_PATH=/nix/store/vcxly22i1csi2xbjyjvsn6vk19cf5lnw-pcre2-10.35-dev/include/pcre2.h USE_SYSTEM_DSFMT=0 USE_SYSTEM_BLAS=1 USE_SYSTEM_LAPACK=1 USE_SYSTEM_GMP=1 USE_SYSTEM_LIBUV=1 USE_SYSTEM_LLVM=1 USE_SYSTEM_MPFR=1 USE_SYSTEM_SUITESPARSE=1 USE_SYSTEM_UTF8PROC=1 USE_SYSTEM_MBEDTLS=1 USE_SYSTEM_LIBSSH2=1 USE_SYSTEM_CURL=1 USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_PATCHELF=1 USE_SYSTEM_ZLIB=1 USE_SYSTEM_P7ZIP=1 USE_SYSTEM_OPENLIBM=1
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
rm: cannot remove 'scratch/dsfmt-2.2.3/': No such file or directory
patching file dSFMT.h
Hunk #2 succeeded at 181 (offset 2 lines).
Hunk #3 succeeded at 202 (offset 2 lines).
Hunk #4 succeeded at 220 (offset 2 lines).
Hunk #5 succeeded at 244 (offset 2 lines).
Hunk #6 succeeded at 265 (offset 2 lines).
Hunk #7 succeeded at 284 (offset 2 lines).
Hunk #8 succeeded at 295 (offset 2 lines).
Hunk #9 succeeded at 307 (offset 2 lines).
Hunk #10 succeeded at 318 (offset 2 lines).
Hunk #11 succeeded at 330 (offset 2 lines).
Hunk #12 succeeded at 341 (offset 2 lines).
Hunk #13 succeeded at 353 (offset 2 lines).
Hunk #14 succeeded at 376 (offset 2 lines).
Hunk #15 succeeded at 391 (offset 2 lines).
Hunk #16 succeeded at 407 (offset 2 lines).
Hunk #17 succeeded at 423 (offset 2 lines).
Hunk #18 succeeded at 439 (offset 2 lines).
Hunk #19 succeeded at 449 (offset 2 lines).
Hunk #20 succeeded at 459 (offset 2 lines).
Hunk #21 succeeded at 470 (offset 2 lines).
Hunk #22 succeeded at 483 (offset 2 lines).
Hunk #23 succeeded at 497 (offset 2 lines).
Hunk #24 succeeded at 511 (offset 2 lines).
Hunk #25 succeeded at 520 (offset 2 lines).
Hunk #26 succeeded at 529 (offset 2 lines).
Hunk #27 succeeded at 539 (offset 2 lines).
Hunk #28 succeeded at 549 (offset 2 lines).
Hunk #29 succeeded at 559 (offset 2 lines).
Hunk #30 succeeded at 569 (offset 2 lines).
Hunk #31 succeeded at 579 (offset 2 lines).
Hunk #32 succeeded at 592 (offset 2 lines).
Hunk #33 succeeded at 605 (offset 2 lines).
Hunk #34 succeeded at 618 (offset 2 lines).
Hunk #35 succeeded at 630 (offset 2 lines).
patching file test.c
patching file dSFMT.c
cat: /build/julia-1.5.2/usr/manifest/suitesparse-wrapper: No such file or directory
WARNING: using mismatched version for :
  want
Creating usr/etc/julia/startup.jl
Copying in usr/share/man/man1/julia.1
/build/julia-1.5.2/contrib/install.sh 755 /build/julia-1.5.2/contrib/julia-config.jl /build/julia-1.5.2/usr/share/julia/
PATH="/nix/store/1krkymlcn5yvg71camz6nckrhssqz5f8-gnum4-1.4.18/bin:/nix/store/z23xkm52b6jdpb098sv9h63rxi8zhnwj-which-2.21/bin:/nix/store/38vxlxrvg3yji3jms44qn94lxdysbj5j-perl-5.32.0/bin:/nix/store/rxjkdqgziqgy71yj7k6j65by8dzp6z84-gfortran-wrapper-9.3.0/bin:/nix/store/h770niqkf08ksgm0n6a271kc21w95b13-binutils-wrapper-2.31.1/bin:/nix/store/hiwz81i1g3fn3p0acjs042a4h5fri6dh-binutils-2.31.1/bin:/nix/store/kbysa4xpmn0013bvplw3fl2i1yydifjc-glibc-2.32-bin/bin:/nix/store/7fdm1ij6f7xwjmhpici1bh3xh08n0sp7-coreutils-8.31/bin:/nix/store/346skv0d24rqnf4npknbp9h5bs14j8zy-python3-3.8.6/bin:/nix/store/r9djcvv9hh6cma006jxlcz3xljz7zck0-cmake-3.18.2/bin:/nix/store/qza56yd923g39sbnn6gn5xh3awrz4lfl-patchelf-0.12/bin:/nix/store/8madhhcxhxlf0rrc5hdsk65l206r3cc6-gcc-wrapper-9.3.0/bin:/nix/store/shs8kax43dk69fh94g5w4bc1ra3nv2j2-gcc-9.3.0/bin:/nix/store/kbysa4xpmn0013bvplw3fl2i1yydifjc-glibc-2.32-bin/bin:/nix/store/7fdm1ij6f7xwjmhpici1bh3xh08n0sp7-coreutils-8.31/bin:/nix/store/vcxly22i1csi2xbjyjvsn6vk19cf5lnw-pcre2-10.35-dev/bin:/nix/store/m2qy16xk9cn5yprqm3a09ik8cxyv3igr-pcre2-10.35-bin/bin:/nix/store/xwa3yfa0zxy5zs914pq9wa3rsj310yn9-mbedtls-2.16.3/bin:/nix/store/bnqwzn3ywfxq2kn4l4s2c68jnbx2ilbg-curl-7.72.0-dev/bin:/nix/store/19k826d182q6a3k2ply1wrsb2knimsz4-nghttp2-1.41.0-bin/bin:/nix/store/8ry6c2r2dl5v9v45dvk7mgxh6416ckn7-libkrb5-1.18-dev/bin:/nix/store/vl9drphppxnxjwkfkr7sijh4d0gvcg3n-libkrb5-1.18/bin:/nix/store/wp81dnxgrpi4lgy20r82kfg9nwrmypkm-openssl-1.1.1g-bin/bin:/nix/store/vcvcpdn0bspcl722qkwp2s72wws9gw7s-curl-7.72.0-bin/bin:/nix/store/qza56yd923g39sbnn6gn5xh3awrz4lfl-patchelf-0.12/bin:/nix/store/jw0gpz7s9y8jbxp7h36pr7sj72bssm45-p7zip-17.01/bin:/nix/store/jqvbvnyl82c0zdiwl12spr2qh9hs1im6-llvm-8.0.1/bin:/nix/store/3gxpdac17sbrxmc7iv3dfsw5pskdllrz-ncurses-6.2-dev/bin:/nix/store/jxk39sgwpfnsfb00dsff5p96mxaj6m33-ncurses-6.2/bin:/nix/store/8ard3kjhyhqbp3ylxa7djfhndqfybbi8-xz-5.2.5-bin/bin:/nix/store/7fdm1ij6f7xwjmhpici1bh3xh08n0sp7-coreutils-8.31/bin:/nix/store/ny7vshs45d5k193ql3b51dak0hjwwqc2-findutils-4.7.0/bin:/nix/store/1avv8qnga55v6kyh9bvv72pf1nwhivv7-diffutils-3.7/bin:/nix/store/98an38bw0fwk6a5mickblc0h07jf00vz-gnused-4.8/bin:/nix/store/4crjhnvd8yhk063r7pwxpw6kd8aq0rbv-gnugrep-3.5/bin:/nix/store/ycf3h904fbai22by992pm9a0iwnwyxhp-gawk-5.1.0/bin:/nix/store/3kg5y8qb1fmq7sv77lj7xy5vldx2m9za-gnutar-1.32/bin:/nix/store/qxybvmiq4xksmzy26ivc0hd61c94bln0-gzip-1.10/bin:/nix/store/71w61r13p1pvmgz16106k0ps0k7xpb6m-bzip2-1.0.6.0.1-bin/bin:/nix/store/vc851g6wnyyykb5jxgbal3whi5495xia-gnumake-4.3/bin:/nix/store/k8p54jg8ipvnfz435mayf5bnqhw4qqap-bash-4.4-p23/bin:/nix/store/p7g6l88dl48fjx41qb628sh72vpv1v0h-patch-2.7.6/bin:/nix/store/8ard3kjhyhqbp3ylxa7djfhndqfybbi8-xz-5.2.5-bin/bin:/build/julia-1.5.2/usr/tools" PATCHELF="patchelf" FC="gfortran -m64" /build/julia-1.5.2/contrib/fixup-libgfortran.sh --verbose /build/julia-1.5.2/usr/lib
Discovered traces of libgfortran within
Got SONAMES of
Warning: git information unavailable; versioning information limited
    PERL base/pcre_h.jl
    PERL base/errno_h.jl
    PERL base/build_h.jl.phony
    PERL base/file_constants.jl
make[1]: *** No rule to make target '/build/julia-1.5.2/usr/include/uv/errno.h', needed by 'uv_constants.jl'.  Stop.
make: *** [Makefile:66: julia-base] Error 2
builder for '/nix/store/w8z1i4130m7jhqnllzgpfbpbspq91k87-julia-1.5.2.drv' failed with exit code 2
error: build of '/nix/store/w8z1i4130m7jhqnllzgpfbpbspq91k87-julia-1.5.2.drv' failed

01:38:56|~/config|master⚡*

@r-burns r-burns mentioned this pull request Oct 29, 2020
10 tasks
@cstich
Copy link
Contributor

cstich commented Oct 29, 2020

Just as feedback, I can successfully build this locally on my machine. I do not really know how to upload anything to cachix but if someone walks me through the steps, then I am happy to do that.

@r-burns
Copy link
Contributor

r-burns commented Oct 29, 2020

@doronbehar I tried building the current merge-head and got the same error as @matklad. I followed the basic setup instructions for cachix and then did a

nix-build . -A julia.buildInputs | cachix push r-burns

I haven't used cachix before but hopefully I did that right and it can be of some assistance

@doronbehar
Copy link
Contributor Author

I followed the basic setup instructions for cachix and then did a

nix-build . -A julia.buildInputs | cachix push r-burns

I haven't used cachix before but hopefully I did that right and it can be of some assistance

Luckily that product is excellently built (:green_heart: @domenkozar ) and I think you did it correctly, so thanks for effort. However I can't verify I got your llvm build. Unfortunately even if it would have worked I don't know how would I go about and fix the issue described in @matklad's comment. I'm also not as motivated as (I think) you are or as I was a while ago.. Sorry.

@doronbehar
Copy link
Contributor Author

The last push is just a rebase and a merge conflict fix.

@doronbehar
Copy link
Contributor Author

Closed in favor of #101933 .

@doronbehar doronbehar closed this Nov 14, 2020
@doronbehar
Copy link
Contributor Author

Oh but there's @Lucus16 commit I forgot here which fixes 10.. I will open a new PR.

@doronbehar doronbehar mentioned this pull request Nov 14, 2020
10 tasks
@doronbehar doronbehar deleted the pkg/julia/latest branch March 2, 2023 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Too many old Julia versions, 1.5 is not packaged, and all versions fail in tests