graalvm-ce-20.2.0 -> 20.3.0 + darwin support. #105815
Conversation
ofborg
bot
added
6.topic: darwin
|
Result of 2 packages failed to build:
2 packages built:
babashka |
|
Fixed the babashka build, but clj-kondo still fails: /cc @borkdude |
|
@stevana Babashka only compiles with 20.2.0 at this moment, due to a dependency on svm.jar for version 20.2.0. The next version of babashka will use 20.3.0. See https://github.com/borkdude/babashka/blob/master/doc/build.md |
|
@borkdude Babashka seems to compile fine with 20.3.0 (checkout this branch, and do |
|
@stevana Same thing: https://github.com/borkdude/clj-kondo/blob/master/doc/build.md Clj-kondo has a hard dependency on |
|
Thanks for the pointers! It seems to me that specific dependencies cannot be overridden on the nix level, because the nix package simply pulls a clj-kondo uberjar from github releases and feeds that to native-image. I think I'll leave this to the clj-kondo nix package maintainers to figure out. Worst case we can just wait until clj-kondo supports 20.3.0. |
|
I will bump babashka and clj-kondo both to 20.3.0 before the end of the year (at least that's the plan). |
|
Cool, no rush from my side. By the way, in case you didn't see it already, the following one-line change is needed for babashka to work with 20.3.0: d255424 . |
|
@stevana Thanks. Yes, that change is also in my 20.3.0 branch. |
|
I welcome darwin support and I've been the person that upgraded clj-kondo and babashka the most this year. babashka can be version bumped from my side (on graalvm-ce-20.2) before this is merged, but clj-kondo v2020.12.12 fails. So it appears that there is a dependency here from clj-kondo and in a minor way for babashka (due to the one-line removal). Having two PRs open and having it depend on it one another is very messy. |
|
clj-kondo v2020.12.12 uses graalvm-ce-20.3.0 (as opposed to the previous version which used graalvm-ce-20.2.0). |
|
Both clj-kondo and babashka newest are now compiled using GraalVM 20.3.0. |
|
This is my first nixpkgs PR, so I don't know what the right protocol is neither. The fact that the same code can compile with version V of GraalVM and not compile on version V+1 is garbage. I see two options:
Both are ugly, but then so is GraalVM, so what can you do? :-) My guess is that the first option is the least work for everybody involved, is it "allowed" from a nixpkgs PR perspective though? Or are there other options (assuming three dependent PRs is out of question)? |
|
@stevana Note that this is not entirely GraalVM's fault. clj-kondo and babashka depend on a specific library which is pinned to a GraalVM version to work around a Java 11 problem in Clojure's Reflector class. https://github.com/borkdude/clj-reflector-graal-java11-fix This causes the main incompatibility problem with other GraalVM versions. |
|
So overriding that library to the GraalVM version you want to use, should solve the main problem. But this will require you to make your own uberjars and not depend on the one that is distributed on Github releases. |
|
For reproduction purposes I never went the "build the package in nixos" because you need to declare every single dependency in Nix. That would open a whole new can of worms as Hydra won't allow to access the internet to grab dependencies. (Although clojars and maven should be fine for a nixpkgs translation, because they are append-only as far as I know.) |
|
@bennyandresen In case you wanna try going down that route, the following worked for me in another project and could probably be adopted for babashaka/clj-kondo: { sources ? import ./nix/sources.nix
, pkgs ? import sources.nixpkgs {}
}:
with pkgs;
let
mvn2nix = import (fetchTarball https://github.com/fzakaria/mvn2nix/archive/master.tar.gz) {};
mavenRepository =
mvn2nix.buildMavenRepositoryFromLockFile { file = ./mvn2nix-lock.json; };
inherit (import sources.gitignore {}) gitignoreSource;
in stdenv.mkDerivation rec {
pname = "...";
version = "latest";
name = "${pname}-${version}";
src = gitignoreSource ./.;
buildInputs = [ clojure jdk11_headless graalvm11-ce ];
buildPhase = ''
export CLASSPATH=$(find ${mavenRepository} -name "*.jar" -printf ':%h/%f')
export builddir=$TMP/classes
mkdir -p $builddir
echo "compiling lock fix workaround"
javac java/src/lockfix/LockFix.java -cp $CLASSPATH -d $builddir
echo "compiling clojure sources"
# On Darwin `clj` tries to create some folder in the home directory...
${lib.optionalString stdenv.isDarwin ''
export HOME=$TMP/home
mkdir -p $HOME
''}
clj -Scp src:$CLASSPATH:$builddir \
-J-Djava.awt.headless=true \
-J-Dclojure.compile.path=$builddir \
-M -e "(compile (quote ${pname}.core))"
echo "creating manifest file"
echo "Main-Class: ${pname}.core" > manifest.txt
echo "Class-Path: ." >> manifest.txt
find ${mavenRepository} -name '*.jar' -printf ' %h/%f\n' >> manifest.txt
cat manifest.txt
echo "creating fat/uber jar"
jar cvfm ${name}.jar manifest.txt -C $builddir .
echo "compiling native image"
native-image \
-jar ${name}.jar \
-H:Name=${pname} \
${lib.optionalString stdenv.isDarwin ''-H:-CheckToolchain''} \
-H:+ReportExceptionStackTraces \
-J-Dclojure.spec.skip-macros=true \
-J-Dclojure.compiler.direct-linking=true \
-J-Djava.awt.headless=true \
-J-Dclojure.tools.logging.factory=clojure.tools.logging.impl/slf4j-factory \
--initialize-at-build-time \
--initialize-at-run-time=sun.font.SunFontManager \
--initialize-at-run-time=sun.font.StrikeCache \
--initialize-at-run-time=sun.font.SunLayoutEngine \
--initialize-at-run-time=sun.font.FontManagerNativeLibrary \
--initialize-at-run-time=sun.awt.X11GraphicsConfig \
--initialize-at-run-time=javax.imageio.ImageTypeSpecifier \
--initialize-at-run-time=sun.java2d.SurfaceData \
--initialize-at-run-time='com.sun.imageio.plugins.jpeg.JPEG$JCS' \
--initialize-at-run-time='sun.awt.dnd.SunDropTargetContextPeer$EventDispatcher' \
--report-unsupported-elements-at-runtime \
--allow-incomplete-classpath \
--verbose \
--no-fallback \
--no-server
'';
installPhase = ''
mkdir -p $out/bin
install -Dm755 ${pname} $out/bin/${pname}
'';
}Before you build (and after any dependency change), you need to run: clj -Spom
nix run -f https://github.com/fzakaria/mvn2nix/archive/master.tar.gz \
--command mvn2nix --repositories=https://repo1.maven.org/maven2/ https://repo.clojars.org > mvn2nix-lock.jsonto pin the dependencies in nix. |
|
|
|
@taku0 Thanks for testing! Could you please have another look to see if my last commits fixed the problem on MacOS? |
| $out/bin/node -e 'console.log(1 + 1)' | ||
|
|
||
| echo '1 + 1' | $out/bin/graalpython | ||
| echo '1 + 1' | $out/bin/irb |
There was a problem hiding this comment.
Still having Could not find OpenSSL headers, install via Homebrew or MacPorts or set OPENSSL_PREFIX.
OPENSSL_PREFIX=${openssl} bash $out/languages/ruby/lib/truffle/post_install_hook.sh may resolve this, but it results in <internal:core> core/kernel.rb:260:in `require': Library "/usr/lib/libSystem.B.dylib" does not exist. (RuntimeError). I have no idea how to fix this.
Linux version also have a problem. bin/ruby -e 'require "openssl"' results in languages/ruby/lib/mri/openssl.so:in `<LoadModulesNode>': libtruffleruby.so: cannot open shared object file: No such file or directory (NFIUnsatisfiedLinkError) (RuntimeError)
There was a problem hiding this comment.
languages/ruby/lib/mri/openssl.so requires libssl.so.10 and libcrypto.so.10 but we only have libssl.so.1.1 and libcrypto.so.1.1. Neither patchelf --set-rpath ${openssl_1_0_2} nor patchelf --replace-needed libssl.so.10 libssl.so.1.1 --replace-needed libcrypto.so.10 libcrypto.so.1.1 --set-rpath ${openssl} work. It checks OpenSSL version at runtime. The standard way to fix this is using languages/ruby/lib/truffle/post_install_hook.sh, but it invokes languages/llvm/native/bin/clang and it not works. I have no idea how to fix this.
There was a problem hiding this comment.
I commented out the irb check and added a TODO comment about it not working.
(I'm not using ruby and I suspect that this didn't work in graalvm-ce-20.2.0 either, so I guess this PR doesn't introduce a regression.)
|
BTW 21.0.0 is released, maybe we should bump it again? |
|
Fixed conflicts, lets bump to 21.0.0 in a separate PR? (This will break babashka and clj-kondo again.) |
|
@SuperSandro2000 can you merge this? |
|
Gentle ping @SuperSandro2000, could you help us here? :) |
|
I just manually change the version to 21.0.0 and it worked like a charm, it'd be really good if we could merge this to then open another PR to the 21.0.0 |
|
Lets ignore the remaining darwin build failures for now. I would merge it if I could squash the build history or the commits would fit the contributing guide. |
|
@stevana could you squash the commits into a single one? |
stevana
force-pushed
the
graalmv-20.3.0-darwin
branch
from
6cd99fd
to
d89f153
Compare
|
Squashed. |
| propagatedBuildInputs = [ setJavaClassPath zlib ] ++ | ||
| # On Darwin native-image calls clang and it | ||
| # tries to include <Foundation/Foundation.h>, | ||
| # and Interactive Ruby (irb) requires OpenSSL | ||
| # headers. | ||
| lib.optionals stdenv.hostPlatform.isDarwin | ||
| [ darwin.apple_sdk.frameworks.Foundation openssl ]; |
There was a problem hiding this comment.
| propagatedBuildInputs = [ setJavaClassPath zlib ] ++ | |
| # On Darwin native-image calls clang and it | |
| # tries to include <Foundation/Foundation.h>, | |
| # and Interactive Ruby (irb) requires OpenSSL | |
| # headers. | |
| lib.optionals stdenv.hostPlatform.isDarwin | |
| [ darwin.apple_sdk.frameworks.Foundation openssl ]; | |
| propagatedBuildInputs = [ setJavaClassPath zlib ] ++ | |
| # On Darwin native-image calls clang and it | |
| # tries to include <Foundation/Foundation.h>, | |
| # and Interactive Ruby (irb) requires OpenSSL | |
| # headers. | |
| lib.optionals stdenv.hostPlatform.isDarwin | |
| [ Foundation openssl ]; |
| @@ -1,48 +1,60 @@ | |||
| { lib, stdenv, fetchurl, perl, unzip, glibc, zlib, setJavaClassPath }: | |||
| { lib, stdenv, fetchurl, perl, unzip, glibc, zlib, setJavaClassPath, darwin, openssl }: | |||
There was a problem hiding this comment.
| { lib, stdenv, fetchurl, perl, unzip, glibc, zlib, setJavaClassPath, darwin, openssl }: | |
| { lib, stdenv, fetchurl, perl, unzip, glibc, zlib, setJavaClassPath, Foundation, openssl }: |
Please inherit Foundation from in top level. inherit (darwin.apple_sdk.frameworks) Foundation
There was a problem hiding this comment.
I don't understand your suggestion. How can I inherit (darwin.apple_sdk.frameworks) Foundation when darwin isn't in scope?
There was a problem hiding this comment.
I mean something like this https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix#L1956.
There was a problem hiding this comment.
I see, thanks, fixed!
stevana
force-pushed
the
graalmv-20.3.0-darwin
branch
from
d89f153
to
069e1f4
Compare
|
@GrahamcOfBorg build graalvm8-ce graalvm11-ce babashka clj-kondo |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
(Also bumps babashka and clj-kondo.)
stevana
force-pushed
the
graalmv-20.3.0-darwin
branch
from
069e1f4
to
582fa3d
Compare
|
This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch). Result of 4 packages built:
|
|
This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch). Result of 4 packages built:
|
Motivation for this change
Bump version to the latest release and add Darwin support (used to be Linux only).
Things done
sandboxinnix.confon non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"./result/bin/)nix path-info -Sbefore and after)