openjdk: enable bootstrapping on ARM #65247

lopsided98 · 2019-07-22T02:04:11Z

Motivation for this change

I started out with the goal of adding support for 32-bit ARM to adoptopenjdk-bin, but then quickly realized that this binary package could be used to bootstrap the OpenJDK build on these platforms. This led to me also making a lot of changes to clean up the OpenJDK packages.

Things done

This PR makes openjdk8 and openjdk11 use adoptopenjdk-bin for bootstrapping. AdoptOpenJDK is a well-known organization that provides prebuilt OpenJDK binaries for many architectures. Their build scripts are open source and their infrastructure is publicly visible, which I believe makes Nix's OpenJDK bootstrapping process more transparent and trustworthy than the status quo, which uses a bootstrapping tarball of undocumented origin. From a technical standpoint, AdoptOpenJDK binaries make it easier to bootstrap OpenJDK on more platforms.

While I was working on this, I tried to clean up parts of the packages that I believe are no longer necessary or could be done better, but in doing so I there is the possibility that I broke something. Are there any particularly problematic packages I should test?

Testing

	`x86_64-linux`	`i686-linux`	`aarch64-linux`	`armv6l-linux`	`armv7l-linux`
`openjdk8`	✔️	✔️	✔️	🚫	🚫
`openjdk8_headless`	✔️	✔️	✔️	🚫	🚫
`openjdk11`	✔️	❌¹	✔️	✔️	✔️
`openjdk11_headless`	✔️	✔️	✔️	✔️	✔️
`bazel`	✔️		✔️
`jmeter`	✔️
`jetbrains.idea-community`²	✔️
`javaPackages`	✔️

✔️ = builds and runs
❌ = fails to build
🚫 = not supported
blank = not tested

¹openjfx11 (and therefore openjdk11) fails to build on i686-linux for non-obvious reasons, even without this PR, and I did not try to fix it.

²Overridden to use OpenJDK 11 built from source

I do not have the ability to test on macOS, but I did not really touch anything Darwin related.

lopsided98 · 2019-07-22T02:33:18Z

Some other notes:

openjfx11 does not support ARM, but Debian has some patches that add support. I spent a little time trying to integrate them, but gave up because my aarch64 machines don't have enough RAM and there were strange errors on 32-bit ARM.

OpenJDK 8 does not directly support ARM, but there is an upstream branch (which is used by AdoptOpenJDK for their builds) which supports aarch64. Again, Debian has some patches that make it work on 32-bit ARM, but I did not try to integrate them.

lheckemann · 2019-07-22T05:59:35Z

gave up because my aarch64 machines don't have enough RAM

Are you aware of the community builder? https://github.com/nix-community/aarch64-build-box

lopsided98 · 2019-07-22T11:54:22Z

Yes, I thought about trying the community builder, but I was getting kind of tired of working on this PR and I didn't really like the idea of cluttering nixpkgs with a bunch of patches for a feature that likely gets little use.

timokau

LGTM after some superficial review. I don't know anything about jdk bootstrapping, so I can't greenlight this on my own in good conscience. But this looks like a great improvement, thank you for this!

Some nitpicks included.

pkgs/development/compilers/adoptopenjdk-bin/generate-sources.py

timokau · 2019-08-07T15:04:24Z

pkgs/development/compilers/adoptopenjdk-bin/jdk-linux-base.nix

-    rm $out/lib/libfreetype.so
-
-    # for backward compatibility
-    ln -s $out $out/jre


Do we not need this backwards compatibility any more?

I couldn't figure out what the backwards compatibility concern was here, since openjdk11 does not have this symlink. Also, users are unlikely to use adoptopenjdk directly anymore, so I felt that backwards compatibility was less important.

Maybe @taku0 who added the line initially has an opinion?

Oracle JDK and other JDKs had jre folder and some packages depend on it. I forgot which packages depend on it.

If openjdk11 doesn't have it, and no one's complaining, then I don't think adoptopenjdk needs it either.

pkgs/development/compilers/adoptopenjdk-bin/jdk-linux-base.nix

timokau · 2019-08-07T15:17:32Z

CC @taku0 who init'ed adoptopenjdk and may be able to provide a more useful review. Also @bennofs, @tomfitzhenry, @wkennington, @NeQuissimus, @thoughtpolice as a semi-random sample of people intereseted in (adopt-)openjdk according to git shortlog. Hopefully we'll be able to keep this from bitrotting too badly :)

taku0 · 2019-08-10T07:16:17Z

Running a simple AWT application results in NullPointerException:

Caused by: java.lang.NullPointerException
        at java.desktop/sun.awt.FontConfiguration.getVersion(Unknown Source)
        at java.desktop/sun.awt.FontConfiguration.readFontConfigFile(Unknown Source)
        at java.desktop/sun.awt.FontConfiguration.init(Unknown Source)
        at java.desktop/sun.awt.X11FontManager.createFontConfiguration(Unknown Source)
        at java.desktop/sun.font.SunFontManager$2.run(Unknown Source)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.desktop/sun.font.SunFontManager.<init>(Unknown Source)
        at java.desktop/sun.awt.FcFontManager.<init>(Unknown Source)
        at java.desktop/sun.awt.X11FontManager.<init>(Unknown Source)
        ... 73 more

May or may not related to adoptium/temurin-build#693.

Reverting autoPatchelf to manual patchelf and adding fontconfig to the buildInput fixes this (I don't know why).
https://gist.github.com/taku0/74fe5aba9e9230b363dc8a19fc7c07f1

lopsided98 · 2019-08-11T05:24:37Z

I fixed the fontconfig issue. There are a few other libraries that could use a similar fix (e.g. gtk3), but they shouldn't cause any crashes, just missing features such as native look and feel. Now that we have normal OpenJDK builds for all platforms that AdoptOpenJDK supports, very few users will need to use AdoptOpenJDK, so these features will probably not be missed.

I have tested with the example @taku0 posted, as well with some JavaFX examples I found here: https://github.com/callicoder/javafx-examples

grahamc · 2019-08-12T11:47:14Z

@GrahamcOfBorg eval

timokau · 2019-08-14T10:24:23Z

@lopsided98 would you mind rebasing this on current master? I don't know why ofBorg complains about a chromium trace, I can only assume it was an issue with the version of master this PR was branched off of.

lopsided98 · 2019-08-15T02:39:49Z

I rebased it, but it didn't fix anything. I think ofborg merges the PR into master before testing, so it shouldn't matter what commit it is based on.

timokau · 2019-08-15T09:36:03Z

Okay looks like this line is causing the trace:

nixpkgs/pkgs/applications/networking/browsers/chromium/common.nix

Line 153 in 07cf130

] ++ optionals (!stdenv.cc.isClang && (versionRange "71" "72")) [

Which complains because the current oldest chromium version in nixpkgs is 76, so there is no point in keeping that old patch.

What I don't understand is why the trace only appears on this PR. It should pop up every time you eval chromium on a non-clang machine (e.g. x86 linux). What would make more sense is the patch below that, which is aarch64 specific. Aarch64 chromium probably wasn't eval'd by ofBorg before because the jdk was unfree and the aarch64 specific patch is also for an outdated version. But that is not the one the trace complains about.

@lopsided98 maybe just remove both outdated patches?

(CC chromium maintainers @bendlas @ivan)

timokau · 2019-08-16T13:20:10Z

I would argue that other chromium changes don't need to be included in this PR, removing the patches was only necessary to appease ofBorg. Any other cleanup can be done in followup PRs.

Any objections to merging this as-is? @taku0 I'd especially appreciate a final comment from you, since you did a lot of the prior work and had some comments that are (I think) resolved.

ivan · 2019-08-16T13:37:06Z

I am a nixpkgs chromium maintainer and that chromium change is fine with me.

timokau · 2019-08-16T13:48:00Z

I just noticed this targets master. It should probably target staging instead, both for the relatively high rebuild count and the additional testing.

taku0 · 2019-08-16T14:13:10Z

AdoptOpenJDK seems to be fine.

This allows OpenJDK 11 to build for armv6l, armv7l and aarch64, and OpenJDK 8 to build for aarch64.

One of these patches was causing a warning message, which broke ofborg evaluation.

timokau · 2019-08-19T23:19:25Z

@volth for what it's worth, I built jvmci8 on x86 locally. Not sure if I tested it on aarch64. I suspect ofBorg will time out.

lopsided98 · 2019-08-19T23:22:49Z

I haven't really addressed the valgrind-light availability check, because I don't know that the proper solution is. Other than that I think this is ready.

timokau · 2019-08-20T10:43:36Z

@matthewbauer @volth objections to merge?

timokau · 2019-08-20T21:55:37Z

graalvm builds on x86 (while eating a TON of ram, wow). Thanks again @lopsided98 for taking on this PR.

FRidh · 2019-08-31T10:54:46Z

This broke the tarball job on staging. Please fix because it would block the next staging -> staging-next merge.
https://hydra.nixos.org/build/98995742

timokau · 2019-08-31T14:45:11Z

Apparently for some reason openjdk11.meta.license cannot be found (no attribute license). Not sure why, but a hotfix would probably be to specify the license in jfx11 explicitly. I don't currently have time to make a PR, will you take this on @lopsided98?

lopsided98 · 2019-08-31T16:03:48Z

openjdk11 on macOS is a Zulu JDK build (https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/openjdk/darwin/11.nix) which does not have a license attribute. It's probably less error prone to just give OpenJFX its own license attribute, but I'll add the license to the macOS JDK as well.

On a related note, it would probably make sense to use AdoptOpenJDK instead of Zulu on macOS, but I don't have any macOS machines so I can't do it myself.

lopsided98 · 2019-08-31T16:16:49Z

See #67839

timokau · 2019-09-01T15:35:46Z

@grahamc shouldn't ofBorg have caught this? Its essentially an eval failure on mac, isn't it?

vcunat · 2019-09-08T14:09:00Z

Any idea why this MR introduced this channel-blocking breakage? #68307

fgaz · 2019-09-10T17:17:28Z

I think this PR broke all gradle-based packages, specifically commit de5e65a (found by bisecting).

I am getting this error:

> Task :PD-classes:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':PD-classes:compileJava'.
> Could not find tools.jar. Please check that /nix/store/6l52g0dy4v2wrxjwrphcsl7wzf3brb9w-openjdk-8u212-ga-jre/lib/openjdk/jre contains a valid JDK installation.

For some reason gradle seems to use the jre instead of the jdk regardless of the java parameter.

lopsided98 · 2019-09-10T18:45:45Z

@fgaz #68442 should fix it

Ma27 · 2019-09-15T11:59:20Z

pkgs/development/compilers/openjdk/8.nix

    preFixup = ''
-      prefix=$jre stripDirs "$STRIP" "$stripDebugList" "''${stripDebugFlags:--S}"
-      patchELF $jre
-      propagatedBuildInputs+=" $jre"


The JRE inputs need to be propagated to jdk as well to ensure that the setJavaClassPath hook is used.
This breaks for instance scalafmt and scalafix:

https://hydra.nixos.org/build/100896633

https://hydra.nixos.org/build/100895668

related to ZHF #68361

timokau approved these changes Aug 7, 2019

View reviewed changes

lopsided98 force-pushed the openjdk-arm branch from e2e4bfd to 79fabbd Compare August 7, 2019 22:37

ofborg bot added the 2.status: merge conflict label Aug 7, 2019

lopsided98 force-pushed the openjdk-arm branch from 79fabbd to 29ebbd6 Compare August 7, 2019 22:44

lopsided98 force-pushed the openjdk-arm branch from 29ebbd6 to 67bdce6 Compare August 11, 2019 05:06

ofborg bot removed the 2.status: merge conflict label Aug 11, 2019

flokli requested a review from andir August 12, 2019 11:01

lopsided98 force-pushed the openjdk-arm branch from 67bdce6 to 99a09e6 Compare August 15, 2019 02:29

ofborg bot added 8.has: clean-up 8.has: package (new) labels Aug 16, 2019

ofborg bot requested review from NeQuissimus, edwtjo and abbradar August 16, 2019 12:16

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 501+ 10.rebuild-linux: 501-1000 labels Aug 16, 2019

lopsided98 added 8 commits August 19, 2019 18:54

adoptopenjdk-bin: add OpenJDK 8

ded7ec0

openjdk: boostrap with adoptopenjdk-bin and cleanup

de5e65a

This allows OpenJDK 11 to build for armv6l, armv7l and aarch64, and OpenJDK 8 to build for aarch64.

openjfx: fix package name

4aaaf51

openjdk8: 8u212 -> 8u222

6f8f3ed

openjdk11: 11.0.3 -> 11.0.4

d4e2cb0

adoptopenjdk: fix fontconfig dependency

b4480b7

openjdk12: cleanup based on refactoring of openjdk11

5d07172

chromium: remove obsolete patches

8b0a684

One of these patches was causing a warning message, which broke ofborg evaluation.

lopsided98 force-pushed the openjdk-arm branch from 9c5bc92 to 8b0a684 Compare August 19, 2019 23:20

lheckemann added this to the 19.09 milestone Aug 20, 2019

matthewbauer approved these changes Aug 20, 2019

View reviewed changes

timokau merged commit c5c4720 into NixOS:staging Aug 20, 2019

lopsided98 deleted the openjdk-arm branch August 27, 2019 18:18

lopsided98 mentioned this pull request Aug 31, 2019

openjfx: fix licenses #67839

Merged

10 tasks

vcunat mentioned this pull request Sep 8, 2019

manual build fails because of jing-trang blocking channels #68307

Closed

lopsided98 mentioned this pull request Sep 10, 2019

openjdk8: replacing jre symlinking with copy #68382

Closed

10 tasks

Ma27 reviewed Sep 15, 2019

View reviewed changes

lopsided98 mentioned this pull request Nov 5, 2019

openjdk8: restore correct handling of cmm directory #72843

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

openjdk: enable bootstrapping on ARM #65247

openjdk: enable bootstrapping on ARM #65247

lopsided98 commented Jul 22, 2019 •

edited

lopsided98 commented Jul 22, 2019

lheckemann commented Jul 22, 2019

lopsided98 commented Jul 22, 2019

timokau left a comment

timokau Aug 7, 2019

lopsided98 Aug 11, 2019

timokau Aug 12, 2019

taku0 Aug 14, 2019

lopsided98 Aug 15, 2019

timokau commented Aug 7, 2019

taku0 commented Aug 10, 2019

lopsided98 commented Aug 11, 2019

grahamc commented Aug 12, 2019

timokau commented Aug 14, 2019

lopsided98 commented Aug 15, 2019

timokau commented Aug 15, 2019

timokau commented Aug 16, 2019

ivan commented Aug 16, 2019

timokau commented Aug 16, 2019

taku0 commented Aug 16, 2019

timokau commented Aug 19, 2019

lopsided98 commented Aug 19, 2019

timokau commented Aug 20, 2019

timokau commented Aug 20, 2019

FRidh commented Aug 31, 2019 •

edited

timokau commented Aug 31, 2019

lopsided98 commented Aug 31, 2019 •

edited

lopsided98 commented Aug 31, 2019

timokau commented Sep 1, 2019

vcunat commented Sep 8, 2019 •

edited

fgaz commented Sep 10, 2019

lopsided98 commented Sep 10, 2019

Ma27 Sep 15, 2019 •

edited

openjdk: enable bootstrapping on ARM #65247

openjdk: enable bootstrapping on ARM #65247

Conversation

lopsided98 commented Jul 22, 2019 • edited

Motivation for this change

Things done

Testing

lopsided98 commented Jul 22, 2019

lheckemann commented Jul 22, 2019

lopsided98 commented Jul 22, 2019

timokau left a comment

Choose a reason for hiding this comment

timokau Aug 7, 2019

Choose a reason for hiding this comment

lopsided98 Aug 11, 2019

Choose a reason for hiding this comment

timokau Aug 12, 2019

Choose a reason for hiding this comment

taku0 Aug 14, 2019

Choose a reason for hiding this comment

lopsided98 Aug 15, 2019

Choose a reason for hiding this comment

timokau commented Aug 7, 2019

taku0 commented Aug 10, 2019

lopsided98 commented Aug 11, 2019

grahamc commented Aug 12, 2019

timokau commented Aug 14, 2019

lopsided98 commented Aug 15, 2019

timokau commented Aug 15, 2019

timokau commented Aug 16, 2019

ivan commented Aug 16, 2019

timokau commented Aug 16, 2019

taku0 commented Aug 16, 2019

timokau commented Aug 19, 2019

lopsided98 commented Aug 19, 2019

timokau commented Aug 20, 2019

timokau commented Aug 20, 2019

FRidh commented Aug 31, 2019 • edited

timokau commented Aug 31, 2019

lopsided98 commented Aug 31, 2019 • edited

lopsided98 commented Aug 31, 2019

timokau commented Sep 1, 2019

vcunat commented Sep 8, 2019 • edited

fgaz commented Sep 10, 2019

lopsided98 commented Sep 10, 2019

Ma27 Sep 15, 2019 • edited

Choose a reason for hiding this comment

lopsided98 commented Jul 22, 2019 •

edited

FRidh commented Aug 31, 2019 •

edited

lopsided98 commented Aug 31, 2019 •

edited

vcunat commented Sep 8, 2019 •

edited

Ma27 Sep 15, 2019 •

edited