preview: add Shenandoah GC to openjdk packages by default #37722

thoughtpolice · 2018-03-24T02:55:01Z

Motivation for this change

Shenandoah is a new, ultra-low pause time garbage collector for the JVM that collects garbage concurrently as the program runs, allowing GC times that do not scale proportional to the heap size: collecting a heap with a size of of 100GB or a 2GB size has the same pause behavior, which should be predictable.

Background

Shenandoah is enabled by updating the hotspot component of the OpenJDK build to
point to the Shenandoah fork -- thanks to a JDK 9 backport from the developers that is continuously kept up to date with the latest JDK versions, this is viable without bitrotting the packages, or having to put them under separate names.

While the hotspot component comes from the Shenandoah fork, it is not the default GC in this branch -- G1 still reigns by default on x86_64 Linux, and as the branch is kept up to date, still gets all the latest fixes.

This change was based on the Gentoo IcedTea build (which features Shenandoah support as a USE flag).

User impact

None by default; Shenandoah is opt-in only. This can be enabled with -XX:+UseShenandoahGC, e.g.

$ ./result/bin/java -XX:+UseShenandoahGC -Xlog:gc -version
[0.014s][info][gc] Using Shenandoah
openjdk version "9.0.4-internal"
OpenJDK Runtime Environment (build 9.0.4-internal+0-adhoc..jdk9u-jdk-9.0.412)
OpenJDK 64-Bit Server VM (build 9.0.4-internal+0-adhoc..jdk9u-jdk-9.0.412, mixed mode)
[0.076s][info][gc] Cancelling concurrent GC: Stopping VM

Still not done

The developers also keep backports of Shenandoah to JDK8 and (the new) JDK10. I would like it to be added to all these packages as well, to give a consistent preview feature set to all openjdk* package users. Currently, for JDK 8, I'm waiting on a new merge with the upstream jdk8u-updates branch (the fork is a tiny bit behind), while JDK10 is still not yet packaged.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- ~~macOS~~
- other Linux distributions (Fedora 25 with Sandboxing)
~~Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests).~~ openjdk9 is not the default JDK (openjdk8 is), so almost nothing is tested by it now. I have done my own testing.
~~Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip".~~ openjdk9 is not the default JDK (openjdk8 is), so almost nothing is rebuilt by changing this. I have done my own testing.
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

GrahamcOfBorg · 2018-03-24T03:05:58Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: openjdk9

Partial log (click to expand)

Cannot nix-instantiate `openjdk9' because:
�[31;1merror:�[0m while evaluating the attribute 'buildInputs' of the derivation 'openjdk-9.0.4-b12' at �[1m/var/lib/gc-of-borg/nix-test-rs-9/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-9/pkgs/stdenv/generic/make-derivation.nix�[0m:148:11:
while evaluating the attribute 'buildInputs' of the derivation 'openjdk-8u172b02' at �[1m/var/lib/gc-of-borg/nix-test-rs-9/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-9/pkgs/stdenv/generic/make-derivation.nix�[0m:148:11:
while evaluating the attribute 'buildCommand' of the derivation 'openjdk-bootstrap' at �[1m/var/lib/gc-of-borg/nix-test-rs-9/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-9/pkgs/stdenv/generic/make-derivation.nix�[0m:148:11:
No bootstrap for system

GrahamcOfBorg · 2018-03-24T03:37:05Z

Success on x86_64-linux (full log)

Attempted: openjdk9

Partial log (click to expand)

shrinking /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre/lib/openjdk/jre/lib/libmanagement_agent.so
shrinking /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre/lib/openjdk/jre/lib/libdt_socket.so
shrinking /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre/lib/openjdk/jre/lib/jli/libjli.so
shrinking /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre/lib/openjdk/jre/lib/libsctp.so
shrinking /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre/lib/openjdk/jre/lib/liblcms.so
strip is /nix/store/fzcs0fn6bb04m82frhlb78nc03ny3w55-binutils-2.28.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre/lib  /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre/bin
patching script interpreter paths in /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre
checking for references to /build in /nix/store/jcnwpykq9529kyam43a2p88zmysrkkxn-openjdk-9.0.4-b12-jre...
/nix/store/55vbjsraj032ljyqzxyqspq997k5hcjp-openjdk-9.0.4-b12

shlevy · 2018-03-24T05:19:22Z

Are there any benchmarks available for real world nixpkgs use cases? E.g. libreoffice?

thoughtpolice · 2018-03-24T05:39:50Z

To be honest, I'd think this is more likely to be useful for server workloads where consistent pause times at high heap sizes and large loads are a bigger priority than throughput; most desktop apps I'd hope aren't taking obscene amounts of RAM where proportional GC time starts being a major issue -- does LibreOffice really suffer from such bad pause times with large heaps? It really depends on how they set up their JRE to run...

Then again, desktop applications are precisely the place you want consistent latency, so maybe it's worth trying! But I haven't done that, I'm testing .jar file applications that can be monitored remotely through uses of tools like (the newly added) jhiccup.

That said, I can't give my own conclusive benchmarks yet, but in my limited testing, the basic results seem to align with what're on the Shenandoah wiki -- greatly lowered and far more consistent pause times with GC-heavy workloads at medium-ish heap sizes (in my case, about 16GB-20GB live).

GrahamcOfBorg · 2018-03-30T03:33:39Z

Success on aarch64-linux (full log)

Attempted: diffoscope

The following builds were skipped because they don't evaluate on aarch64-linux: openjdk, openjdk10, openjdk8

Partial log (click to expand)

Cannot nix-instantiate `openjdk8' because:
�[31;1merror:�[0m while evaluating the attribute 'buildInputs' of the derivation 'openjdk-8u172b02' at �[1m/var/lib/gc-of-borg/nix-test-rs-6/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-6/pkgs/stdenv/generic/make-derivation.nix�[0m:148:11:
while evaluating the attribute 'buildCommand' of the derivation 'openjdk-bootstrap' at �[1m/var/lib/gc-of-borg/nix-test-rs-6/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/grahamc-aarch64-community-6/pkgs/stdenv/generic/make-derivation.nix�[0m:148:11:
No bootstrap for system

these derivations will be built:
  /nix/store/hba2ya5bfzn4qjlmxjn2jaajgr15p9fv-rpm-4.14.1.drv
  /nix/store/89sj69z9n6a7cmfdspsj7y3b695pfclz-diffoscope-91.drv
waiting for locks or build slots...
/nix/store/2yrjbjzmm4hhj5l2rxvb8b32dfdl5kyj-diffoscope-91

GrahamcOfBorg · 2018-03-30T03:54:24Z

Failure on x86_64-linux (full log)

Attempted: diffoscope, openjdk, openjdk10, openjdk8

Partial log (click to expand)

strange: no string table
strange: no string table
strange: no string table
strange: no string table
strange: no string table
strange: no string table
strange: no string table
strange: no string table
strange: no string table
�[31;1merror:�[0m build of '/nix/store/jg0sa9l8305p1801a6ymplfqmrlg32rr-openjdk-8u172b02.drv' failed

Signed-off-by: Austin Seipp <aseipp@pobox.com>

thoughtpolice · 2018-04-01T20:26:25Z

Now with JDK 10 support; JDK 8 will be coming up soon once the Shenandoah backports for jdk8u are available...

GrahamcOfBorg · 2018-04-01T20:42:31Z

Success on x86_64-linux (full log)

Attempted: openjdk10

Partial log (click to expand)

/nix/store/31zz36pr6h3py87lk0i26s4asgfg24wp-openjdk-10-b46

GrahamcOfBorg · 2018-04-01T20:42:32Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: openjdk10

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

bobvanderlinden · 2019-04-13T21:39:35Z

A package option would be the least intrusive way to introduce this feature. Mainly because seems not clear what other effects Shenandoah has when -XX:+UseShenandoahGC. Having a package option makes it very explicit whether Shenandoah is used in nixpkgs.

thoughtpolice · 2019-04-14T01:00:17Z

A package option would be the least intrusive way to introduce this feature.

I disagree.

Mainly because seems not clear what other effects Shenandoah has when -XX:+UseShenandoahGC.

There is no difference except this GC is enabled and compiled in (the backport Shenandoah branches are always merged to stable JDK branches with no other differences, ports normally need to deal with a little code movement and that's it). That's why I enabled it unilaterally.

Having a package option makes it very explicit whether Shenandoah is used in nixpkgs.

I don't agree adding options for everything is necessary, even in this case. But It doesn't actually matter in the long run anymore, because Shenandoah is part of JDK 12 officially, finally (as well as ZGC in JDK11). So this should be closed, regardless.

thoughtpolice added 0.kind: enhancement 8.has: documentation 8.has: package (update) 9.needs: community feedback 11.by: nixpkgs-member labels Mar 24, 2018

thoughtpolice self-assigned this Mar 24, 2018

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 and removed 8.has: documentation labels Mar 24, 2018

thoughtpolice changed the title ~~preview: add Shenandoah GC to jdk packages, by default~~ preview: add Shenandoah GC to openjdk packages by default Mar 24, 2018

thoughtpolice force-pushed the shenandoah branch from e32979c to 2ef7771 Compare March 30, 2018 03:20

GrahamcOfBorg added 8.has: documentation 8.has: clean-up 10.rebuild-linux: 101-500 and removed 10.rebuild-linux: 1-10 labels Mar 30, 2018

openjdk10: build Shenandoah fork

b6cf384

Signed-off-by: Austin Seipp <aseipp@pobox.com>

thoughtpolice force-pushed the shenandoah branch from 2ef7771 to b6cf384 Compare April 1, 2018 20:24

GrahamcOfBorg added 10.rebuild-linux: 1-10 and removed 8.has: documentation 10.rebuild-linux: 101-500 labels Apr 1, 2018

ryantm added the 2.status: merge conflict label Feb 26, 2019

thoughtpolice closed this Apr 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

preview: add Shenandoah GC to openjdk packages by default #37722

preview: add Shenandoah GC to openjdk packages by default #37722

thoughtpolice commented Mar 24, 2018 •

edited

GrahamcOfBorg commented Mar 24, 2018

GrahamcOfBorg commented Mar 24, 2018

shlevy commented Mar 24, 2018

thoughtpolice commented Mar 24, 2018

GrahamcOfBorg commented Mar 30, 2018

GrahamcOfBorg commented Mar 30, 2018

thoughtpolice commented Apr 1, 2018

GrahamcOfBorg commented Apr 1, 2018

GrahamcOfBorg commented Apr 1, 2018

bobvanderlinden commented Apr 13, 2019

thoughtpolice commented Apr 14, 2019

preview: add Shenandoah GC to openjdk packages by default #37722

preview: add Shenandoah GC to openjdk packages by default #37722

Conversation

thoughtpolice commented Mar 24, 2018 • edited

Motivation for this change

Background

User impact

Still not done

Things done

GrahamcOfBorg commented Mar 24, 2018

GrahamcOfBorg commented Mar 24, 2018

shlevy commented Mar 24, 2018

thoughtpolice commented Mar 24, 2018

GrahamcOfBorg commented Mar 30, 2018

GrahamcOfBorg commented Mar 30, 2018

thoughtpolice commented Apr 1, 2018

GrahamcOfBorg commented Apr 1, 2018

GrahamcOfBorg commented Apr 1, 2018

bobvanderlinden commented Apr 13, 2019

thoughtpolice commented Apr 14, 2019

thoughtpolice commented Mar 24, 2018 •

edited