SWT reproducibility #59999

bb010g · 2019-04-22T02:16:51Z

Motivation for this change

More reproducibility! And less likelihood of breakage moving forward into new versions. Also, the build-support/setup-hooks/canonicalize-jars-hook hook added to support this was pulled from the unused build-support/release/ant-build (& functions only used there sourced from build-support/release/functions.sh), which has been deleted due to being weird and old and not imported by any package built on Hydra in the last 3+ years.

Things done

bb010g · 2019-04-27T06:22:23Z

@edolstra may also want to review this, as I think the build-support/release stuff was mostly him?

rycee

I think this is very nice! Having the ability to easily canonicalize Jar files is very convenient!

pkgs/build-support/java/canonicalize-jar.nix

pkgs/build-support/java/canonicalize-jar.sh

pkgs/development/libraries/java/swt/default.nix

bb010g · 2019-10-07T00:44:09Z

Ok, rebased and ready for review.

doronbehar · 2020-02-23T17:16:26Z

Please fix the merge conflicts.

bb010g · 2020-02-25T09:36:16Z

Rebased.

stale · 2020-08-23T10:20:30Z

Hello, I'm a bot and I thank you in the name of the community for your contributions.

Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do:

If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list.

If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past.

If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments.

Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel.

bb010g · 2021-01-16T00:55:31Z

Rebased.

bb010g · 2021-06-02T22:33:28Z

Rebased. Reminder that this is blocking SWT upgrades.

pkgs/top-level/all-packages.nix

pkgs/development/libraries/java/swt/default.nix

pkgs/build-support/setup-hooks/canonicalize-jars.sh

pkgs/build-support/java/canonicalize-jar.sh

fzakaria · 2021-06-28T17:34:39Z

I recently wrote https://fzakaria.com/2021/06/27/java-nix-reproducibility.html
Curious about the delta between the hook and other canonicalization vs. just adding strip-nondeterminism to the fixupPhase?

fzakaria · 2021-06-28T17:38:01Z

Can the PR description's motivation be augmented to describe the problem and the fixes?
I'm aware of the mtime fixes but not the other canonicalizations that are needed.

Doing it in a hook vs the fixupPhase is also of interest (see earlier comment)

bb010g · 2021-09-21T06:32:06Z

Rebased, and @pSub removed as maintainer due to a lack of activity with this PR over the last 2 years.

@fzakaria It looks like META-INF/MANIFEST.MF is sorted & empty lines removed? It's been a while since I submitted this patch; that behavior was carried over from the original canonicalizeJarManifest function. Further documentation or fixes are welcome; I know that this PR works and just want it to finally get merged so I can use ipscan from upstream Nixpkgs.

Doing it in a hook vs the fixupPhase is also of interest

canonicalizeJarsIn is called in fixupOutputHooks, which are ran directly after runHook preFixup in the default fixupPhase.

doronbehar

I'd also squash the last two commits into 1.

pkgs/top-level/all-packages.nix

pkgs/development/libraries/java/swt/default.nix

doronbehar · 2021-09-25T16:51:38Z

pkgs/development/libraries/java/swt/default.nix

+    sed -i 's/ \+$//' ./*.mak
+  '';
+
+  postPatch = let makefile-sed = builtins.toFile "swt-makefile.sed" (''


Have you considered applying a patch and suggesting it upstream?

Yes, but I figured upstream submissions could be figured out after our SWT was up-to-date first. I don't think 4.5 is still maintained. My original idea was to upgrade SWT after this was merged first, to keep each PR easier to follow.

I tried to update SWT once..

#75609

Do you still plan to update SWT now that this PR has been merged? @bb010g

I'm trying to fix ipscan, which fails to start up, with:

***WARNING: GTK+ version too new (major mismatch) ***WARNING: SWT requires GTK 2.18.0 ***WARNING: Detected: 2.24.33

And then a JVM segfault:

# A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fe4605448bc, pid=2188, tid=2189 # # JRE version: OpenJDK Runtime Environment (17.0.1+12) (build 17.0.1+12-nixos) # Java VM: OpenJDK 64-Bit Server VM (17.0.1+12-nixos, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64) # Problematic frame: # C [libX11.so.6+0x308bc] XInternAtom+0x3c

Hoping an updated SWT will fix this...

doronbehar

I noticed some dependencies can be cleaned and added:

diff --git i/pkgs/development/libraries/java/swt/default.nix w/pkgs/development/libraries/java/swt/default.nix
index 348af9d255d..f965a1dc3c5 100644
--- i/pkgs/development/libraries/java/swt/default.nix
+++ w/pkgs/development/libraries/java/swt/default.nix
@@ -1,6 +1,15 @@
-{ lib, stdenv, fetchzip, pkgsBuildHost
-, atk, glib, gtk2, jdk, libGL, libGLU, libXi, libXt, libXtst, libsoup
-, webkitgtk, xorg
+{ lib
+, stdenv
+, fetchzip
+, pkgsBuildHost
+, atk
+, glib
+, gtk2
+, jdk
+, libGL
+, libGLU
+, libXtst
+, gnome2
 }:
 
 let
@@ -56,11 +65,10 @@ in stdenv.mkDerivation rec {
     jdk
     libGL
     libGLU
-    libXi
-    libXt
     libXtst
-    libsoup
-    webkitgtk
+    gnome2.gnome_vfs
+    gnome2.libgnomeui
+    gnome2.libgnome
   ];
 
   patches = [ ./awt-libs.patch ./gtk-libs.patch ];

pkgs/development/libraries/java/swt/default.nix

doronbehar · 2021-10-25T08:30:02Z

ping @bb010g .

A build hook to run functions previously only implemented privately in `pkgs/build-support/release/functions.sh`.

The sole consumer in Nixpkgs of `releaseTools.antBuild` is `pkgs/development/libraries/junit`, which has been broken since 2015-09-08. The sole consumer in Nixpkgs of `junit` is `pkgs/development/libraries/junixsocket`, which hasn't built due to `junit` since 2015-09-08. All three are removed due to their obvious lack of use. All other packages in Nixpkgs depending on junit consume `pkgs/development/java-modules/junit`, which is not broken. Any downstreams that have kept using these `junit` or `junixsocket` packages since 2015-09-08 have basically already vendored the packages via patching them, so no aliases are provided.

Cleaner nested unpacking, as well as general robustness improvements. Turns out the LFLAGS stuff was from upstream not trusting pkg-config on their boxes, but it works great for us. (Or rather, it works great after fixing some of their pkg-config invocations.) Assisted by the diffoscope ( https://diffoscope.org/ ) and readelf grepping based on its output.

doronbehar · 2021-10-27T11:02:02Z

Apparently swt_jdk8 does need libXt. I added that dependency conditionally, and marked areca as broken` since it won't launch a GUI.

bb010g requested a review from Ericson2314 as a code owner April 22, 2019 02:16

GrahamcOfBorg added 8.has: clean-up 8.has: package (new) labels Apr 22, 2019

GrahamcOfBorg requested a review from pSub April 22, 2019 02:32

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Apr 22, 2019

rycee reviewed Apr 27, 2019

View reviewed changes

veprbl added the 2.status: merge conflict label Oct 7, 2019

bb010g force-pushed the swt-reproducibility branch 2 times, most recently from 8a9a17c to f8f3d03 Compare October 7, 2019 00:43

ofborg bot removed the 2.status: merge conflict label Oct 7, 2019

doronbehar mentioned this pull request Dec 13, 2019

swt: 4.5 -> 4.14 #75609

Closed

10 tasks

bb010g force-pushed the swt-reproducibility branch from f8f3d03 to 48f1ac2 Compare February 25, 2020 09:36

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Aug 23, 2020

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 3, 2020

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

bb010g force-pushed the swt-reproducibility branch from 48f1ac2 to c7ffdab Compare January 16, 2021 00:55

bb010g force-pushed the swt-reproducibility branch from c7ffdab to 71ff627 Compare January 16, 2021 00:57

ofborg bot added 10.rebuild-darwin: 1 and removed 2.status: merge conflict labels Jan 16, 2021

bb010g force-pushed the swt-reproducibility branch from 71ff627 to 8821095 Compare June 2, 2021 22:32

bb010g requested a review from rycee June 2, 2021 22:36

ofborg bot added the 11.by: package-maintainer label Jun 2, 2021

SuperSandro2000 requested changes Jun 6, 2021

View reviewed changes

bb010g force-pushed the swt-reproducibility branch from 8821095 to 3250a5f Compare September 21, 2021 06:21

doronbehar reviewed Sep 25, 2021

View reviewed changes

doronbehar reviewed Sep 30, 2021

View reviewed changes

pkgs/development/libraries/java/swt/default.nix Outdated Show resolved Hide resolved

pkgs/development/libraries/java/swt/default.nix Outdated Show resolved Hide resolved

Mindavi added the 6.topic: reproducible builds label Oct 13, 2021

bb010g force-pushed the swt-reproducibility branch from 3250a5f to bf8c49c Compare October 25, 2021 20:42

bb010g and others added 5 commits October 27, 2021 13:21

canonicalize-jars-hook: add

29fedf2

A build hook to run functions previously only implemented privately in `pkgs/build-support/release/functions.sh`.

swt: make reproducible

20ccd74

areca: Mark as broken

0ed0c3c

doronbehar force-pushed the swt-reproducibility branch from bf8c49c to 0ed0c3c Compare October 27, 2021 11:00

doronbehar merged commit 51acb65 into NixOS:master Oct 27, 2021

bb010g deleted the swt-reproducibility branch November 9, 2021 22:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SWT reproducibility #59999

SWT reproducibility #59999

bb010g commented Apr 22, 2019

bb010g commented Apr 27, 2019

rycee left a comment

bb010g commented Oct 7, 2019

doronbehar commented Feb 23, 2020

bb010g commented Feb 25, 2020

stale bot commented Aug 23, 2020

bb010g commented Jan 16, 2021

bb010g commented Jun 2, 2021

fzakaria commented Jun 28, 2021

fzakaria commented Jun 28, 2021

bb010g commented Sep 21, 2021

doronbehar left a comment

doronbehar Sep 25, 2021

bb010g Sep 25, 2021

doronbehar Sep 26, 2021

kylesferrazza Dec 4, 2021 •

edited

doronbehar left a comment •

edited

doronbehar commented Oct 25, 2021

doronbehar commented Oct 27, 2021

SWT reproducibility #59999

SWT reproducibility #59999

Conversation

bb010g commented Apr 22, 2019

Motivation for this change

Things done

bb010g commented Apr 27, 2019

rycee left a comment

Choose a reason for hiding this comment

bb010g commented Oct 7, 2019

doronbehar commented Feb 23, 2020

bb010g commented Feb 25, 2020

stale bot commented Aug 23, 2020

bb010g commented Jan 16, 2021

bb010g commented Jun 2, 2021

fzakaria commented Jun 28, 2021

fzakaria commented Jun 28, 2021

bb010g commented Sep 21, 2021

doronbehar left a comment

Choose a reason for hiding this comment

doronbehar Sep 25, 2021

Choose a reason for hiding this comment

bb010g Sep 25, 2021

Choose a reason for hiding this comment

doronbehar Sep 26, 2021

Choose a reason for hiding this comment

kylesferrazza Dec 4, 2021 • edited

Choose a reason for hiding this comment

doronbehar left a comment • edited

Choose a reason for hiding this comment

doronbehar commented Oct 25, 2021

doronbehar commented Oct 27, 2021

kylesferrazza Dec 4, 2021 •

edited

doronbehar left a comment •

edited