Bloop: 1.3.4 -> 1.4.3 #91758

Tomahna · 2020-06-29T11:39:19Z

Motivation for this change

Updates bloop from 1.3.4 to 1.4.2.

During the 1.4.+ upgrade, bloop has changed the way it is packaged. While the previous way of packaging it still works, bloop now uses graalvm native images in other distributions. This PR updates the derivation in order to be as close as possible to the way bloop is packaged in the archlinux user repository

This PR should supersedes #89359, commits and attributions have been kept.

Things done

Tomahna · 2020-06-29T11:52:58Z

@karolchmist & @svalaskevicius please take a look. The problem of non-fixed hash should be fixed.

Sorry to multiply the PRs, but it just seemed much easier to just throw my modifications into a new branch and fiddle with git than trying to update the other PR.

karolchmist · 2020-06-29T12:44:14Z

Hello @Tomahna, nice! I just tested it, works great. I have no power of merging, but it's 👍 from me.

svalaskevicius · 2020-06-29T19:32:16Z

Looks good to me too :) thanks!

Not too sure what's the process of actually merging it, last few times @worldofpeace helped.

ludovicc · 2020-07-03T00:05:19Z

Reviewed points

package name fits guidelines
package version fits guidelines
package build on MacOS Catalina
executables tested on MacOS Catalina
all depending packages build

Possible improvements

Comments

Tomahna · 2020-07-03T16:35:52Z

Updated with bloop 1.4.3

pkgs/development/tools/build-managers/bloop/default.nix

* Include all completions * Update derivation to make it similar to archlinux packaging

worldofpeace · 2020-07-08T16:43:10Z

What test can I do to determine this upgrade has been successful?

Tomahna · 2020-07-08T17:14:28Z

You can use bloop about. It will start the build server and print you the current version (1.4.3).

Other than that, well you would have to set up a basic scala environment and build a scala project with it.

ludovicc · 2020-07-09T21:05:33Z

On Darwin and with bloop 1.4.3 from your latest update, bloop about does not start the build server:

bloop about
Could not connect to server 127.0.0.1:8212

Have you forgotten to start bloop's server? Run it with `bloop server`.
Check our usage instructions in https://scalacenter.github.io/bloop/

If I install bloop using coursier, bloop about starts the build server.

Tomahna · 2020-07-10T06:59:14Z

It is strange, it starts the server properly on nixos.

No server running at 127.0.0.1:8212, let's fire one...
Resolving ch.epfl.scala:bloop-frontend_2.12:1.4.3...
Starting bloop server at 127.0.0.1:8212...
Attempting a connection to the server...
Attempting a connection to the server...
Attempting a connection to the server...
Attempting a connection to the server...
bloop v1.4.3

Using Scala v2.12.8 and Zinc v1.3.0-M4+45-d4354be3
Running on Java JDK v11.0.7-internal (/nix/store/q0ap8aa5br4vr2y75n7hmacwlihmpdgq-openjdk-11.0.7-ga/lib/openjdk)
  -> Supports debugging user code, Java Debug Interface (JDI) is available.
Maintained by the Scala Center (Jorge Vicente Cantero, Martin Duhem)

Can you try starting the build server manually (through the bloop server command). And send me the log of both your coursier install and the nix one.

My guess is that the installer does not select the correct binary.

ludovicc · 2020-07-10T09:13:04Z

coursier install bloop --only-prebuilt=true --verbose
Using install directory /Users/lclaude/Library/Application Support/Coursier/bin
https://repo1.maven.org/maven2/io/get-coursier/apps/maven-metadata.xml
  No new update since 2020-06-30 14:51:44
Found app bloop in channel io.get-coursier:apps
https://repo1.maven.org/maven2/org/scala-lang/scala-library/maven-metadata.xml
  No new update since 2020-06-25 23:27:11
https://repo1.maven.org/maven2/ch/epfl/scala/bloopgun_2.12/maven-metadata.xml
  No new update since 2020-07-01 00:19:57
https://repo1.maven.org/maven2/ch/epfl/scala/bloopgun_2.12/maven-metadata.xml
  No new update since 2020-07-01 00:19:57
Found prebuilt launcher at https://github.com/scalacenter/bloop/releases/download/v1.4.3/bloop-x86_64-apple-darwin (37.2 MiB)
Wrote /Users/lclaude/Library/Application Support/Coursier/bin/.bloop.part
Wrote /Users/lclaude/Library/Application Support/Coursier/bin/bloop
Wrote /Users/lclaude/Library/Application Support/Coursier/bin/.bloop.aux
Wrote bloop

❯ which bloop
/Users/lclaude/Library/Application Support/Coursier/bin/bloop
❯ bloop about
No server running at 127.0.0.1:8212, let's fire one...
Resolving ch.epfl.scala:bloop-frontend_2.12:1.4.3...
Starting bloop server at 127.0.0.1:8212...
Attempting a connection to the server...
Attempting a connection to the server...
Attempting a connection to the server...
Attempting a connection to the server...
Attempting a connection to the server...
Attempting a connection to the server...
Attempting a connection to the server...
bloop v1.4.3

Using Scala v2.12.8 and Zinc v1.3.0-M4+45-d4354be3
Running on Java JDK v11.0.1 (/nix/store/sm4xb0ggmxy6sn5ahbkmrm53v8hv4lj6-zulu11.2.3-jdk11.0.1)
  -> Supports debugging user code, Java Debug Interface (JDI) is available.
Maintained by the Scala Center (Jorge Vicente Cantero, Martin Duhem)

Tomahna · 2020-07-11T09:42:44Z

Hi @ludovicc,

I've adapted the derivation to be able to fetch correct binaries on darwin. Unfortunately I do not have a mac to test it.

Could you give it a try ? You would have to provide the correct sha256 for darwin (line 56). I've not been able to figure out how nixos computes the recursive sha256 yet. (Or to find out how to test it for another platform).

ludovicc · 2020-07-11T12:03:16Z

@Tomahna here are the logs of a build on Darwin:

Downloading https://repo1.maven.org/maven2/ch/epfl/scala/bloopgun_2.12/1.4.3/bloopgun_2.12-1.4.3.pom
Downloaded https://repo1.maven.org/maven2/ch/epfl/scala/bloopgun_2.12/1.4.3/bloopgun_2.12-1.4.3.pom
Downloading https://github.com/scalacenter/bloop/releases/download/v1.4.3/bloop-x86_64-apple-darwin
Still downloading:
https://github.com/scalacenter/bloop/releases/download/v1.4.3/bloop-x86_64-apple-darwin (73.60 %, 28741767 / 39049932)

Downloaded https://github.com/scalacenter/bloop/releases/download/v1.4.3/bloop-x86_64-apple-darwin
Wrote bloop
Warning: /nix/store/f1yca88p3swljsai9rhprhz88v7jcgbw-bloop-coursier-1.4.3 is not in your PATH
To fix that, add the following line to ~/.bashrc

export PATH="$PATH:/nix/store/f1yca88p3swljsai9rhprhz88v7jcgbw-bloop-coursier-1.4.3"
hash mismatch in fixed-output derivation '/nix/store/85sfggf1iynf2ig927gg2940bzx5zfsx-bloop-coursier-1.4.3':
  wanted: sha256:1ncl34f39mvk0zb5jl1l77cwjdg3xfnhjxbzz11pdfqw0d7wqywj
  got:    sha256:06c885w088yvh8l1r1jbrz0549gx2xvc8xr6rlxy6y27jk5655p2
note: keeping build directory '/private/var/folders/v5/83p3n3f546nc81syhhrfs9mr0000gn/T/nix-build-bloop-coursier-1.4.3.drv-0'
cannot build derivation '/nix/store/g37csdll1dw7fghlddc47c57vind95hj-bloop-1.4.3.drv': 1 dependencies couldn't be built
error: build of '/nix/store/g37csdll1dw7fghlddc47c57vind95hj-bloop-1.4.3.drv' failed

If you use 06c885w088yvh8l1r1jbrz0549gx2xvc8xr6rlxy6y27jk5655p2 as sha256 code for the darwin binary, it works well:

bloop v1.4.3

Using Scala v2.12.8 and Zinc v1.3.0-M4+45-d4354be3
Running on Java JDK v11.0.1 (/nix/store/sm4xb0ggmxy6sn5ahbkmrm53v8hv4lj6-zulu11.2.3-jdk11.0.1)
  -> Supports debugging user code, Java Debug Interface (JDI) is available.
Maintained by the Scala Center (Jorge Vicente Cantero, Martin Duhem)

Tomahna · 2020-07-11T12:05:41Z

Awesome, I've updated this PR with the provided sha.

ludovicc · 2020-07-13T08:26:41Z

I re-tested on Darwin (macOs Catalina 10.15.5), it works well.

svalaskevicius · 2020-07-16T17:11:09Z

does anyone know what's left to be done before this can be merged? :)

ludovicc · 2020-07-16T21:19:26Z

@jonringer can you review this pr? LGTM on my side.

jonringer

seems to only work if java on PATH:

[nix-shell:/home/jon/.cache/nixpkgs-review/pr-91758]$ nix-shell --pure -p "with import ./nixpkgs {}; bloop"
these derivations will be built:
  /nix/store/cl19rgxmxni5c2y1aw8bjnwq3ick49b9-bloop-1.4.3.drv
...

[nix-shell:/home/jon/.cache/nixpkgs-review/pr-91758]$ bloop --help
No server running at 127.0.0.1:8212, let's fire one...
Resolving ch.epfl.scala:bloop-frontend_2.12:1.4.3...
Starting bloop server at 127.0.0.1:8212...
java.io.IOException: Cannot run program "java" (in directory "/home/jon/.cache/nixpkgs-review/pr-91758"
....

[nix-shell:/home/jon/.cache/nixpkgs-review/pr-91758]$ nix-shell --pure -p "with import ./nixpkgs {}; [ bloop openjdk.jre ]"

[nix-shell:/home/jon/.cache/nixpkgs-review/pr-91758]$ bloop about
No server running at 127.0.0.1:8212, let's fire one...
Resolving ch.epfl.scala:bloop-frontend_2.12:1.4.3...
Starting bloop server at 127.0.0.1:8212...
Attempting a connection to the server...
Attempting a connection to the server...
Attempting a connection to the server...
bloop v1.4.3

Using Scala v2.12.8 and Zinc v1.3.0-M4+45-d4354be3
Running on Java JRE v1.8.0_242 (/nix/store/7qknskbanpwq7a71s2n6dv3l5b3frj7d-openjdk-8u242-b08-jre/lib/openjdk/jre)
  -> Doesn't support debugging user code, runtime doesn't implement Java Debug Interface (JDI).
Maintained by the Scala Center (Jorge Vicente Cantero, Martin Duhem)

Tomahna · 2020-07-16T22:41:31Z

I'm not sure what to do about that. The precompiled binary is expecting Java on the classpath, I don't see how we can patch that.

Should the derivation do something about it ? Is it really a problem ?

jonringer · 2020-07-16T23:35:10Z

wrapping doesn't seem to work, but doing propagatedBuildInputs = [ openjdk.jre ]; does work

pkgs/development/tools/build-managers/bloop/default.nix

jonringer · 2020-07-16T23:47:59Z

pkgs/development/tools/build-managers/bloop/default.nix

+
+  nativeBuildInputs = [ autoPatchelfHook ] ;
+  phases = [ "installPhase" "fixupPhase" ];
+  buildInputs = [ stdenv.cc.cc.lib zlib ] ;


fix java missing issue

Suggested change

buildInputs = [ stdenv.cc.cc.lib zlib ] ;

buildInputs = [ stdenv.cc.cc.lib zlib ] ;

propagatedBuildInputs = [ openjdk.jre ];

this does increase the closure size quite a bit:

[nix-shell:/home/jon/.cache/nixpkgs-review/pr-91758-4]$ nix path-info -Sh ./results/bloop /nix/store/fp1kc7lxw0rhhs6my624wm154hn8494w-bloop-1.4.3 77.2M [nix-shell:/home/jon/.cache/nixpkgs-review/pr-91758-4]$ nix path-info -Sh ./result /nix/store/c0ls7dh4vnjd4wyqib3mz6yyfz2n3g49-bloop-1.4.3 434.8M

However, it's more representative of it's actual runtime closure:

[16:52:28] jon@jon-desktop /home/jon/projects/nixpkgs (buildDotnetCorePackage) $ nix path-info -Sh ./result-jre /nix/store/7qknskbanpwq7a71s2n6dv3l5b3frj7d-openjdk-8u242-b08-jre 396.5M

This does still mean that we can change to any java we want when using bloop, right? :)

In fact, why are we adding java 8? Does the build not work without it or usage?

Looks like a similar thing is done for sbt already - https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/build-managers/sbt/default.nix, and we can override jvm for bloop processes, so yeah (answering my own question) :)

Would be good if there was a way to configure preferred java provider (or at least the major version) in one place though, but thats much more generic question than this PR

as with most things with nix, there's a way to do anything, it's just how much pain is it to introduce flexibility. I guess you could pass jre to this derivation, and then just say:

bloop = callPackage ../.. { jre = openjdkXX.jre; };

and then if you want to change it, then you would do something like:

bloop.override { jre = blah.jre; };

It does not seem necessary to explicitly pass the jre in the callPackage call, several other derivations which use jre do not do it (coursier, scalafmt, metals, ...).

However by passing the jre inside the derivation, it seems to respect the programs.java.package property or can be overriden with derivation.override.

pkgs/development/tools/build-managers/bloop/default.nix

worldofpeace · 2020-07-17T01:05:33Z

pkgs/development/tools/build-managers/bloop/default.nix

+  };
+
+  nativeBuildInputs = [ autoPatchelfHook ] ;
+  phases = [ "installPhase" "fixupPhase" ];


manually defining phases is deprecated. See the stdenv chapter of the nixpkgs manual to disable phases like dontBuild = true; etc.

It's replaced by dontUnpack = true.

worldofpeace · 2020-07-17T01:06:01Z

pkgs/development/tools/build-managers/bloop/default.nix

+                     else throw "unsupported platform";
+  };
+
+  nativeBuildInputs = [ autoPatchelfHook ] ;


Suggested change

nativeBuildInputs = [ autoPatchelfHook ] ;

nativeBuildInputs = [ autoPatchelfHook ];

this is done in various places

Should be fixed

Tomahna · 2020-07-18T20:24:27Z

All your review comments should have been taken into account and fixed.

svalaskevicius · 2020-07-28T07:23:45Z

@worldofpeace @jonringer hi, please can you review if you're happy with the changes?

Would be good to get this merged as the old bloop version is not really usable anymore (not compatible with the new scala version etc)

jonringer

LGTM

Result of nixpkgs-review pr 91758 1

1 package built:

- bloop

jonringer · 2020-07-28T18:36:58Z

sorry, went on vacation for a week and a half

svalaskevicius · 2020-07-28T19:12:55Z

thanks!

also thanks to everyone involved, especially @Tomahna - for continuously maintaining this derivation - over a few bloop releases now :)

Bloop: 1.3.4 -> 1.4.1

614a72a

ofborg bot added 11.by: package-maintainer 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Jun 29, 2020

karolchmist mentioned this pull request Jun 30, 2020

Bloop: 1.3.4 -> 1.4.1 #89359

Closed

10 tasks

ludovicc approved these changes Jul 2, 2020

View reviewed changes

Tomahna force-pushed the bloop branch from 9827f94 to 265ad21 Compare July 3, 2020 16:34

Tomahna changed the title ~~Bloop: 1.3.4 -> 1.4.2~~ Bloop: 1.3.4 -> 1.4.3 Jul 3, 2020

worldofpeace reviewed Jul 8, 2020

View reviewed changes

pkgs/development/tools/build-managers/bloop/default.nix Outdated Show resolved Hide resolved

worldofpeace reviewed Jul 8, 2020

View reviewed changes

pkgs/development/tools/build-managers/bloop/default.nix Outdated Show resolved Hide resolved

karolchmist and others added 2 commits July 8, 2020 11:05

Bloop: rework bloop packaging

488b211

* Include all completions * Update derivation to make it similar to archlinux packaging

Bloop: 1.4.1 -> 1.4.3

e58afbc

Tomahna force-pushed the bloop branch from 265ad21 to e58afbc Compare July 8, 2020 09:05

Tomahna force-pushed the bloop branch from cfa8831 to 65eb6c7 Compare July 11, 2020 12:04

ofborg bot added 8.has: package (new) 10.rebuild-darwin: 1-10 labels Jul 11, 2020

ofborg bot added 10.rebuild-darwin: 1 and removed 10.rebuild-darwin: 0 labels Jul 11, 2020

jonringer reviewed Jul 16, 2020

View reviewed changes

worldofpeace reviewed Jul 17, 2020

View reviewed changes

Bloop: adapt derivation to darwin

a6715ae

Tomahna force-pushed the bloop branch from 65eb6c7 to a6715ae Compare July 18, 2020 20:19

jonringer approved these changes Jul 28, 2020

View reviewed changes

jonringer merged commit df6e489 into NixOS:master Jul 28, 2020

	buildInputs = [ stdenv.cc.cc.lib zlib ] ;
	buildInputs = [ stdenv.cc.cc.lib zlib ] ;
	propagatedBuildInputs = [ openjdk.jre ];

	nativeBuildInputs = [ autoPatchelfHook ] ;
	nativeBuildInputs = [ autoPatchelfHook ];

Bloop: 1.3.4 -> 1.4.3 #91758

Bloop: 1.3.4 -> 1.4.3 #91758

Conversation

Tomahna commented Jun 29, 2020

Motivation for this change

Things done

Tomahna commented Jun 29, 2020 • edited

karolchmist commented Jun 29, 2020

svalaskevicius commented Jun 29, 2020

ludovicc commented Jul 3, 2020

Reviewed points

Possible improvements

Comments

Tomahna commented Jul 3, 2020

worldofpeace commented Jul 8, 2020

Tomahna commented Jul 8, 2020

ludovicc commented Jul 9, 2020

Tomahna commented Jul 10, 2020 • edited

ludovicc commented Jul 10, 2020

Tomahna commented Jul 11, 2020

ludovicc commented Jul 11, 2020

Tomahna commented Jul 11, 2020

ludovicc commented Jul 13, 2020

svalaskevicius commented Jul 16, 2020 • edited

ludovicc commented Jul 16, 2020

jonringer left a comment

Choose a reason for hiding this comment

Tomahna commented Jul 16, 2020

jonringer commented Jul 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svalaskevicius Jul 17, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svalaskevicius Jul 17, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tomahna commented Jul 18, 2020

svalaskevicius commented Jul 28, 2020

jonringer left a comment

Choose a reason for hiding this comment

jonringer commented Jul 28, 2020

svalaskevicius commented Jul 28, 2020

Tomahna commented Jun 29, 2020 •

edited

Tomahna commented Jul 10, 2020 •

edited

svalaskevicius commented Jul 16, 2020 •

edited

svalaskevicius Jul 17, 2020 •

edited

svalaskevicius Jul 17, 2020 •

edited