bazel-deps: init at 2018-05-31 #43018

uri-canva · 2018-07-04T12:36:40Z

Motivation for this change

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

ryantm · 2018-07-06T14:04:01Z

pkgs/development/tools/build-managers/bazel/bazel-deps/default.nix

+
+buildBazelPackage rec {
+  name = "bazel-deps";
+  version = "2018-05-31";


When we use a date as the version we don't put the hyphens in it, because hyphens have other meanings in the names. Can you use 20180531?

No, “The date must be in "YYYY-MM-DD" format.”: https://nixos.org/nixpkgs/manual/#sec-package-naming. (There is also an "-unstable" suffix requirement, but it seems better to skip it for packages that do not plan to do versioned releases.)

Thanks for the correction!

ryantm · 2018-07-06T14:05:47Z

pkgs/development/tools/build-managers/bazel/bazel-deps/default.nix

+{ stdenv, buildBazelPackage, lib, fetchFromGitHub, git, jre, makeWrapper }:
+
+buildBazelPackage rec {
+  name = "bazel-deps";


Please add the version to the name: name = "bazel-deps-${version}";

ryantm · 2018-07-06T14:06:32Z

pkgs/development/tools/build-managers/bazel/bazel-deps/default.nix

+      cp bazel-bin/src/scala/com/github/johnynek/bazel_deps/parseproject_deploy.jar $out/bin/bazel-bin/src/scala/com/github/johnynek/bazel_deps
+    '';
+  };
+}


I'm not familiar with buildBazelPackage, but it seems like the meta attribute might be missing here.

orivej · 2018-07-06T14:31:15Z

pkgs/development/tools/build-managers/bazel/bazel-deps/default.nix

+        done
+        find . -type d -empty -delete
+      popd
+    '';


Could you add some Bash comments that explain the goals of this preInstall?

uri-canva · 2018-07-07T02:06:18Z

Thanks for the reviews, comments addressed.

orivej · 2018-07-07T02:13:57Z

@GrahamcOfBorg build bazel-deps

GrahamcOfBorg · 2018-07-07T02:14:28Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: bazel-deps

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-07T02:17:10Z

Failure on x86_64-linux (full log)

Attempted: bazel-deps

Partial log (click to expand)

+ cd /build/output/external
+ rm -rf /build/output/external/io_bazel_rules_scala /build/output/external/io_bazel_rules_scala
+ git clone git://github.com/bazelbuild/rules_scala /build/output/external/io_bazel_rules_scala
Cloning into '/build/output/external/io_bazel_rules_scala'...
fatal: Unable to look up github.com (port 9418) (Name or service not known)
INFO: Elapsed time: 1.015s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
builder for '/nix/store/ygpv94yyny7w0sh307l82fmwcp13zfmi-bazel-deps-2018-05-31.drv' failed with exit code 1
error: build of '/nix/store/ygpv94yyny7w0sh307l82fmwcp13zfmi-bazel-deps-2018-05-31.drv' failed

GrahamcOfBorg · 2018-07-07T02:17:36Z

Failure on x86_64-darwin (full log)

Attempted: bazel-deps

Partial log (click to expand)

owner=client
cwd=/private/tmp/nix-build-bazel-deps-2018-05-31-deps.drv-0/output/external/scalac_rules_protobuf_java

Waiting for it to complete...
.............
~/output/external ~/source
~/source
fixed-output derivation produced path '/nix/store/73k0mmh89swix0inbaja2a2q4n2d11kz-bazel-deps-2018-05-31-deps' with sha256 hash '080lz22irpqryrx09m66kv35aqz2fk1xsz4yhwz56h9rpny8d9ac' instead of the expected hash '0d33zh40wk2qm7z7a777470k8dqiki0dlj00mqrl475kydc0xfxc'
cannot build derivation '/nix/store/qb351ls5yhr2b7fmp1lcsxy4db23g5kk-bazel-deps-2018-05-31.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/qb351ls5yhr2b7fmp1lcsxy4db23g5kk-bazel-deps-2018-05-31.drv' failed

uri-canva · 2018-07-07T02:20:25Z

fatal: Unable to look up github.com (port 9418) (Name or service not known)

Is this because it uses git:// instead of https://? Are only port 80 and 443 allowed on hydra?

fixed-output derivation produced path '/nix/store/73k0mmh89swix0inbaja2a2q4n2d11kz-bazel-deps-2018-05-31-deps' with sha256 hash '080lz22irpqryrx09m66kv35aqz2fk1xsz4yhwz56h9rpny8d9ac'

That's exactly the hash I get on my Linux and Nixos boxes, so I must have something wrong on my darwin box.

orivej · 2018-07-07T02:26:23Z

fatal: Unable to look up github.com (port 9418) (Name or service not known)

Is this because it uses git:// instead of https://?

No, this is a temporary error in host name resolution. It almost never happens on properly set up hosts and I have not seen it from the bot before, but it is possible. (EDIT: /cc @grahamc)

That's exactly the hash I get on my Linux and Nixos boxes, so I must have something wrong on my darwin box.

Could you copy the store path from the Darwin box onto the Linux box (with rsync -a) and compare them (with git diff --no-index)?

uri-canva · 2018-07-07T02:33:38Z

Yeah I did that and the difference is on Darwin the deps don't include the source jars, just the code jars, but on Linux the deps include the source jars.

Other than that they're the same, and both are deterministic. I'm checking whether it has something to do with having Maven installed or not.

uri-canva · 2018-07-07T02:35:33Z

@GrahamcOfBorg build bazel-deps

uri-canva · 2018-07-07T02:38:14Z

I remember I saw this and thought it was suspicious: bazelbuild/bazel#4798.
That means that if one has built bazel-deps locally before, and the cache is reused, then the derivation will end up not having the source jars, and will not match the hash. I'm not sure where the cache would be reused from though, as the directory where nix builds is temporary isn't it?

orivej · 2018-07-07T03:04:19Z

@GrahamcOfBorg build bazel-deps

You will need to apply for this by sending a PR to https://github.com/NixOS/ofborg (see other PRs).

@GrahamcOfBorg build bazel-deps

I'm not sure where the cache would be reused from though, as the directory where nix builds is temporary isn't it?

Nix builds in a clean directory. Bazel may be guessing your home directory from your user id, or it may be communicating with the already running daemon. You may (or may not) be able to affect this by setting some environment variables (such as HOME, XDG_CACHE_HOME). Please try to figure this out.

GrahamcOfBorg · 2018-07-07T03:04:54Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: bazel-deps

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-07T03:07:08Z

Success on x86_64-darwin (full log)

Attempted: bazel-deps

Partial log (click to expand)

INFO: Elapsed time: 139.941s, Critical Path: 50.87s
INFO: 46 processes: 1 local, 41 processwrapper-sandbox, 4 worker.
INFO: Build completed successfully, 91 total actions
installing
post-installation fixup
strip is /nix/store/7ddbq63v97nk8gkbf7gcsfmby37h6gbl-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/rdvhrp2ig5phz518d1aif433y341hn0n-bazel-deps-2018-05-31/bin
patching script interpreter paths in /nix/store/rdvhrp2ig5phz518d1aif433y341hn0n-bazel-deps-2018-05-31
/nix/store/rdvhrp2ig5phz518d1aif433y341hn0n-bazel-deps-2018-05-31/bin/.gen_maven_deps.sh-wrapped: interpreter directive changed from "/bin/bash" to "/nix/store/q2wqq1k20v8kc3vckapqf5nws30brnni-bash-4.4-p23/bin/bash"
/nix/store/rdvhrp2ig5phz518d1aif433y341hn0n-bazel-deps-2018-05-31

GrahamcOfBorg · 2018-07-07T03:10:22Z

Failure on x86_64-linux (full log)

Attempted: bazel-deps

Partial log (click to expand)

+ cd /build/output/external
+ rm -rf /build/output/external/io_bazel_rules_scala /build/output/external/io_bazel_rules_scala
+ git clone git://github.com/bazelbuild/rules_scala /build/output/external/io_bazel_rules_scala
Cloning into '/build/output/external/io_bazel_rules_scala'...
fatal: Unable to look up github.com (port 9418) (Name or service not known)
INFO: Elapsed time: 2.334s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
builder for '/nix/store/ygpv94yyny7w0sh307l82fmwcp13zfmi-bazel-deps-2018-05-31.drv' failed with exit code 1
error: build of '/nix/store/ygpv94yyny7w0sh307l82fmwcp13zfmi-bazel-deps-2018-05-31.drv' failed

uri-canva · 2018-07-07T06:14:22Z

Found it, the state survives between one build and the next in the /private/var/tmp/_bazel_nixbld* directories, which are the user_output_root directories, which default to a subdirectory of the home directory on Linux, and a subdirectory of /var/tmp on macOS. The reason it worked on Linux but not macOS is that we made sure the state didn't survive the build by exporting HOME to point to NIX_BUILD_TOP, so that bazel would use a user_output_root directory under it. I added --user_output_root which is more explicit and works for Darwin. For some reason setting HOME is still necessary on Linux, possibly because it's a single user installation? Having both seems to cover all cases and have no issues. I have verified this by building the derivation and looking for /var/tmp/_bazel_nixbld* on macOS (multi user installation) and ~/.cache/bazel on Ubuntu (single user installation).

https://github.com/bazelbuild/bazel/blob/f19d2c102730df663cfc8aa58655dadbdaf8062d/src/main/cpp/blaze_util_linux.cc#L47

https://github.com/bazelbuild/bazel/blob/f19d2c102730df663cfc8aa58655dadbdaf8062d/src/main/cpp/blaze_util_darwin.cc#L95

uri-canva · 2018-07-07T06:15:02Z

I've also submitted NixOS/ofborg#192, thanks for running the builds for me in the meantime.

orivej · 2018-07-07T17:58:33Z

Failure on x86_64-linux

So this was not an accident, but a sandbox restriction: the fetch derivation (configured with fetchAttrs) did not produce everything needed (in this case io_bazel_rules_scala directory is missing from bazel-deps.deps output), so the build derivation attempted to download it and was denied by the sandbox.

uri-canva · 2018-07-09T10:34:40Z

Addressed review.

I haven't been able to build it on nixOS, because my nixOS box is slightly broken after the latest virtualbox update.

orivej · 2018-07-09T12:26:48Z

@GrahamcOfBorg build bazel-deps

GrahamcOfBorg · 2018-07-09T12:27:03Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: bazel-deps

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-07-09T12:34:25Z

Success on x86_64-darwin (full log)

Attempted: bazel-deps

Partial log (click to expand)

INFO: Elapsed time: 135.621s, Critical Path: 51.67s
INFO: 46 processes: 1 local, 41 processwrapper-sandbox, 4 worker.
INFO: Build completed successfully, 91 total actions
installing
post-installation fixup
strip is /nix/store/7ddbq63v97nk8gkbf7gcsfmby37h6gbl-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/f2mi4pyzpbfm8h0ab4wchwpl9vngg091-bazel-deps-2018-05-31/bin
patching script interpreter paths in /nix/store/f2mi4pyzpbfm8h0ab4wchwpl9vngg091-bazel-deps-2018-05-31
/nix/store/f2mi4pyzpbfm8h0ab4wchwpl9vngg091-bazel-deps-2018-05-31/bin/.gen_maven_deps.sh-wrapped: interpreter directive changed from "/bin/bash" to "/nix/store/q2wqq1k20v8kc3vckapqf5nws30brnni-bash-4.4-p23/bin/bash"
/nix/store/f2mi4pyzpbfm8h0ab4wchwpl9vngg091-bazel-deps-2018-05-31

GrahamcOfBorg · 2018-07-09T12:47:52Z

Success on x86_64-linux (full log)

Attempted: bazel-deps

Partial log (click to expand)

INFO: Build completed successfully, 91 total actions
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/29saxz4a6pcdqf1cqwxlkxhxbs806qkm-bazel-deps-2018-05-31
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/29saxz4a6pcdqf1cqwxlkxhxbs806qkm-bazel-deps-2018-05-31/bin
patching script interpreter paths in /nix/store/29saxz4a6pcdqf1cqwxlkxhxbs806qkm-bazel-deps-2018-05-31
/nix/store/29saxz4a6pcdqf1cqwxlkxhxbs806qkm-bazel-deps-2018-05-31/bin/.gen_maven_deps.sh-wrapped: interpreter directive changed from "/bin/bash" to "/nix/store/8zkg9ac4s4alzyf4a8kfrig1j73z66dw-bash-4.4-p23/bin/bash"
checking for references to /build in /nix/store/29saxz4a6pcdqf1cqwxlkxhxbs806qkm-bazel-deps-2018-05-31...
/nix/store/29saxz4a6pcdqf1cqwxlkxhxbs806qkm-bazel-deps-2018-05-31

orivej · 2018-07-09T22:40:38Z

Thank you!

Could you explain a bit, why did you find it easier to work without nix hacks than with them?

uri-canva · 2018-07-10T04:34:01Z

enableNixHacks does two things:

Makes Bazel assume the external workspaces are up to date by disabling the marker check.
Makes Bazel propagate environment variables by disabling environment sandboxing.

In this case neither was necessary, the external workspaces are up to date and pass the check, and the scala compilation does not use any environment variables from outside.

Before this PR the only derivation using buildBazelPackage was tensorflow, which is why the hacks weren't parametrised. It's possible once we build tensorflow from source without the wheel that derivation won't need the hacks at all.

uri-canva force-pushed the bazel-deps branch from 1ed62d9 to 9849bb2 Compare July 4, 2018 12:38

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Jul 4, 2018

ryantm requested changes Jul 6, 2018

View reviewed changes

orivej reviewed Jul 6, 2018

View reviewed changes

uri-canva force-pushed the bazel-deps branch from 9849bb2 to f537e01 Compare July 7, 2018 02:05

uri-canva force-pushed the bazel-deps branch from f537e01 to 9fe02c7 Compare July 7, 2018 02:34

GrahamcOfBorg added 10.rebuild-darwin: 1-10 and removed 10.rebuild-darwin: 0 labels Jul 7, 2018

uri-canva force-pushed the bazel-deps branch 2 times, most recently from eda961c to 6005815 Compare July 7, 2018 05:44

uri-canva force-pushed the bazel-deps branch from 6005815 to e4ff988 Compare July 9, 2018 03:23

uri-canva force-pushed the bazel-deps branch 2 times, most recently from 1f1f806 to bb12088 Compare July 9, 2018 06:52

bazel-deps: init at 2018-05-31

9059b52

uri-canva force-pushed the bazel-deps branch from bb12088 to 9059b52 Compare July 9, 2018 09:59

orivej-nixos merged commit ed98822 into NixOS:master Jul 9, 2018

uri-canva deleted the bazel-deps branch August 23, 2018 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bazel-deps: init at 2018-05-31 #43018

bazel-deps: init at 2018-05-31 #43018

uri-canva commented Jul 4, 2018 •

edited

ryantm Jul 6, 2018

orivej Jul 6, 2018

ryantm Jul 6, 2018

ryantm Jul 6, 2018

ryantm Jul 6, 2018

orivej Jul 6, 2018

uri-canva commented Jul 7, 2018

orivej commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

uri-canva commented Jul 7, 2018

orivej commented Jul 7, 2018 •

edited

uri-canva commented Jul 7, 2018

uri-canva commented Jul 7, 2018

uri-canva commented Jul 7, 2018

orivej commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

uri-canva commented Jul 7, 2018

uri-canva commented Jul 7, 2018

orivej commented Jul 7, 2018

uri-canva commented Jul 9, 2018

orivej commented Jul 9, 2018

GrahamcOfBorg commented Jul 9, 2018

GrahamcOfBorg commented Jul 9, 2018

GrahamcOfBorg commented Jul 9, 2018

orivej commented Jul 9, 2018

uri-canva commented Jul 10, 2018

bazel-deps: init at 2018-05-31 #43018

bazel-deps: init at 2018-05-31 #43018

Conversation

uri-canva commented Jul 4, 2018 • edited

Motivation for this change

Things done

ryantm Jul 6, 2018

Choose a reason for hiding this comment

orivej Jul 6, 2018

Choose a reason for hiding this comment

ryantm Jul 6, 2018

Choose a reason for hiding this comment

ryantm Jul 6, 2018

Choose a reason for hiding this comment

ryantm Jul 6, 2018

Choose a reason for hiding this comment

orivej Jul 6, 2018

Choose a reason for hiding this comment

uri-canva commented Jul 7, 2018

orivej commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

uri-canva commented Jul 7, 2018

orivej commented Jul 7, 2018 • edited

uri-canva commented Jul 7, 2018

uri-canva commented Jul 7, 2018

uri-canva commented Jul 7, 2018

orivej commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

GrahamcOfBorg commented Jul 7, 2018

uri-canva commented Jul 7, 2018

uri-canva commented Jul 7, 2018

orivej commented Jul 7, 2018

uri-canva commented Jul 9, 2018

orivej commented Jul 9, 2018

GrahamcOfBorg commented Jul 9, 2018

GrahamcOfBorg commented Jul 9, 2018

GrahamcOfBorg commented Jul 9, 2018

orivej commented Jul 9, 2018

uri-canva commented Jul 10, 2018

uri-canva commented Jul 4, 2018 •

edited

orivej commented Jul 7, 2018 •

edited