Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bazel: 0.13.0 -> 0.15.0 #42735

Merged
merged 3 commits into from Jul 17, 2018
Merged

bazel: 0.13.0 -> 0.15.0 #42735

merged 3 commits into from Jul 17, 2018

Conversation

mboes
Copy link
Contributor

@mboes mboes commented Jun 28, 2018

Motivation for this change

Upgrade to latest Bazel.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

Upgrade to latest Bazel.
@jirkadanek
Copy link
Member

What's the policy on upgrading of things in nixpkgs/pkgs/development/tools/build-managers/bazel/buildtools/? Is it preferred to keep it current with the bazel version as a matter of principle, or is it touched only if truly needed?

@jirkadanek
Copy link
Member

Cannot compile envoy with this 0.15

$ git clone git@github.com:envoyproxy/envoy.git
$ cd envoy

$ bazel build --verbose_failures --worker_verbose
INFO: Analysed 0 targets (0 packages loaded).
INFO: Found 0 targets...
ERROR: Process exited with status 126: Process exited with status 126
/bin/sh: bazel/get_workspace_status: /bin/bash: bad interpreter: No such file or directory
INFO: Elapsed time: 0.110s, Critical Path: 0.01s
FAILED: Build did NOT complete successfully

@xeji
Copy link
Contributor

xeji commented Jun 28, 2018

That issue is probaly not directly related to this PR.
envoy in nixpkgs is still pinned to bazel_0_4 because apparently it does not compile with newer versions. Don't know why though.

@jirkadanek
Copy link
Member

@xeji It is completely unrelated to PR. It is because envoy brings its own get_workspace_status script (in its repo) and that obviously is not patched when bazel is built... my mistake

@xeji
Copy link
Contributor

xeji commented Jun 28, 2018

@GrahamcOfBorg build bazel

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: bazel

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.


@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: bazel

Partial log (click to expand)

./tools/test/collect_coverage.sh: interpreter directive changed from "/bin/bash -x" to "/nix/store/xain263p9gxvb4h2a3cy2836k872qcgx-bash/bin/bash -x"
./tools/test/test-setup.sh: interpreter directive changed from "/bin/bash" to "/nix/store/xain263p9gxvb4h2a3cy2836k872qcgx-bash/bin/bash"
configuring
no configure script, doing nothing
building
Building Bazel from scratch....../usr/bin/xcrun clang -fobjc-arc -framework CoreServices -framework Foundation -o /tmp/bazel_F73NsEhq/archive/_embedded_binaries/xcode-locator tools/osx/xcode_locator.m
xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'.
/nix/store/7amnc2662p390bgzj6c0snylj19vxmj2-stdenv-darwin/setup: line 1291: ./output/bazel: No such file or directory
builder for '/nix/store/sc9vd5sj9y1j0kzi3w8bdm6xw30j70qy-bazel-0.15.0.drv' failed with exit code 127
�[31;1merror:�[0m build of '/nix/store/sc9vd5sj9y1j0kzi3w8bdm6xw30j70qy-bazel-0.15.0.drv' failed

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: bazel

Partial log (click to expand)

Analyzing: target //scripts:bash_completion (2 packages loaded)
Analyzing: target //scripts:bash_completion (54 packages loaded)
ERROR: /build/tools/android/BUILD:182:1: no such package '@desugar_jdk_libs//': Error downloading [https://github.com/google/desugar_jdk_libs/archive/f5e6d80c6b4ec6b0a46603f72b015d45cf3c11cd.zip] to /tmp/.bazel/7b7747ec045ae606eb720a1222f56098/external/desugar_jdk_libs/f5e6d80c6b4ec6b0a46603f72b015d45cf3c11cd.zip: Unknown host: github.com and referenced by '//tools/android:desugar_jdk_libs'
ERROR: Analysis of target '//scripts:bash_completion' failed; build aborted: Analysis failed
INFO: Elapsed time: 3.540s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (133 packages loaded)
FAILED: Build did NOT complete successfully (133 packages loaded)
builder for '/nix/store/wz99blk3qqdnfx4l7cdwfqpvs0gbqv83-bazel-0.15.0.drv' failed with exit code 1
error: build of '/nix/store/wz99blk3qqdnfx4l7cdwfqpvs0gbqv83-bazel-0.15.0.drv' failed

Copy link
Contributor

@xeji xeji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build fails, please fix

@kalbasit
Copy link
Member

kalbasit commented Jul 3, 2018

@mboes Have you been able to debug this issue? It's blocking me from moving to NixOS as we rely on Bazel 0.14 at work 😞

@jirkadanek
Copy link
Member

x86_64-linux fail seems environmental. not sure about the other two.

I've been using the 0.15 since the PR was submitted and it works fine for me (NixOS x86_64).

There is one bug in the NixOS package i found, though the bug was present in previous nix versions as well. The --spawn_strategy=linux-sandbox is broken. Bazel always check if the system supports this strategy, and the test is implemented as running /bin/true in a sandbox. NixOS obviously does not have /bin/true, so the linux-sandbox is then not made available.

See https://github.com/bazelbuild/bazel/blob/bc898cabe7cece0cf868447392f6863cb134d85c/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedSpawnRunner.java#L71-L73

@Profpatsch
Copy link
Member

I've been using the 0.15 since the PR was submitted and it works fine for me (NixOS x86_64).

If the build succeeds that xmeans you don’t have the nix sandbox enabled and bazel was able to access the network from inside the build and download the missing package. Hydra/ofborg have sandboxing enabled to keep builds deterministic.
@jdanekrh if you set sandbox = true in your /etc/nix/nix.conf (nix.useSandbox = true in a NixOS config), you have the same behavior locally.

@jirkadanek
Copy link
Member

@Profpatsch Thanks for the explanation. I knew about the sandbox (from the default github issue template) but I did not realize it blocks network, too, I thought it is only to restrict access to random files in the filesystem.

@Profpatsch
Copy link
Member

Okay, this small patch makes bazel fail before it starts the build in case there’s still dependencies missing.

tools/cpp/cc_configure and a few others reference a project called @bazel_tools, which doesn’t seem to be mentioned in the WORKSPACE file, so perhaps this is an oversight of upstream.
Will try to fix this tomorrow.

@Profpatsch
Copy link
Member

Profpatsch commented Jul 11, 2018

I think I have a pretty good grasp on the bootstrap now, and the build should be fixed by tomorrow. In the meantime, have some debugging notes:

   - Yet another try at fixing the bazel build
     - https://github.com/bazelbuild/bazel/pull/4860/commits/5d4a78883393ad411e1dfb9671581fab6b5519f1
       looked promising, but was not it.
     - The internal module `@bazel_tools` is not recognized in the last step of
       the bootstrap.
     - Bootstrap: first compile bazel once with javac
       (~scripts/bootstrap/compile.sh~), then set everything up, so it can be
       compiled again with itself; ~libblaze.jar~ is the output (~blaze~ is the
       name of bazel’s server part).
       In this it fails to setup the bazel_tools repository to the correct path,
       so bazel doesn’t know where to find these repositories when no fetching
       is allowed (which was not set by default!)
     - If ~$BAZEL~ is set it will use the given bazel, otherwise it will use
       ~libbblaze.jar~ in the function ~run_bazel_jar~ at the bottom of
       ~scripts/bootstrap/compile.js~.
     - bazel takes an ~install_base~ argument. This is where the bazel
       distribution should be located.
     - The ~tools.WORKSPACE~ file for ~bazel_tools~ contains the directive
       ~local_repository(name = "bazel_tools", path = __embedded_dir__ + "/embedded_tools")~
       with the magic symbol ~__embedded_dir__~. This symbol is replaced in
       ~WorkspaceFactory.java~ by the actual path of
       ~$install_base/_embedded_binaries"~, so the combined path where bazel
       searches for ~bazel_tools~ should be
       ~$install_base/_embedded_binaries/embedded_tools~
     - If all of that sounds a little ad-hoc, it is because it is. Also, it is
       wrapped in a few hundred lines of ad-hoc shell code.

@Profpatsch
Copy link
Member

cross ref: bazelbuild/bazel#4860 (comment)

There should probably be an issue upstream about adding fetch=no to the bootstrap and why it can’t find builtin packages if that’s set, which looks like a bug.

@GrahamcOfBorg build bazel

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Partial log (click to expand)

Merge failed

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Partial log (click to expand)

Merge failed

@Profpatsch
Copy link
Member

Uh, merge conflict because the file got 300% larger in the meantime. #42832

I think that should be reverted and reworked.

@Profpatsch
Copy link
Member

@GrahamcOfBorg build bazel

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: bazel

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.


@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: bazel

Partial log (click to expand)

//examples/java-native/src/test/java/com/example/myproject:hello         PASSED in 0.6s

Executed 2 out of 2 tests: 2 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
INFO: Build completed successfully, 22 total actions
installing
post-installation fixup
patching script interpreter paths in /nix/store/vgqphn5iy4wcrx2d934db4qgnn1qngn4-bazel-0.15.0
checking for references to /tmp in /nix/store/vgqphn5iy4wcrx2d934db4qgnn1qngn4-bazel-0.15.0...
/nix/store/vgqphn5iy4wcrx2d934db4qgnn1qngn4-bazel-0.15.0

@Profpatsch
Copy link
Member

Will smuggle this in here, since ofborg doesn’t work on issues (yet):
#30590 (comment)

@GrahamcOfBorg build bazel_0_4

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: bazel_0_4

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.


@Profpatsch
Copy link
Member

Oh, and
@GrahamcOfBorg eval

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: bazel_0_4

Partial log (click to expand)

	at com.sun.tools.javac.util.JavacMessages.lambda$add$0(JavacMessages.java:106)
	at com.sun.tools.javac.util.JavacMessages.getBundles(JavacMessages.java:125)
	... 15 more
�[32mINFO: �[0mBuilding complete.

�[1A�[KTarget //src:bazel failed to build
�[32mINFO: �[0mElapsed time: 474.710s, Critical Path: 311.48s

ERROR: Could not build Bazel
builder for ‘/nix/store/6im424r01pc1f4p6p735pks0416xd9rp-bazel-0.4.5.drv’ failed with exit code 1
error: build of ‘/nix/store/6im424r01pc1f4p6p735pks0416xd9rp-bazel-0.4.5.drv’ failed

@kalbasit
Copy link
Member

I would wait on merging this anyway until Bazel 0.15.1 is released. Bazel 0.15.0 is broken on Mac. See bazelbuild/bazel#5468 for context.

@Profpatsch
Copy link
Member

@kalbasit Nice catch. It looks like the patch version might be released today or tomorrow. I’d merge this now and do the transition to 0.15.1 as soon as it comes out. Then we see whether we can apply a cleaned up Mac support.

@Profpatsch
Copy link
Member

@mboes should we merge?

@mboes
Copy link
Contributor Author

mboes commented Jul 13, 2018

If merging this doesn't introduce any regressions for anyone, then yes. But if macOS used to work but now can't because of an upstream bug then let's wait for 0.15.1.

@mboes
Copy link
Contributor Author

mboes commented Jul 15, 2018

@Profpatsch FYI I found a way to be --sandbox compliant without disabling bash autocompletion. The trick is to use the --experimental_distdir flag (which incidentally, will be renamed to --distdir in v0.16).

@Profpatsch
Copy link
Member

@GrahamcOfBorg build bazel

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: bazel

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.


@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: bazel

Partial log (click to expand)

./tools/test/collect_coverage.sh: interpreter directive changed from "/bin/bash -x" to "/nix/store/xain263p9gxvb4h2a3cy2836k872qcgx-bash/bin/bash -x"
./tools/test/test-setup.sh: interpreter directive changed from "/bin/bash" to "/nix/store/xain263p9gxvb4h2a3cy2836k872qcgx-bash/bin/bash"
configuring
no configure script, doing nothing
building
Building Bazel from scratch....../usr/bin/xcrun clang -fobjc-arc -framework CoreServices -framework Foundation -o /tmp/.bazel-30001/bazel_QCHv7PZS/archive/_embedded_binaries/xcode-locator tools/osx/xcode_locator.m
xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'.
/nix/store/7amnc2662p390bgzj6c0snylj19vxmj2-stdenv-darwin/setup: line 1291: ./output/bazel: No such file or directory
builder for '/nix/store/v4aj0yr802r1jvs6aq58aq9d0zl2aw3z-bazel-0.15.0.drv' failed with exit code 127
�[31;1merror:�[0m build of '/nix/store/v4aj0yr802r1jvs6aq58aq9d0zl2aw3z-bazel-0.15.0.drv' failed

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: bazel

Partial log (click to expand)

//examples/java-native/src/test/java/com/example/myproject:hello         PASSED in 0.3s

Executed 2 out of 2 tests: 2 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
INFO: Build completed successfully, 22 total actions
installing
post-installation fixup
patching script interpreter paths in /nix/store/wfw5h52dm7yckh76hyw41jpqcrim4alh-bazel-0.15.0
checking for references to /tmp/.bazel-1000 in /nix/store/wfw5h52dm7yckh76hyw41jpqcrim4alh-bazel-0.15.0...
/nix/store/wfw5h52dm7yckh76hyw41jpqcrim4alh-bazel-0.15.0

@kalbasit
Copy link
Member

Bazel 0.15.1 is now out, can we get this PR updated and merged?

@Profpatsch
Copy link
Member

@kalbasit I did push the update to the branch a few minutes ago, but it didn’t appear here. I think Github was unavailable a few minutes ago, maybe the servers are still catching up.

@Profpatsch
Copy link
Member

@GrahamcOfBorg build bazel

@GrahamcOfBorg
Copy link

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: bazel

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.


@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: bazel

Partial log (click to expand)

./tools/test/collect_coverage.sh: interpreter directive changed from "/bin/bash -x" to "/nix/store/xain263p9gxvb4h2a3cy2836k872qcgx-bash/bin/bash -x"
./tools/test/test-setup.sh: interpreter directive changed from "/bin/bash" to "/nix/store/xain263p9gxvb4h2a3cy2836k872qcgx-bash/bin/bash"
configuring
no configure script, doing nothing
building
Building Bazel from scratch....../usr/bin/xcrun clang -fobjc-arc -framework CoreServices -framework Foundation -o /tmp/.bazel-30001/bazel_SgiPW7m3/archive/_embedded_binaries/xcode-locator tools/osx/xcode_locator.m
xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'.
/nix/store/7amnc2662p390bgzj6c0snylj19vxmj2-stdenv-darwin/setup: line 1291: ./output/bazel: No such file or directory
builder for '/nix/store/csjzj361w5rjf1ml0c3f2ilzb018hcip-bazel-0.15.1.drv' failed with exit code 127
�[31;1merror:�[0m build of '/nix/store/csjzj361w5rjf1ml0c3f2ilzb018hcip-bazel-0.15.1.drv' failed

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: bazel

Partial log (click to expand)

//examples/java-native/src/test/java/com/example/myproject:hello         PASSED in 0.2s

Executed 2 out of 2 tests: 2 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.
INFO: Build completed successfully, 22 total actions
installing
post-installation fixup
patching script interpreter paths in /nix/store/zzvg2cql1pi00ickrdcidyg1bpf206r7-bazel-0.15.1
checking for references to /tmp/.bazel-1000 in /nix/store/zzvg2cql1pi00ickrdcidyg1bpf206r7-bazel-0.15.1...
/nix/store/zzvg2cql1pi00ickrdcidyg1bpf206r7-bazel-0.15.1

@Profpatsch
Copy link
Member

Okay, I think we are ready to merge!

@Profpatsch Profpatsch merged commit 75942f9 into NixOS:master Jul 17, 2018
@Profpatsch
Copy link
Member

cc @uri-canva Now we can try to fix the Darwin build, it should in principle work with 0.15.1.

mboes added a commit to tweag/rules_haskell that referenced this pull request Jul 21, 2018
We no longer need to pull bazel from cachix, now that
NixOS/nixpkgs#42735 has been merged and has
hit the cache.nixos.org cache.
mboes added a commit to tweag/rules_haskell that referenced this pull request Jul 23, 2018
We no longer need to pull bazel from cachix, now that
NixOS/nixpkgs#42735 has been merged and has
hit the cache.nixos.org cache.
thumphries pushed a commit to thumphries/rules_haskell that referenced this pull request Jul 27, 2018
We no longer need to pull bazel from cachix, now that
NixOS/nixpkgs#42735 has been merged and has
hit the cache.nixos.org cache.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants