Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bazel 0.21.0 #53412

Merged
merged 2 commits into from Jan 10, 2019
Merged

Bazel 0.21.0 #53412

merged 2 commits into from Jan 10, 2019

Conversation

Profpatsch
Copy link
Member

First commit is a patch created by @dtzWill to improve the substituteAll stuff.

Second commit is my progress of getting 0.21 to run so far. 0.21 removed the bundled openjdk-distribution. Instead, tries to fetch the “right” distribution on-the-fly when building. So we need to provide our own openjdk.

According to bazelbuild/bazel#6865 (comment) we should set --host_javabase if we want to do that.

We copy the buildJdk nix path to a temporary directory, make it look like a bazel repository (init BUILD and WORKSPACE) and point to it via local_repository.
As it turns out, exposing the files via filegroup is not enough, the bazel build needs it to expose a JavaRuntimeInfo provider:

ERROR: /tmp/.bazel-1000/bazel_HPQZBqno/out/external/bazel_tools/tools/jdk/BUILD:19:1:
in :alias attribute of java_host_runtime_alias rule
@bazel_tools//tools/jdk:current_host_java_runtime:
'@nix_jdk//:all-files' does not have mandatory providers:
'JavaRuntimeInfo'

I am unsure how to achieve that.

This commit also replaces the line-number-based sed invocations with
something more stable.

I actually tried to follow the upstream suggestion first and use --host_javabase='@local_jdk//:jdk', but that did not work at all:

ERROR: /build/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/BUILD:82:1: error executing shell command: '/nix/store/l04m8mvir9ljd9dngnkxxgmw83wrxzs2-bash/bin/bash -c set -e;rm -rf bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output;mkd...' failed (Exit 127): bash failed: error executing command
  (cd /tmp/.bazel-1000/bazel_18rkBpuk/out/execroot/io_bazel && \
  exec env - \
    PATH=/nix/store/can00lfiynqkbsdkkmgp6qg8p8w92cxa-bash-4.4-p23/bin:/nix/store/0q4i5ll9gxs6giq7kqkniww934j9j8dk-coreutils-8.30/bin:/nix/store/izhcvddpv3n052wp82zm5s8zcg514kr3-findutils-4.6.0/bin:/nix/store/zzzq8a9af192wfsi7lvf0mndpc8ykp4q-gnugrep-3.1/bin:/nix/store/4q0b6gz1yvb4bdzfcbyicz3vmlq05nxa-gnused-4.5/bin:/nix/store/c2czqxjf05amp81si5lvjxswx2bd7j6a-which-2.21/bin \
  /nix/store/l04m8mvir9ljd9dngnkxxgmw83wrxzs2-bash/bin/bash -c 'set -e;rm -rf bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output;mkdir bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/libbootstrap.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/libskylark-deps.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/auto/auto-value-1.5.4.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/error_prone/error_prone_annotations-2.2.0.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/guava/guava-25.1-jre.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/jcip_annotations/jcip-annotations-1.0-1.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn third_party/jsr305/jsr-305.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
unzip -qn bazel-out/host/bin/src/main/java/com/google/devtools/build/lib/shell/libshell-skylark.jar -d bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
external/local_jdk/bin/jar cmf bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_MANIFEST.MF bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar -C bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output .
touch bazel-out/host/bin/src/java_tools/singlejar/java/com/google/devtools/build/singlejar/bootstrap_deploy.jar.build_output
')
Execution platform: @bazel_tools//platforms:host_platform

@local_jdk looks like a magic repository, from what I’ve seen it also has special support in bazel source code. Maybe it’s still the way to go.

@mboes @uri-canva @guibou, maybe you can find a way to get this to work.

@dtzWill @Profpatsch
bazel: greatly reduce time spent substituting, be smart which files
2dc6e99 
Files inspected: 2756 -> 40
Total size of inspected files: 20M -> 1016K
Profpatsch referenced this pull request in dtzWill/nixpkgs Jan 4, 2019
@dtzWill
bazel: greatly reduce time spent substituting, be smart which files
9ed2c64 
Files inspected: 2756 -> 40
Total size of inspected files: 20M -> 1016K
@Profpatsch Profpatsch mentioned this pull request Jan 4, 2019
Merged
@Profpatsch
Copy link
Member Author

Profpatsch commented Jan 8, 2019 •
edited

It was brought to my attention that there exists an undocumented built-in java_runtime rule that should provide the JavaRuntimeInfo provider and thus hopefully the last missing piece to get this working.

https://github.com/bazelbuild/bazel/blob/master/src/main/tools/jdk.BUILD#L166-L174

@Profpatsch
bazel: 0.20.0 -> 0.21.0
581bc7f 
0.21 removed the bundled openjdk-distribution. Instead, tries to fetch
the “right” distribution on-the-fly when building.
So we need to provide our own openjdk.

According to
bazelbuild/bazel#6865 (comment)
we should set `--host_javabase="@local_jdk//:jdk` if we want to do
that. This uses the jdk that is currently in the environment, which is
openjdk 8 in our case. 0.21 defaulted to a toolchain for JDK9, which
we don’t package in nixpkgs, so we use the JDK8 toolchain.

This commit also replaces the line-number-based sed invocations with
something more stable.
@Profpatsch
Copy link
Member Author

Adding java_runtime does work, but has the same effect as just using @local_jdk//:jdk, leading back to the same build failure. The solution was to add unzip to defaultShellPath in addition to nativeBuildInputs.

@GrahamcOfBorg build bazel

@Profpatsch Profpatsch changed the title WIP: Bazel 0.21.0 Bazel 0.21.0 Jan 10, 2019
@Profpatsch Profpatsch merged commit 9e9fec6 into NixOS:master Jan 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Profpatsch @GrahamcOfBorg @dtzWill