there is already #58116

there is already #58116

Haha! I should have checked! I know @uri-canva ;)

Ok, I'm going to try again.

@marsam @Profpatsch
I'm looking at this line: https://github.com/NixOS/nixpkgs/pull/58147/files#diff-4746b81564a01e241a84d455575437e7R244

Would that usage of TMP cause sandbox builds to fail?

Locally, I get a different error to hydra. I get the following when running with sandbox = true:
mkdir: cannot create directory '/tmp/.bazel-501': Operation not permitted

When running with sandbox = false it appears to build on Darwin if I create a fake directory /blah with 777 chmod and change that line to use /blah

Hydra timing out after 600 seconds of silence, might be because the compilation takes longer than 10 minutes? I suspect that maybe hydra doesn't have the same TERM or something to see the fancy colored Bazel compilation progress. It takes at least 20-30 minutes to build.

I've hacked the derivation a bit and it's almost working now. Failing at the installCheckPhase.

In file included from examples/cpp/hello-world.cc:1:
./examples/cpp/hello-lib.h:4:10: fatal error: 'string' file not found
#include <string>
         ^~~~~~~~
1 error generated.
INFO: Elapsed time: 34.407s, Critical Path: 0.16s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
//examples/java-native/src/test/java/com/example/myproject:hello      NO STATUS

I'll continue looking at this tomorrow. In the meantime, if anybody can review my changes and suggest better ways to do it, I'm happy to modify things within reason.

Note, my error above is with sandbox = false (I suspect it's a CPP or C toolchain issue).

If I build with sandbox = true I get failed with exit code 134 so I suspect I'm doing something that is banned in sandbox mode.

It might be because you move .bazelrc into bazel_src and then run the tests from outside it, so the tests don't pick up .bazelrc, which is necessary for the bazel build to pick up the right flags from nix.

It might be because you move .bazelrc into bazel_src and then run the tests from outside it, so the tests don't pick up .bazelrc, which is necessary for the bazel build to pick up the right flags from nix.

Good point. I'll check this. I thought that I was copying the .bazelrc along for the ride and then cd ./bazel_src would make it available. I'll try to debug.

Ok, this is now building successfully on my Mac with sandbox = false.

With sandbox = true I still get failed with exit code 134

Exit codes higher than 128 are usually from signals, 134-128 = 6 is SIGABRT. Do you know what command is aborting? Is it nix itself or one of the bazel commands?

Exit codes higher than 128 are usually from signals, 134-128 = 6 is SIGABRT. Do you know what command is aborting? Is it nix itself or one of the bazel commands?

Yes, I suspect the same thing.

Building Bazel from scratch
builder for '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed with exit code 134
error: build of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed

Patched my build to output tracing. I think this is the reason why I fail to build with sandbox = true.

...
++ test -z /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5
++ JAVAC=/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac
++ [[ -x /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac ]]
+++ /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac -version
++ JAVAC_VERSION='dyld: Library not loaded: /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
  Referenced from: /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac
  Reason: no suitable image found.  Did find:
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()'
+ run_atexit_handlers 134
+ local exit_code=134
...

@GrahamcOfBorg build bazel

Darwin build is timing out.

There's headless versions of jdk11 and jre for Linux. If we could have headless versions of jdk8 for macOS, then the build wouldn't attempt to access Cocoa and it might work. Did something change recently with how sandboxing is enforced? I think system libraries on macOS were still being access in non-pure fashion until a couple of months ago.

@Profpatsch

Darwin build is timing out.

I see that. What would you suggest now? This is the same as others have seen trying to get a new version of Bazel over the line. How can we get more build output? On my own Mac, I see a lot more output.

Given the local sandbox failure loading Cocoa, I think it's possible that it never built successfully on Darwin with sandbox = true or the sandboxing mechanisms have changed recently.

I'll keep digging, but it's looking like this might have to be changed to a linux only pkg if sandboxed Darwin builds are a blocker.

Quick post to summarise the difference between failing sandbox builds on a Mac vs on Hydra.

Mac (Tracing disabled. The underlying failure is due to a javac call reaching outside the sandbox for Cocoa libraries):

fixing libtool script ./third_party/protobuf/3.6.1/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googletest/build-aux/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googlemock/build-aux/ltmain.sh
no configure script, doing nothing
preBuildPhase
building
Building Bazel from scratch
builder for '/nix/store/aqg0l592dmr8ny5xgiiv3k3vbgdxg0lz-bazel-0.23.2.drv' failed with exit code 134
error: build of '/nix/store/aqg0l592dmr8ny5xgiiv3k3vbgdxg0lz-bazel-0.23.2.drv' failed

Hydra (https://logs.nix.ci/?key=nixos/nixpkgs.58147&attempt_id=2a30e2bb-eabb-41f2-8786-a53b3be2509a):

fixing libtool script ./third_party/protobuf/3.6.1/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googlemock/build-aux/ltmain.sh
fixing libtool script ./third_party/protobuf/3.6.1/third_party/googletest/googletest/build-aux/ltmain.sh
no configure script, doing nothing
preBuildPhase
building
building of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' timed out after 600 seconds of silence
error: build of '/nix/store/14gpkg2y5fg7my3kc5fpjm0qfcbm22vz-bazel-0.23.2.drv' failed

Note the missing "Building Bazel from scratch" output in Hydra. That line comes from ./bazel_src/compile.sh. Either we are losing logging somehow or buildPhase isn't even starting.

I'm going to try enable tracing and kick off a build, but I'm unsure if the bot will listen to me.

@GrahamcOfBorg build bazel

Just in case you're curious, even if I patch away the line in the build where it initially sets JAVAC_VERSION to hardcode "javac 1.8" it still fails with sandbox = true. This is failing with the core invocation of javac to compile the Bazel java classes.

/nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac -classpath third_party/truth8/truth-java8-extension-0.42.jar:third_party/javax_annotations/javax.annotation-api-1.3.2.jar:third_party/javax_annotations/javax.annotation-api-1.3.2-sources.jar
<SNIP ... />
-sourcepath src/java_tools/singlejar/java/com/google/devtools/build/zip:src/main/java:src/tools/xcode-common/java/com/google/devtools/build/xcode/common:src/tools/xcode-common/java/com/google/devtools/build/xcode/util:tools/java/runfiles:third_party/java/dd_plist/java:/private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_S1B9kq7r/src -d /private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_S1B9kq7r/classes -source 1.8 -target 1.8 -encoding UTF-8 @/private/var/folders/0f/v0lxnbcx4v94rbhtz37d6f2m0000gn/T/nix-build-bazel-0.23.2.drv-0/bazel_nuJxfgmI/param
dyld: Library not loaded: /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa
  Referenced from: /nix/store/9qyzw01bnla43457iwp6l9r6fbp7gk4y-zulu1.8.0_121-8.20.0.5/bin/javac
  Reason: no suitable image found.  Did find:
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
        /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa: file system sandbox blocked stat()
++ exit 0
+ run_atexit_handlers 0
+ local exit_code=0
+ local failed=no
+ for handler in ${ATEXIT_HANDLERS}

Apologies for all the spamming.

The above indicates to me that it is not possible to compile Java applications on Darwin with sandbox = true unless there is a way to prevent javac from reaching out for Cocoa. I hope that I'm mistaken. Does anybody know of any other Java applications that are building on Darwin with sandboxing enabled?

If not, I think this leaves Bazel 0.23.2 on Darwin dead in the water as an official Nixpkg unless a headless jdk comes along for Darwin.

0.24.0 was just released. @Profpatsch any ideas how we can debug this issue? It looks like the issues we get when building locally on macOS are completely different from building in hydra on macOS to be honest, I'm not sure spending time fixing what we're encountering locally is even helpful.

I can close this one if people wish to try 0.24.0. I've just attempted it and it's failing to compile for me on OSX, so I don't think it's ready just yet.

I'm not sure spending time fixing what we're encountering locally is even helpful.
I think it would be great to see why Hydra isn't outputting any logs once it starts the build. My current branch has tracing enabled, so if we can get that to run on Hydra, we'll at least see where it stops.

I agree that the Hydra Darwin builds outputting no logs could be a different issue / PR though. That assumes it's a general issue with Hydra and Darwin and not something specific to building Bazel.

Status update.

Building successfully on Darwin with these settings in /etc/nix/nix.conf:

sandbox = true
extra-sandbox-paths = /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib

Failing on linux. Investigating now, but it looks like a standard issue.

@groodt the linux build succeeds when the sandbox is turned off (it already is on Darwin). The reason is that in order to be sandbox friendly, you need to prefetch dependencies, since there is no network access inside the sandbox. We already need to do this, so you probably just need to add new tarballs to srcDeps (grep Bazel's default.nix).

@mboes This is true somewhat, but if you look at the game of whack-a-mole I've been playing in this thread, it's not so straightforward. Ofborg has been running with sandboxing enabled, so Darwin builds have been "timing out" (actually SIGABRT sandbox errors).

This resulted in us chasing our tails and making changes to the already working derivation of 0.22. It might be that some of these changes I've made to make Darwin sandbox happy, are causing it to fail on linux.

The early builds of this derivation were passing on linux and "timed out" on Darwin. Now it is working on Darwin and failing on Linux. 👍

Anyway, it's probably just a matter of downloading additional files or rolling back some of my changes that made Darwin sandbox happy. I'll take a look today.

One thing I thought of in the meantime: skipping 0.23.x could make it harder for users to upgrade, since Bazel keeps only one version of backwards compatibility. We should have two separate upgrade commits, even if they're one right after the other, so that people who pin nixpkgs can upgrade to 0.23.2 first and then to 0.24.0.

I reopened #58116 in case we want to go ahead and merge that first.

Having a Hydra build of bazel 0.23.2 for pinning sounds good to me.

I'm currently testing linux sandboxed builds of this PR, but I think it's also ready now.

This is now building and ready for review @Profpatsch @mboes Got there in the end!

Please see @uri-canva comment about thoughts of doing 0.22 -> 0.23.2 first. I don't mind if that is done first. I don't see too much benefit in sticking to an old Bazel for very long, so we should get this one merged shortly after if possible.

See below table for how I've tested this derivation on local Darwin and Linux. I'm confident that it will work on Hydra now. Linux should work on Ofborg and Hydra. Darwin might fail on Ofborg, but should succeed on Hydra given the testing that I've done.

@GrahamcOfBorg build bazel

Yep, we should merge even if we skip 0.23, the releases are too fast for the manpower we have here.

Thanks for the great change!

Should this be backported to 19.03 as well?

Should this be backported to 19.03 as well?

I'd like this too, but it's a bit late in the game for 19.03 I think. They've been working really hard to get it out the door. I don't know what the policy is for late inclusions. I guess with Nix it should be safe, but 🤷‍♂️

Nah, it’s a major, incompatible update, so we shouldn’t backport it.