Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bazel-remote: init at 2019-01-12 #53840

Merged
merged 1 commit into from Mar 11, 2019
Merged

bazel-remote: init at 2019-01-12 #53840

merged 1 commit into from Mar 11, 2019

Conversation

uri-canva
Copy link
Contributor

@uri-canva uri-canva commented Jan 12, 2019 •
edited

Motivation for this change

Adds bazel-remote. This the implementation for a bazel cache http backend the official docs refer to, though it's experimental: https://docs.bazel.build/versions/master/remote-caching.html#bazel-remote-cache. It's authored by a member of the Bazel team, even though it's not published under the bazelbuild org because it's not officially supported. There is no officially supported self-hostable caching backend though.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

uri-canva
uri-canva commented Jan 12, 2019
View reviewed changes
pkgs/build-support/build-bazel-package/default.nix
@@ -46,6 +46,7 @@ in stdenv.mkDerivation (fBuildAttrs // {

# Patching markers to make them deterministic
sed -i 's, -\?[0-9][0-9]*$, 1,' $bazelOut/external/\@*.marker
sed -i '/^ENV:TMP.*/d' $bazelOut/external/\@*.marker
Copy link
Contributor Author

@uri-canva uri-canva Jan 12, 2019 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can change the hash of the dependencies, I've tested pythonPackages.tensorflow and it still matches, but both bazel-deps and bazel-watcher don't build with the version of Bazel we have on master so I can't check.

@Profpatsch
Copy link
Member

@GrahamcOfBorg build bazel-remote

@Profpatsch
Copy link
Member

Build fails for linux and darwin (not sure why the status is “neutral”).

@uri-canva
Copy link
Contributor Author

Getting the same issue in #51723 (comment) as well since it uses gazelle too, will continue looking at it.

@uri-canva
Copy link
Contributor Author

Done. Applied the same fix as in #51723 (comment).

@Profpatsch
Copy link
Member

@GrahamcOfBorg build bazel-remote

@Profpatsch
Copy link
Member

Nope, doesn’t build (on Linux):

ERROR: /build/source/config/BUILD.bazel:3:1: no such package '@in_gopkg_yaml_v2//': failed to fetch in_gopkg_yaml_v2: 2019/01/19 10:24:31 unrecognized import path "gopkg.in/yaml.v2"
 and referenced by '//config:go_default_library'
ERROR: /build/source/BUILD.bazel:11:1: no such package '@com_github_abbot_go_http_auth//': failed to fetch com_github_abbot_go_http_auth: # cd .; git clone https://github.com/abbot/go-http-auth /build/output/external/com_github_abbot_go_http_auth
Cloning into '/build/output/external/com_github_abbot_go_http_auth'...
error: could not lock config file /build/output/external/com_github_abbot_go_http_auth/.git/config: No such file or directory
fatal: could not set 'remote.origin.fetch' to '+refs/heads/*:refs/remotes/origin/*'
2019/01/19 10:24:31 exit status 128
 and referenced by '//:go_default_library'
ERROR: /build/source/config/BUILD.bazel:3:1: no such package '@in_gopkg_yaml_v2//': failed to fetch in_gopkg_yaml_v2: 2019/01/19 10:24:31 unrecognized import path "gopkg.in/yaml.v2"
 and referenced by '//config:go_default_library'
ERROR: Evaluation of query "deps(//:bazel-remote)" failed: errors were encountered while computing transitive closure

Also I’m confused about the ofborg build error for linux, we don’t use oraclejdk for bazel.

@uri-canva
Copy link
Contributor Author

Sorry I updated the ubuntu box so it didn't have sandboxing configured anymore. Could repro the failure with sandboxing enabled, will fix.

@uri-canva uri-canva force-pushed the bazel-remote branch 2 times, most recently from 988b83c to e8c28d4 Compare January 22, 2019 00:55
@uri-canva
Copy link
Contributor Author

Can't figure out why it's failing. Is there a way to run the sandboxed build in an interactive way? NixOS/nix#903 seems to imply there isn't.

@Profpatsch
Copy link
Member

Profpatsch commented Jan 23, 2019 •
edited

On linux you could try dropping yourself into a nix shell with the network namespace unshared:

sudo unshare --user --net -- nix-shell -A bazel /path/to/nixpkgs

Attention, you are root inside that shell. If it needs to download dependencies, first go into the shell once without unsharing. --user makes it a user namespace.

@uri-canva
Copy link
Contributor Author

Won't have time to debug why it's failing on Linux, but I changed the supported platforms to be only Darwin so we can still use it as it is now.

@uri-canva
Copy link
Contributor Author

ping @Profpatsch

@Profpatsch
Copy link
Member

Ah, sorry. Merged.

@Profpatsch Profpatsch merged commit 130d987 into NixOS:master Mar 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Profpatsch Profpatsch Awaiting requested review from Profpatsch Profpatsch is a code owner

Assignees
No one assigned
Labels
8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 0 11.by: package-maintainer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@uri-canva @Profpatsch @GrahamcOfBorg