Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bazel: move darwin ad hoc code to darwin patch #45512

Closed

Conversation

guibou
Copy link
Contributor

@guibou guibou commented Aug 23, 2018

Commit 8bf1a5c added theses lines to
fix the darwin build, but they were added to the patch for all systems
instead of the patch limited to darwin.

Motivation for this change

When trying to upgrade to bazel 0.16 using an override, I had to remove theses line to get a successful build on linux.

Using git blame, I discovered that they were added by commit 8bf1a5c as a fix for darwin. It works on linux without these lines, so I move them to the appropriate darwinPatch section.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

Commit 8bf1a5c added theses lines to
fix the darwin build, but they were added to the patch for all systems
instead of the patch limited to darwin.
@GrahamcOfBorg GrahamcOfBorg added the 6.topic: darwin Running or building packages on Darwin label Aug 23, 2018
@guibou
Copy link
Contributor Author

guibou commented Aug 23, 2018

@Profpatsch I know you care about bazel

@Profpatsch
Copy link
Member

@uri-canva Was there a good reason this block went into the general configuration?
In your original patch it was pretty hard to spot the point where darwin-only and the general patch phase were concatenated, I have tidied that up in the meantime.

@uri-canva
Copy link
Contributor

uri-canva commented Aug 24, 2018

Yes, I added it to the general configuration because it's not a workaround for some Darwin specific issue, but rather a way to pass the cc configuration that Nix prepares to Bazel. If we don't do that explicitly, Bazel will clear the environment when invoking the toolchain, and the configuration will be lost. The reason it worked on Linux and not Darwin is that only Darwin required building native code at that time.

I'd like it to remain in the general configuration for two reasons:

  1. If native code is required to build Bazel on Linux, it can use the same patch. If we find a better way of doing it on Linux, chances are it will work on Darwin as well.
  2. If Bazel is changed in a way that the patch breaks the build, people are more likely to notice and fix it if it's part of the general configuration, rather than the Darwin specific one.

@uri-canva
Copy link
Contributor

@guibou The patch has hardcoded line numbers, it's very likely the line numbers need to be updated. If that is the case, moving the patch from general to Darwin specific will fix the Linux build, but will break the Darwin build.

@Profpatsch
Copy link
Member

The patch has hardcoded line numbers

Uh, what? Can we please not hardcode line numbers?

If that is the case, moving the patch from general to Darwin specific will fix the Linux build, but will break the Darwin build.

If it is written in a way that might break one build but not fix the other, it should doubly not be inside the general configuration.

@guibou
Copy link
Contributor Author

guibou commented Aug 24, 2018

@uri-canva I was actually trying to override the derivation to build bazel 0.17rc1 when I hit on these lines which were incompatible with that version of bazel. Actually I realize I did really poor engineering by submitting this PR and moving my problem to you.

If that is the case, moving the patch from general to Darwin specific will fix the Linux build, but will break the Darwin build.

Actually you are right, I was doing exactly that and I think darwin build will break on future bazel release.

Could you try to build bazel 0.17rc using this changes?:

diff --git a/pkgs/development/tools/build-managers/bazel/default.nix b/pkgs/development/tools/build-managers/bazel/default.nix
index 8d4b95c8808..d7933d3b1ae 100644
--- a/pkgs/development/tools/build-managers/bazel/default.nix
+++ b/pkgs/development/tools/build-managers/bazel/default.nix
@@ -11,8 +11,8 @@
 let
   srcDeps = lib.singleton (
     fetchurl {
-      url = "https://github.com/google/desugar_jdk_libs/archive/f5e6d80c6b4ec6b0a46603f72b015d45cf3c11cd.zip";
-      sha256 = "c80f3f3d442d8a6ca7adc83f90ecd638c3864087fdd6787ffac070b6f1cc8f9b";
+       url = "https://github.com/google/desugar_jdk_libs/archive/fd937f4180c1b557805219af4482f1a27eb0ff2b.zip";
+       sha256 = "43b8fcc56a180e178d498f375fbeb95e8b65b9bf6c2da91ae3ae0332521a1a12";
     }
   );
 
@@ -26,7 +26,7 @@ let
 in
 stdenv.mkDerivation rec {
 
-  version = "0.15.2";
+  version = "0.17.0rc1";
 
   meta = with lib; {
     homepage = "https://github.com/bazelbuild/bazel/";
@@ -39,8 +39,8 @@ stdenv.mkDerivation rec {
   name = "bazel-${version}";
 
   src = fetchurl {
-    url = "https://github.com/bazelbuild/bazel/releases/download/${version}/bazel-${version}-dist.zip";
-    sha256 = "1w83zi6d9npi1jmiy022v92xp1cwdvn2qqgghlnl2v9sprryqlxz";
+    url = "https://releases.bazel.build/0.17.0/rc1/bazel-0.17.0rc1-dist.zip";
+    sha256 = "46dfffac884ccd51fcb493dd86463cb8c21be949fdb17634ca37805fd544beae";
   };
 
   sourceRoot = ".";
@@ -119,12 +119,15 @@ stdenv.mkDerivation rec {
       echo "fetch --experimental_distdir=${distDir}" >> .bazelrc
       echo "build --copt=\"$(echo $NIX_CFLAGS_COMPILE | sed -e 's/ /" --copt=\"/g')\"" >> .bazelrc
       echo "build --host_copt=\"$(echo $NIX_CFLAGS_COMPILE | sed -e 's/ /" --host_copt=\"/g')\"" >> .bazelrc
-      echo "build --linkopt=\"-Wl,$(echo $NIX_LDFLAGS | sed -e 's/ /" --linkopt=\"-Wl,/g')\"" >> .bazelrc
-      echo "build --host_linkopt=\"-Wl,$(echo $NIX_LDFLAGS | sed -e 's/ /" --host_linkopt=\"-Wl,/g')\"" >> .bazelrc
-      sed -i -e "361 a --copt=\"$(echo $NIX_CFLAGS_COMPILE | sed -e 's/ /" --copt=\"/g')\" \\\\" scripts/bootstrap/compile.sh
-      sed -i -e "361 a --host_copt=\"$(echo $NIX_CFLAGS_COMPILE | sed -e 's/ /" --host_copt=\"/g')\" \\\\" scripts/bootstrap/compile.sh
-      sed -i -e "361 a --linkopt=\"-Wl,$(echo $NIX_LDFLAGS | sed -e 's/ /" --linkopt=\"-Wl,/g')\" \\\\" scripts/bootstrap/compile.sh
-      sed -i -e "361 a --host_linkopt=\"-Wl,$(echo $NIX_LDFLAGS | sed -e 's/ /" --host_linkopt=\"-Wl,/g')\" \\\\" scripts/bootstrap/compile.sh
+      
+      echo "build --java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8" >> .bazelrc
+
+      #echo "build --linkopt=\"-Wl,$(echo $NIX_LDFLAGS | sed -e 's/ /" --linkopt=\"-Wl,/g')\"" >> .bazelrc
+      #echo "build --host_linkopt=\"-Wl,$(echo $NIX_LDFLAGS | sed -e 's/ /" --host_linkopt=\"-Wl,/g')\"" >> .bazelrc
+      #sed -i -e "361 a --copt=\"$(echo $NIX_CFLAGS_COMPILE | sed -e 's/ /" --copt=\"/g')\" \\\\" scripts/bootstrap/compile.sh
+      #sed -i -e "361 a --host_copt=\"$(echo $NIX_CFLAGS_COMPILE | sed -e 's/ /" --host_copt=\"/g')\" \\\\" scripts/bootstrap/compile.sh
+      #sed -i -e "361 a --linkopt=\"-Wl,$(echo $NIX_LDFLAGS | sed -e 's/ /" --linkopt=\"-Wl,/g')\" \\\\" scripts/bootstrap/compile.sh
+      #sed -i -e "361 a --host_linkopt=\"-Wl,$(echo $NIX_LDFLAGS | sed -e 's/ /" --host_linkopt=\"-Wl,/g')\" \\\\" scripts/bootstrap/compile.sh
 
       # --experimental_strict_action_env (which will soon become the
       # default, see bazelbuild/bazel#2574) hardcodes the default

@uri-canva
Copy link
Contributor

The patch seems to apply cleanly, the hardcoded line numbers are not an issue, but it seems the version of clang we're using is not new enough, as it's missing this: http://llvm.org/viewvc/llvm-project?view=revision&revision=241542:

tools/osx/xcode_locator.m:67:3: error: type arguments cannot be applied to non-parameterized class 'NSMutableDictionary'
  NSMutableDictionary<NSString *, XcodeVersionEntry *> *dict) {
  ^                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools/osx/xcode_locator.m:117:8: error: type arguments cannot be applied to non-parameterized class 'NSMutableDictionary'
static NSMutableDictionary<NSString *, XcodeVersionEntry *> *FindXcodes()
       ^                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools/osx/xcode_locator.m:121:3: error: type arguments cannot be applied to non-parameterized class 'NSMutableDictionary'
  NSMutableDictionary<NSString *, XcodeVersionEntry *> *dict =
  ^                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools/osx/xcode_locator.m:179:3: error: type arguments cannot be applied to non-parameterized class 'NSMutableDictionary'
  NSMutableDictionary<NSString *, XcodeVersionEntry *> *dict) {
  ^                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools/osx/xcode_locator.m:180:3: error: type arguments cannot be applied to non-parameterized class 'NSSet'
  NSSet<XcodeVersionEntry *> *distinctValues =
  ^    ~~~~~~~~~~~~~~~~~~~~~
tools/osx/xcode_locator.m:182:35: error: type arguments cannot be applied to non-parameterized class 'NSMutableSet'
  NSMutableDictionary<NSString *, NSMutableSet <NSString *> *> *aliasDict =
                                  ^            ~~~~~~~~~~~~~
tools/osx/xcode_locator.m:182:3: error: type arguments cannot be applied to non-parameterized class 'NSMutableDictionary'
  NSMutableDictionary<NSString *, NSMutableSet <NSString *> *> *aliasDict =
  ^                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools/osx/xcode_locator.m:206:3: error: type arguments cannot be applied to non-parameterized class 'NSMutableDictionary'
  NSMutableDictionary<NSString *, XcodeVersionEntry *> *dict) {
  ^                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tools/osx/xcode_locator.m:263:5: error: type arguments cannot be applied to non-parameterized class 'NSMutableDictionary'
    NSMutableDictionary<NSString *, XcodeVersionEntry *> *dict = FindXcodes();
    ^                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Seems odd though given that that rev is from three years ago, might be some configuration issue with how we build clang?

Uh, what? Can we please not hardcode line numbers?

Doesn't diff output also hardcode numbers? I suppose if it's more lenient we could generate a patch and apply it, but it would still depend either on the exact number, or the exact text surrounding the place where we want to add text.

@Profpatsch
Copy link
Member

Profpatsch commented Aug 24, 2018

Doesn't diff output also hardcode numbers? I suppose if it's more lenient we could generate a patch and apply it, but it would still depend either on the exact number, or the exact text surrounding the place where we want to add text.

scripts/bootstrap/compile.sh takes "${@}" as last argument, so it has built-in support for passing additional arguments, no line-based ad-hoc patching needed that will throw strange errors should the file change with an update.

@uri-canva
Copy link
Contributor

But we call compile.sh, not scripts/bootstrap/compile.sh. bazel_build in scripts/bootstrap/bootstrap.sh also forwards any arguments, so we might reimplement the pieces of compile.sh we need so we can call it directly, it should come out to a couple of lines long since compile.sh is only about 100 lines and most of the lines are comments and error messages and handling different environments. What do you think?

@Profpatsch
Copy link
Member

it should come out to a couple of lines long since compile.sh is only about 100 lines and most of the lines are comments and error messages and handling different environments. What do you think?

I like that, removes one layer of indirection.

@c0bw3b
Copy link
Contributor

c0bw3b commented Oct 13, 2018

@guibou @uri-canva @Profpatsch
Do you all agree to close this PR now that #46700 bumped bazel to 0.17.1 ?

@c0bw3b c0bw3b self-assigned this Oct 13, 2018
@guibou
Copy link
Contributor Author

guibou commented Oct 13, 2018

@c0bw3b yes, for sure.

@c0bw3b c0bw3b closed this Oct 13, 2018
@guibou guibou deleted the gb/fix_bazel_darwin_misplaced_code branch October 14, 2018 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants