Allow using LLVM for cross compilation toolchain #55750

matthewbauer · 2019-02-14T06:24:54Z

Motivation for this change

The idea is to allow you to build with LLVM by passing something like this in Nixpkgs:

nix-build -A hello --arg crossSystem '{ config = "aarch64-unknown-linux-gnu"; useLLVM = true; }'

Lots of this is based on the approach taking in wasm-cross repo:

https://github.com/WebGHC/wasm-cross

Unfortunately I haven't gotten libc to cross compile without using GCC.

pkgs/development/compilers/llvm/5/clang/default.nix

dtzWill · 2019-02-14T20:49:12Z

Regarding crtbegin/crtend bits: https://reviews.llvm.org/D28791 .

We may want to start looking at using LLVM utilities for binutils, there's a build-time option and they act like the gnu binutils if symlink'd over I think. No idea how well they'd be suited to our needs but something to keep in mind. And of course optionally using LLD as default linker :).

Looking at these changes presently! :)

dtzWill

Generally looks very good! I haven't been able to look at it more hands-on so as to help catch some more subtle problems, sorry. Offhand I remember something like if we don't specify --gcc-toolchain and clang was built with gcc, it will try to find the path of the compiler that built it-- which can cause some painfully subtle problems re:staging if that's not what you expect. I'm not sure what the behavior is with recent clang, and haven't really dug into this for a while :). Will be looking into this again "soon" but not sure when that'll be so nevermind that :). Exciting!!

dtzWill · 2019-02-14T20:53:30Z

pkgs/development/compilers/llvm/7/bintools.nix

+     ln -sf $prog $out/bin/${prefix}$(basename $prog)
+   done
+   rm -f $out/bin/${prefix}cat
+   ln -s ${lld}/bin/lld $out/bin/${prefix}ld


LOL I see my comment re:using llvm bits symlinked over ... is not news to you! 🗡️

FWIW this might be a great deal simpler and more predictable if instead of shell globs and loops it used linkFarm or whichever over in trivial-builders.nix. I think enumerating what we intend/expect to symlink is reasonable- the number shouldn't grow much higher than it is and this way nothing sneaks in as part of a bump :).

Yeah that probably would be better. Even just listing out all of the utilities here like we do in https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/darwin/binutils/default.nix would be helpful. This is actually originally from wasm-cross here:

https://github.com/WebGHC/wasm-cross/blob/master/llvm-head/default.nix#L58-L82

but I think that change would be a good idea.

matthewbauer · 2019-02-15T00:33:56Z

@GrahamcOfBorg test.hello

matthewbauer · 2019-02-15T00:34:12Z

@GrahamcOfBorg test hello

matthewbauer · 2019-02-15T03:35:38Z

It looks like the built binaries aren't actually working! I get it to hang on the qemu-user emulator.

@GrahamcOfBorg build tests.cross.hello.aarch64-multiplatform-musl

Ericson2314 · 2019-02-16T00:01:32Z

Instead of useLLVM, could we have like something more structured? E.g. *Platform.{cc, bintools} = {"gnu", "llvm", "cctools" /* just for cc */ } that would be defaulted and everything like the lib attribute that exists today? That would support the full combinatorial explosion of use-cases.

matthewbauer · 2019-02-22T19:54:42Z

pkgs/os-specific/linux/musl/default.nix

@@ -92,7 +92,7 @@ stdenv.mkDerivation rec {
  outputs = [ "out" "dev" ];

  dontDisableStatic = true;
-  separateDebugInfo = true;
+  separateDebugInfo = !(stdenv.hostPlatform.useLLVM or false);


This should be set for everything - separate-debug-info relies on gcc binutils

I believe you, but am a bit surprised by this. Is this because of the way we do things or do LLVM bits not support split debug info? Or is "split"/"separate" an important difference? O:)

I looked into it and it looks like there are no build ids produced:

$CC test.c -o test $READELF -n test

Probably configurable?

matthewbauer · 2019-02-22T19:55:51Z

pkgs/development/compilers/llvm/7/default.nix

+      cc = tools.clang-unwrapped;
+      bintools = wrapBintoolsWith {
+        inherit (tools) bintools;
+        libc = null;


should set isLLVM or similar

matthewbauer · 2019-02-26T03:56:56Z

@GrahamcOfBorg build tests.cross.file-llvm.aarch64-multiplatform-musl tests.cross.file-llvm.armv7l-unknown-linux-gnueabihf tests.cross.file-llvm.musl64 tests.cross.file-llvm.armv7l-unknown-linux-gnueabihf tests.cross.file-llvm.musl64 tests.cross.file-llvm.armv7l-unknown-linux-gnueabihf tests.cross.file-llvm.mingwW64 tests.cross.file-llvm.musl-power

matthewbauer · 2019-02-26T15:23:10Z

@GrahamcOfBorg build tests.cross.file-llvm.aarch64-multiplatform-musl tests.cross.file-llvm.armv7l-unknown-linux-gnueabihf tests.cross.file-llvm.musl64 tests.cross.file-llvm.armv7l-unknown-linux-gnueabihf tests.cross.file-llvm.musl64 tests.cross.file-llvm.armv7l-unknown-linux-gnueabihf tests.cross.file-llvm.mingwW64 tests.cross.file-llvm.musl-power

Ericson2314 · 2019-02-26T15:26:07Z

pkgs/development/compilers/llvm/7/default.nix

  });

  libraries = stdenv.lib.makeExtensible (libraries: let
    callPackage = newScope (libraries // buildLlvmTools // { inherit stdenv cmake libxml2 python isl release_version version fetch; });
  in {

-    compiler-rt = callPackage ./compiler-rt.nix {};
+    compiler-rt = callPackage ./compiler-rt.nix {
+      stdenv = if stdenv.hostPlatform.useLLVM or false


Oh can do you do the stdenv.hostPlatform.cc = "llvm" thing?

LLVM should be target independent because it will work with all machine types. This is different from GCC where it needs to know what target to build ahead of time.

(cherry picked from commit 08f5b419b9efc77db044f8c1d725632552617966)

You can build (partially) with LLVM toolchain using the useLLVM flag. This works like so: nix-build -A hello --arg crossSystem '{ system = "aarch64-unknown-linux-musl"; useLLVM = true }' also don’t separate debug info in lldClang It doesn’t work currently with that setup hook. Missing build-id?

ElvishJerricco · 2019-02-27T22:42:50Z

@matthewbauer This is awesome! Is there an example of how to make a nixpkgs instance using this and musl for stdenv?

ElvishJerricco · 2019-02-28T01:09:14Z

@matthewbauer Sweet! And the musl will be buillt with Clang?

dtzWill · 2019-02-28T02:17:32Z

The command provided doesn't seem to work, and switching `config` to `localSystem` evaluates but perhaps unsurprisingly we don't have the means to bootstrap an LLVM stdenv just yet. Using `crossSystem` seems to work, at least looking at the proposed list of derivations it prints. Haven't finished building yet, so not sure, but here's what might work: ``` $ nix-build -A hello --arg crossSystem '{ config = "x86_64-unknown-linux-musl"; useLLVM = true; }' ``` ~Will

…

On Wed, 27 Feb 2019 17:09:18 -0800, Will Fancher ***@***.***> wrote: @matthewbauer Sweet! And the musl will be buillt with Clang? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #55750 (comment) part: text/html

delroth · 2019-02-28T03:18:24Z

This PR seems to have broken compiler-rt on x86_64 (which breaks stuff like chromium). https://hydra.nixos.org/eval/1507125

$ nix-shell -p llvmPackages_latest.compiler-rt
...
CMake Error at cmake/base-config-ix.cmake:153 (add_default_target_arch):
  add_default_target_arch Macro invoked with incorrect arguments for macro
  named: add_default_target_arch
Call Stack (most recent call first):
  cmake/config-ix.cmake:140 (test_targets)
  CMakeLists.txt:168 (include)

delroth · 2019-02-28T03:30:29Z

Sent a fix in #56502.

matthewbauer · 2019-02-28T03:49:13Z

Oops yeah meant crossSystem

dtzWill · 2019-02-28T04:36:19Z

On Wed, 27 Feb 2019 19:49:16 -0800, Matthew Bauer ***@***.***> wrote: Oops yeah meant crossSystem

Okay, figured you did! And can confirm it works! Aw man, this is amazing to have working in nixpkgs proper and especially as cleanly as all this ends up being! Thanks for all your work on this front! :D

…

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #55750 (comment) part: text/html

matthewbauer · 2019-02-28T16:49:20Z

Yeah it’s a lot cheaper than having to recompile the whooe gcc each time too.

matthewbauer requested a review from Ericson2314 as a code owner February 14, 2019 06:24

matthewbauer requested a review from dtzWill February 14, 2019 06:25

GrahamcOfBorg added the 6.topic: stdenv Standard environment label Feb 14, 2019

Ericson2314 reviewed Feb 14, 2019

View reviewed changes

pkgs/development/compilers/llvm/5/clang/default.nix Outdated Show resolved Hide resolved

dtzWill reviewed Feb 14, 2019

View reviewed changes

GrahamcOfBorg added 2.status: merge conflict and removed 2.status: merge conflict labels Feb 15, 2019

matthewbauer commented Feb 22, 2019

View reviewed changes

Ericson2314 mentioned this pull request Feb 22, 2019

gccStdenvNoLibs, clangStdenvNoLibs: Init; libgcc: Use gccStdenvNoLibs #56219

Merged

10 tasks

GrahamcOfBorg requested review from lovek323, 7c6f434c and dtzWill February 24, 2019 10:43

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 labels Feb 24, 2019

ryantm added the 2.status: work-in-progress label Feb 24, 2019

matthewbauer changed the title ~~[wip] Allow using LLVM for cross compilation toolchain~~ Allow using LLVM for cross compilation toolchain Feb 26, 2019

matthewbauer force-pushed the llvm-cross branch 4 times, most recently from c741414 to 364dfdc Compare February 26, 2019 03:42

matthewbauer force-pushed the llvm-cross branch from 4c6db19 to 35a0d37 Compare February 26, 2019 04:25

matthewbauer requested a review from nbp as a code owner February 26, 2019 04:25

Ericson2314 reviewed Feb 26, 2019

View reviewed changes

matthewbauer force-pushed the llvm-cross branch from 35a0d37 to 8c273d6 Compare February 27, 2019 00:43

matthewbauer added 4 commits February 26, 2019 19:45

llvm: don’t use targetPlatform

ead13a3

LLVM should be target independent because it will work with all machine types. This is different from GCC where it needs to know what target to build ahead of time.

libpfm: use hostPlatform

77d7d60

(cherry picked from commit 08f5b419b9efc77db044f8c1d725632552617966)

cross/tests: add llvm-based tests

8e25da0

matthewbauer force-pushed the llvm-cross branch from 8c273d6 to 8e25da0 Compare February 27, 2019 00:48

matthewbauer merged commit 3a4d497 into NixOS:master Feb 27, 2019

This comment has been minimized.

Sign in to view

delroth mentioned this pull request Feb 28, 2019

llvm/7/compiler-rt: define CMAKE_*_COMPILER_TARGET unconditionally #56502

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow using LLVM for cross compilation toolchain #55750

Allow using LLVM for cross compilation toolchain #55750

matthewbauer commented Feb 14, 2019 •

edited

dtzWill commented Feb 14, 2019

dtzWill left a comment

dtzWill Feb 14, 2019

matthewbauer Feb 14, 2019

matthewbauer commented Feb 15, 2019

matthewbauer commented Feb 15, 2019

matthewbauer commented Feb 15, 2019

Ericson2314 commented Feb 16, 2019 •

edited

matthewbauer Feb 22, 2019

dtzWill Feb 22, 2019

matthewbauer Feb 26, 2019

matthewbauer Feb 22, 2019

matthewbauer commented Feb 26, 2019

matthewbauer commented Feb 26, 2019

Ericson2314 Feb 26, 2019

ElvishJerricco commented Feb 27, 2019

This comment has been minimized.

ElvishJerricco commented Feb 28, 2019

dtzWill commented Feb 28, 2019 via email

delroth commented Feb 28, 2019

delroth commented Feb 28, 2019

matthewbauer commented Feb 28, 2019

dtzWill commented Feb 28, 2019 via email

matthewbauer commented Feb 28, 2019

Allow using LLVM for cross compilation toolchain #55750

Allow using LLVM for cross compilation toolchain #55750

Conversation

matthewbauer commented Feb 14, 2019 • edited

Motivation for this change

dtzWill commented Feb 14, 2019

dtzWill left a comment

Choose a reason for hiding this comment

dtzWill Feb 14, 2019

Choose a reason for hiding this comment

matthewbauer Feb 14, 2019

Choose a reason for hiding this comment

matthewbauer commented Feb 15, 2019

matthewbauer commented Feb 15, 2019

matthewbauer commented Feb 15, 2019

Ericson2314 commented Feb 16, 2019 • edited

matthewbauer Feb 22, 2019

Choose a reason for hiding this comment

dtzWill Feb 22, 2019

Choose a reason for hiding this comment

matthewbauer Feb 26, 2019

Choose a reason for hiding this comment

matthewbauer Feb 22, 2019

Choose a reason for hiding this comment

matthewbauer commented Feb 26, 2019

matthewbauer commented Feb 26, 2019

Ericson2314 Feb 26, 2019

Choose a reason for hiding this comment

ElvishJerricco commented Feb 27, 2019

This comment has been minimized.

ElvishJerricco commented Feb 28, 2019

dtzWill commented Feb 28, 2019 via email

delroth commented Feb 28, 2019

delroth commented Feb 28, 2019

matthewbauer commented Feb 28, 2019

dtzWill commented Feb 28, 2019 via email

matthewbauer commented Feb 28, 2019

matthewbauer commented Feb 14, 2019 •

edited

Ericson2314 commented Feb 16, 2019 •

edited