Native support for Apple Silicon #105026
Native support for Apple Silicon #105026
Conversation
| ${extraBefore+"${extraBefore[@]}"} \ | ||
| ${params+"${params[@]}"} \ | ||
| ${extraAfter+"${extraAfter[@]}"} | ||
|
|
||
| if [ -e "@out@/nix-support/post-link-hook" ]; then |
There was a problem hiding this comment.
This is a hook to sign everything the linker produces, without putting explicit signing references in the general linker wrapper.
There was a problem hiding this comment.
How does this interact with stripping binaries? Don't signatures cover the whole binaries? (I am just driving by and reading through the comments)
There was a problem hiding this comment.
Any modification of the binary, including strip and install_name_tool will cause the signature to be invalid. However, upstream's versions of these tools recreate a valid signature. To be like upstream, I've also wrapped strip and install_name_tool to recreate valid signatures. One of the todo list: add a similar wrapper for remove-references-to, which is valid to call on a binary. This is probably the correct solution to #105026 (comment). The current version will have broken this again (sorry @bobrik!).
Initially I used a fixup phase hook, which generally worked, but ran into a few cases that needed extra work. meson, for example, calls install_name_tool internally on binaries that are then executed to generate caches in some package. I initially patched meson to call codesign again, but I think wrapping is more correct. I also had to be very carefully to the signing hook run last in the fixup phase, which seems to run before postFixup.
| @@ -23,7 +23,7 @@ fixDarwinDylibNames() { | |||
| for fn in "$@"; do | |||
| if [ -L "$fn" ]; then continue; fi | |||
| echo "$fn: fixing dylib" | |||
| int_out=$(install_name_tool -id "$fn" "${flags[@]}" "$fn" 2>&1) | |||
| int_out=$(@targetPrefix@install_name_tool -id "$fn" "${flags[@]}" "$fn" 2>&1) | |||
There was a problem hiding this comment.
Most of the prefixed tools from bintools are exposed in corresponding variables in the bintools setup hook. Would it be reasonable to do the same for install_name_tool?
There was a problem hiding this comment.
Yes, if there is a conventional env var for it, go right ahead!
| @@ -18,7 +19,7 @@ let | |||
| mv clang-tools-extra-* $sourceRoot/tools/extra | |||
| ''; | |||
|
|
|||
| nativeBuildInputs = [ cmake python3 lld ] | |||
| nativeBuildInputs = [ cmake python3 buildPackages.llvmPackages_10.lld ] | |||
There was a problem hiding this comment.
I pulled the cross compilation changes for clang out as #100574. This PR is now the motivation for that one.
| @@ -183,8 +202,13 @@ let | |||
|
|
|||
| depsBuildBuild = [ buildPackages.stdenv.cc makeWrapper ]; | |||
|
|
|||
| patches = stdenv.lib.optional stdenv.hostPlatform.isDarwin [ | |||
| ./cross-darwin.patch | |||
There was a problem hiding this comment.
This should be sent upstream. It's harder to make a proper quality patch than it is to make something that works. Is it reasonable to avoid perl here? I think it's only for openssl.
| includeDirs = [ | ||
| "CommonCrypto" "_types" "architecture" "arpa" "atm" "bank" "bsd" "bsm" | ||
| "corecrypto" "corpses" "default_pager" "device" "dispatch" "hfs" "i386" | ||
| "iokit" "kern" "libkern" "mach" "mach-o" "mach_debug" "machine" "malloc" | ||
| "miscfs" "net" "netinet" "netinet6" "netkey" "nfs" "os" "osfmk" "pexpert" | ||
| "platform" "protocols" "pthread" "rpc" "rpcsvc" "secure" "security" | ||
| "servers" "sys" "uuid" "vfs" "voucher" "xlocale" | ||
| ] ++ [ | ||
| "arm" "xpc" | ||
| ]; |
There was a problem hiding this comment.
This isn't open source libSystem, but Apple SDK libSystem. In the upstream sdk all the headers are in a single directory, so I don't know which ones are part of libSystem and which are unrelated. I've approximated it by looking at the output of the current libSystem.
| @@ -48,7 +48,9 @@ in lib.init bootStages ++ [ | |||
| # Prior overrides are surely not valid as packages built with this run on | |||
| # a different platform, and so are disabled. | |||
| overrides = _: _: {}; | |||
| extraBuildInputs = [ ]; # Old ones run on wrong platform | |||
| extraBuildInputs = [ ] # Old ones run on wrong platform | |||
| ++ lib.optionals hostPlatform.isDarwin [ buildPackages.targetPackages.darwin.apple_sdk.frameworks.CoreFoundation ] | |||
There was a problem hiding this comment.
This is something I have found no good solution for, I'm hoping I missed something obvious. This step defines the stdenv that produces targetPackages, so including a dependency package from targetPackages is trivially infinite recursion. In the Darwin stdenv it's easy to build things on a different stage and use them in a later stage, since all platforms are the same. However when cross compiling the adjacent stages are quite rigid. The only way this isn't infinite recursion is by overriding extraBuildInputs = [ ] for stdenvNoCC, and using that to build CoreFoundation.
| cp -d ${llvmPackages.llvm.lib}/lib/libLLVM.dylib $out/lib | ||
| cp -d ${libffi}/lib/libffi*.dylib $out/lib | ||
|
|
||
| mkdir $out/include | ||
| cp -rd ${llvmPackages.libcxx}/include/c++ $out/include | ||
|
|
||
| # copy .tbd assembly utils | ||
| cp -d ${pkgs.darwin.rewrite-tbd}/bin/rewrite-tbd $out/bin |
There was a problem hiding this comment.
This increase in bootstrap tools is unfortunate, but supports extracting the sdk almost immediately.
pkgs/stdenv/generic/default.nix
Outdated
| @@ -19,6 +19,7 @@ let lib = import ../../../lib; in lib.makeOverridable ( | |||
| , setupScript ? ./setup.sh | |||
|
|
|||
| , extraNativeBuildInputs ? [] | |||
| , extraNativeBuildInputsPostStrip ? [] # TODO: this is terrible | |||
There was a problem hiding this comment.
Exactly what the comment says. Signing has a dependency on codesign_allocate, which is prefixed with the targetPrefix, so it needs some way to know the prefix. The other files in this list use the variables introduced in the bintools setup hook, for example strip.sh uses $STRIP.
| @@ -114,6 +114,22 @@ let | |||
| '' + mkExtraBuildCommands cc; | |||
| }; | |||
|
|
|||
| cctoolsClangNoLibcxx = wrapCCWith rec { | |||
There was a problem hiding this comment.
There's a lot of duplication here with the lldClang* versions. This is complicated enough without having to maintain it over ~5 different clang versions. Is there a better way?
There was a problem hiding this comment.
Unfortunately, it's not easy to get around. We could add some helper function since lld, cctools, and binutils will all need the same 3 stage process I think (if binutils works at all).
|
Does nixpkgs support anything like Debian's multiarch? I'm wondering if it would be feasible to default to |
|
@dhess you could mix and match two Nixpkgs evaluations for x86_64 and aarch64 in one environment, though it needs to be done manually. Something like |
|
@kisik21 Right, that makes sense. Thanks! |
| stdenv_ = | ||
| if stdenv.hostPlatform.isDarwin && (stdenv.hostPlatform != stdenv.buildPlatform) | ||
| then overrideCC stdenv buildLlvmTools.cctoolsClangNoCompilerRt | ||
| else if stdenv.hostPlatform.useLLVM or false |
There was a problem hiding this comment.
the useLLVM flag should probably get precedence
| stdenv = overrideCC stdenv buildLlvmTools.lldClangNoCompilerRt; | ||
| compiler-rt = let | ||
| stdenv_ = | ||
| if stdenv.hostPlatform.isDarwin && (stdenv.hostPlatform != stdenv.buildPlatform) |
There was a problem hiding this comment.
We should test that Darwin -> Linux cross compilation still works (if it used to at all). I think hostPlatform.isDarwin is okay here, but we need to make sure cctools is only used when we compile to mach-o systems (either darwin or ios).
| @@ -114,6 +114,22 @@ let | |||
| '' + mkExtraBuildCommands cc; | |||
| }; | |||
|
|
|||
| cctoolsClangNoLibcxx = wrapCCWith rec { | |||
There was a problem hiding this comment.
Unfortunately, it's not easy to get around. We could add some helper function since lld, cctools, and binutils will all need the same 3 stage process I think (if binutils works at all).
| # compiling. It's not very sound. The cross stdenv has: | ||
| # extraBuildInputs = [ targetPackages.darwin.apple_sdks.frameworks.CoreFoundation ] | ||
| # and uses stdenvNoCC. In order to make this not infinitely recursive, we need to exclude | ||
| # this extraBuildInput. |
There was a problem hiding this comment.
This looks similar to https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/static.nix#L24. We may be able to remove that after this PR (should check that pkgsStatic. still works first)
| pname = "libtapi"; | ||
| version = "1000.10.8"; # determined by looking at VERSION.txt | ||
|
|
||
| version = "1100.0.11"; |
There was a problem hiding this comment.
Note to myself to verify iOS cross-compilation still works after this
|
Quite a few things depend on Rust which doesn't look like it provides bootstrap tools for aarch64-apple-darwin yes? |
See NixOS/nix#4310 for some support in Nix. So you can use |
IIRC rust is only yet supported on the nightly build. Stable support should come with the next release. |
I think that Go and Python are in a similar situation, either lacking bootstrapping tools or fundamental libraries need updating. There are a lot of things that don't build in this branch. Once we have a stdenv in place there's going to be a lot of work to make things build, but hopefully a most of that work will be done by the various upstreams in the near future. |
|
@thefloweringash Could you take a look at thefloweringash#2 ? This came up when trying to build aarch64 Nix. Unfortunately it's a mass rebuild. |
Seems like a good change to make, applied. I'm curious what came up, since I have used this to build a native nix. |
|
If I install nix on my system, than only a native one. Is there any instruction on how to install the native nix? If yes, I would be willing to give fixing stuff a try. |
This is nix unstable - although I would think the same would appear in nix 2.3.x. Here is the log:
|
|
@thefloweringash See NixOS/nix#4334 for changes I had to make for Nix to work. |
| @@ -0,0 +1,212 @@ | |||
| diff --git a/Makefile b/Makefile | |||
There was a problem hiding this comment.
Could we look into upstreaming this?
|
Anyone know of aarch64-apple-darwin bootstrap tools for GHC or Rustc? Can't find much although it looks like some people have a compiler working. |
Source: rust-lang/rust#73908 (comment) So we have to decide if we want to package a nightly rust for now, or wait until the release of rust 1.49. |
It seems this is due on December 31st 2020, so ~2 weeks from now. |
|
Go is similar: the only version that supports running on One interesting thing I noticed while playing with Nix on M1 is that
|
I would say that this is to be expected. AArch64 has fixed-length 32-bit instructions. X86_64 has variable length, therefore expect the code density to be higher in x86_64 code. |
|
The difference is not as dramatic for |
Debian uses GCC, Darwin uses Clang. I was able to find an issue, where this is discussed: android/ndk#21 - It says that GCC defaults to |
|
@prusnak in my initial comment I compare Could it be that clang version is at play here? I have 7.1.0 for x86_64 and 10.0.1 for aarch64 in my Here's my test: #include <stdio.h>
int main() {
printf("howdy, fellas\n");
return 0;
}With clang 10.0.1 from aarch64: With clang 7.1.0 from x86_64: And comparing the resulting binaries: Compiling x86_64 with clang10 produces the same size as clang7, which means it's not the clang version by itself. Looking at the output of The difference goes away as binaries themself get larger. This makes sense, since on tiny binaries pages are sparsely populated, while in larger ones they are mostly full. Here's git for example: Compressing tiny binaries brings them closer together as you discard mostly empty page aligned space: |
|
I was getting this error for Debugging it with lldb didn't show the symptom: Turns out the signature was corrupted: It doesn't happen with x86_64 binary that runs under rosetta, because in that case there's no signature to break: I was able to fix this by not running The same can be achieved by moving this step from |
|
How do I switch native builds from I am trying the following but it seems it's building x86_64 binaries: |
|
I'm not sure how it's for nix-shell... but for |
|
And you probably want to choose nixpkgs commit that has enough binaries (even though experimental), e.g. 5be0234 https://hydra.nixos.org/eval/1666126 |
|
I kind of understood how to use it from this branch before the merge, but not sure what to do now? Do we have to wait for the PR updating this to use the bootstrapped tools, before we can use the channel? (sorry if that's something obvious) |
|
Yes, the merged state isn't usable yet (for aarch64-darwin), so I expect you want to use e.g. commit 5be0234. |
|
@vcunat right, I understood that part from your previous message, sorry if I was being unclear. What I've meant is how can I track when |
|
Ah, OK. Someone will surely put a comment into this thread. BTW, the job for new bootstrapping tools is here: https://hydra.nixos.org/build/143059349 (might even finish today) |
|
@thefloweringash The darwin bootstrap job has finished https://hydra.nixos.org/build/143059349 |
Great! Could i get someone with the ability to write to tarballs.nixos.org to upload them there? Then we can reference them from the stdenv. |
|
cc @grahamc |
|
Uploaded to the tarball mirrors under These came from /nix/store/riw6lnfrz2qklbf06rbzx11h66sd8xws-stdenv-bootstrap-tools-aarch64-apple-darwin which came from https://hydra.nixos.org/build/143059349#tabs-buildinputs |
|
@jaen: the |
|
It built ~16k packages succesfully while ~3k are failing. |
|
21.05 and trunk now build packages for aarch64-darwin (about 16k build compared to 24k for x86_64)
|
|
I'll post updates to #95903 |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
|
FYI, for anyone who is interested in migrating all packages to |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/how-to-install-nix-on-macos-with-m1/13791/1 |
Motivation for this change
Native support for Apple's new CPU architecture.
Included here is a cross compiled bootstrap tools, and native stdenv based on them. Since there are no open source components for Big Sur, this relies almost entirely on the Apple SDK for Big Sur. It also includes helper utilities written by myself, for the purpose of generating ad-hoc signatures (
sigtool) and rewriting framework re-exports to fit into nixpkg's frameworks structure (rewrite-tbd).Presently very few packages build. In a number of cases the upstream projects do not have support, and we'll likely to have wait or backport fixes. Failing packages I've noticed include python, neovim, and ctags.
I've added a review for some of the points I'd like help with.
For signing, it looks like upstream cctools-port has support based on
ldid. I made my implementation as part of investigating the requirements on the dtk, and before I was aware of an existing one. I'm not really attached to either implementation. The best possible outcome would be that Apple releases an open source version with signing but I don't know if that's a reasonable expectation.Pre merge tests: https://gist.github.com/thefloweringash/62d7b3f3e71cdc97c92fdcd320a492ad. These check the darwin bootstrap tools and resulting stdenv for x86_64-darwin and aarch64-darwin.
Related pull requests:
-macos_version_min,-platform_version,-sdk_version, etc)darwinArchand set-archin {bintools,cc}-wrapper-fno-commonSome of these are included (in some form) here, but once these are merged this branch can be rebased on them and become more manageable.
Current Known Blockers
Things done
sandboxinnix.confon non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"./result/bin/)nix path-info -Sbefore and after)