darwin (Big Sur): tbd based stdenv #98541

thefloweringash · 2020-09-23T08:02:57Z

Motivation for this change

This is an implementation of Text API (tapi, or .tbd files) based stdenv and Frameworks for Darwin. Closes #91748 and #19906.

As of Big Sur the macOS filesystem no longer contains .dylib files and instead has a precomputed dyld cache. As a result the nixpkgs stdenv cannot link on Big Sur which vastly reduces the usability.

Apple's SDK uses Text API descriptions of the contents of shared libraries. These .tbd files are straightforward YAML, can be passed directly to the linker and behave exactly like having .dylib or framework files. The current implementation of the Darwin stdenv uses a combination of proxy dyld libraries and direct symlinks to system libraries. Switching from the proxy libraries and symlinks to linking against these .tbd files is an improvement to compatibility and reproducibility.

There are two main options for sources of .tbd files: Apple's SDK, or generating them from an existing installation of macOS via tapi stubify. I assert that generating them is less likely to cause legal complications than deriving them from Apple's SDK. The darwin-stubs repository contains appropriate stubs generated from macOS 10.12.6. Technically either source should be trivially interchangeable so we're not locked into either here.

If this branch is merged, then nixpkgs will be usable on Big Sur but not bootstrappable. The bootstrap environment requires the construction of a libSystem proxy in order to run the bootstrap tools. This shouldn't be much of a problem since a usable stdenv will be available from the binary cache. The easy solution is to regenerate the bootstrap tools once again after these changes have propagated.

I've spent some time testing on Big Sur and have been able to build my regular nix-darwin system closure which includes components written in rust, go, haskell (ghc), ruby, python, and using qt5. Note that the qt5 patch included here is the simplest thing that can work and is not generally applicable to upstream. I'm assuming upstream will produce a correct fix soon.

Prerequisites:

Transfer https://github.com/thefloweringash/darwin-stubs to the NixOS GitHub organization
Wait for staging to be merged to master for bootstrap-tools changes in stdenv/darwin: enable tapi support in cctools #93912
Review and merge "stdenv/darwin: put compiler-rt into bootstrap tarball" stdenv/darwin: put compiler-rt into bootstrap tarball #94426
Update bootstrap tools from hydra, ~~including review and merge of "stdenv/darwin: assemble full clang toolchain" stdenv/darwin: assemble full clang toolchain #94427~~

This PR will need to be updated after these prerequisites are satisfied. Opening now as a draft to have a description of the overall plan and to tie the prerequisites together.

cc @matthewbauer and @LnL7

Things done

LnL7 · 2020-09-23T11:34:01Z

pkgs/top-level/darwin-packages.nix

@@ -41,6 +41,8 @@ in

  DarwinTools = callPackage ../os-specific/darwin/DarwinTools { };

+  darwin-stubs = callPackage ../os-specific/darwin/darwin-stubs { };


not sure we should expose this as a package

This way makes it easy to provide to various libraries and does make debugging slightly easier in that you can nix-build -A darwin.darwin-stubs. Did you have another mechanism in mind? I can imagine like treating them a bit like bootstrap tools and only propagating them in the stdenv definition.

I'm in favour of keeping this exposed. And if only it will make debugging easier.

LnL7 · 2020-09-23T11:41:28Z

pkgs/os-specific/darwin/apple-sdk/print-reexports/default.nix

+
+stdenv.mkDerivation {
+  name = "print-reexports";
+  src = stdenv.lib.sourceFilesBySuffices ./. [".c"];


Fine for now, but perhaps it would be best to move this to eg. the stubs repo in the future.

pkgs/stdenv/darwin/default.nix

pkgs/os-specific/darwin/apple-sdk/cf-setup-hook.sh

LnL7 · 2020-10-18T18:55:08Z

I rebased this against an eval and updated using the official bootstrap tools from hydra here LnL7@d7befca.
This comparison should give a decent view of the general state of things with these changes.

https://hydra.nixos.org/eval/1619750?compare=1617819

domenkozar · 2020-10-20T13:38:36Z

Note that installer changes have been merged in Nix, so we only need to get this merged (and hopefully backport to 20.09) cc @NixOS/nixos-release-managers

thefloweringash · 2020-11-09T10:41:09Z

I included a reversion of the cctools bump here since it seemed to be causing problems (#101330). That's being addressed in other tickets (like #103053). Before merging we should drop that commit. I'm avoiding rebasing/removing for now because this branch is indirectly related to a hydra job set and I don't want to make it harder to debug.

This new version has tapi support, which is needed to build the new stubs based libSystem, etc. and Big Sur support. You can verify the provenance of these yourself by checking Hydra here: https://hydra.nixos.org/build/128192471

Adapted from main expression for clang 7.

Build the llvm support libraries (libcxx, libcxxabi) from scratch without using the existing llvm libraries. This is the same spirit and similar implementation as the "useLLVM" bootstrap in llvm package sets. Critically it avoids having libcxxabi provided by the cc-wrapper when building libcxx, which otherwise results in two libcxxabi instances. $ otool -L /nix/store/vd4vvgs9xngqbjzpg3qc41wl6jh42s9i-libc++-7.1.0/lib/libc++.dylib /nix/store/vd4vvgs9xngqbjzpg3qc41wl6jh42s9i-libc++-7.1.0/lib/libc++.dylib: /nix/store/vd4vvgs9xngqbjzpg3qc41wl6jh42s9i-libc++-7.1.0/lib/libc++.1.0.dylib (compatibility version 1.0.0, current version 1.0.0) /nix/store/gmpwk5fyp3iasppqrrdpswxvid6kcp8r-libc++abi-7.1.0/lib/libc++abi.dylib (compatibility version 1.0.0, current version 1.0.0) /nix/store/3hn7azynqgp2pm5gpdg45gpq0ia72skg-libc++abi-7.1.0/lib/libc++abi.dylib (compatibility version 1.0.0, current version 1.0.0) /nix/store/1nq94scbxs6bk7pimqhvz76q6cfmbv97-Libsystem-osx-10.12.6/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1) Additionally move some utilities (clang, binutils, coreutils, gnugrep) to the stage layers so they can be replaced before the final stdenv. This should cause most of stage4 to be built from the toolchain assembled as of stage3 instead of the bootstrap toolchain.

Exposed by the tbd stubs which contains fat libraries. The previously used proxy libraries were x86_64 only.

thefloweringash · 2020-11-09T11:14:04Z

Closed by complete accident, sorry for the noise.

worldofpeace · 2020-11-09T11:44:08Z

pkgs/os-specific/darwin/darwin-stubs/default.nix

+
+  src = fetchFromGitHub {
+    owner = "thefloweringash";
+    repo = "darwin-stubs";


if this is directly to support nixpkgs it would probably be a good idea to transfer the repo here? And also have darwin maintainers access to it

I would definitely like to do this. I included it in the "Prerequisites" section, which isn't quite panning out as I expected.

I think @domenkozar is one of the people with enough permissions to do that.

If someone PM's him on IRC it should be pretty easy to have done

@thefloweringash if you add me to that repo I can transfer it.

domenkozar · 2020-11-11T15:13:27Z

Big Sur will be be released tomorrow. What is left to do here?

Used during bootstrap to check that re-exported libraries are not dangling.

thefloweringash · 2020-11-19T05:45:06Z

tested this with qt5.qtwebengine (qtwebengine-5.12.7) on macOS BigSur (x86_64). Fails with:
ninja: error: '../../../../../../../../../../../usr/lib/libsandbox.1.dylib', needed by 'obj/components/cdm/renderer/librenderer.a', missing and no known rule to make it
@thefloweringash: It looks like pkgs/development/libraries/qt-5/5.12/qtbase.patch.d/0015-qtbase-tbd-frameworks.patch doesn't catch this

This is an explicit impure reference in qtwebengine itself. This should probably be a library in the pkgs/os-specific/darwin directory. Since I missed it from the stubs, we'll need to add it there first, and then wrap it as a library. See NixOS/darwin-stubs#3.

dhess · 2020-11-19T11:16:10Z

Does this commit change the viability of building nixpkgs on macOS with sandboxing enabled? I've tried that in the past, but always end up disabling it, because eventually some important package breaks.

angerman · 2020-11-19T14:07:32Z

Does this commit change the viability of building nixpkgs on macOS with sandboxing enabled? I've tried that in the past, but always end up disabling it, because eventually some important package breaks.

I doubt so, I believe for proper sandboxing we'll need to address how nix does sandboxing on macOS, as it keeps running into limits when it sets them up.

dhess · 2020-11-19T16:27:32Z

@angerman Ahh, right, it's some sort of limitation in Apple's own sandboxing mechanism. Now I remember :)

FRidh · 2020-11-29T15:29:06Z

There was a question on IRC if there is any plan on backporting this change to 20.09.

angerman · 2020-11-30T01:46:23Z

@angerman Ahh, right, it's some sort of limitation in Apple's own sandboxing mechanism. Now I remember :)

I think it's more like: nix could be smarter about this. I believe @LnL7 had some ideas for this. But memory is fuzzy.

angerman · 2020-11-30T01:46:44Z

There was a question on IRC if there is any plan on backporting this change to 20.09.

I'd like to see this as well. If not even 20.03 but that might be a too tall ask.

matthewbauer · 2020-12-02T23:09:17Z

Awesome work @thefloweringash! This improves things greatly!

davidtwco · 2020-12-03T15:31:23Z

There was a question on IRC if there is any plan on backporting this change to 20.09.

That was me, I've opened #105799 to backport this.

Needed on MacOS Big Sur due to NixOS/nixpkgs#98541.

ofborg bot added 6.topic: darwin Running or building packages on Darwin 6.topic: qt/kde 6.topic: stdenv Standard environment 10.rebuild-darwin-stdenv 8.has: package (new) 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 101-500 labels Sep 23, 2020

LnL7 reviewed Sep 23, 2020

View reviewed changes

domenkozar changed the title ~~darwin: tbd based stdenv~~ darwin (Big Sur): tbd based stdenv Oct 15, 2020

LnL7 added this to Big Sur in Darwin Oct 24, 2020

LnL7 and others added 4 commits November 9, 2020 19:59

stdenv/darwin: bump bootstrap tools

7f1a82d

This new version has tapi support, which is needed to build the new stubs based libSystem, etc. and Big Sur support. You can verify the provenance of these yourself by checking Hydra here: https://hydra.nixos.org/build/128192471

darwin/stdenv: assemble full clang toolchain

86ee107

Adapted from main expression for clang 7.

compiler-rt: prevent platform sniffing on Darwin

ddd1720

Exposed by the tbd stubs which contains fat libraries. The previously used proxy libraries were x86_64 only.

thefloweringash force-pushed the darwin-tapi-stdenv branch from 7073fb3 to 3d00391 Compare November 9, 2020 11:02

thefloweringash closed this Nov 9, 2020

thefloweringash reopened this Nov 9, 2020

worldofpeace reviewed Nov 9, 2020

View reviewed changes

thefloweringash marked this pull request as ready for review November 9, 2020 13:23

thefloweringash requested review from matthewbauer and ttuegel as code owners November 9, 2020 13:23

thefloweringash added 3 commits November 12, 2020 00:23

darwin/darwin-stubs: init

8946ff8

darwin/print-reexports: nixpkgs-specific utility to inspect .tbd files

fd33052

Used during bootstrap to check that re-exported libraries are not dangling.

darwin/Libsystem: use darwin-stubs

b176ad9

thefloweringash mentioned this pull request Nov 19, 2020

Add stubs for libsandbox NixOS/darwin-stubs#3

Merged

LnL7 mentioned this pull request Nov 22, 2020

darwin stdenv can't bootstrap on macOS 11.0 Big Sur #104580

Closed

LnL7 moved this from Big Sur to Done in Darwin Nov 22, 2020

This was referenced Nov 30, 2020

bazel: 3.3.1 -> 3.7.2 #105439

Merged

bintools-wrapper change causes builds to fail on darwin - ld: Assertion failed #105573

Closed

davidtwco mentioned this pull request Dec 3, 2020

[backport: 20.09] darwin (Big Sur): tbd based stdenv #105799

Closed

10 tasks

ghedamat mentioned this pull request Dec 15, 2020

Nokogiri fails to build on MacOS BigSur #106973

Closed

stephank mentioned this pull request Dec 21, 2020

Use .tbd files for pure Darwin builds even against impure runtimes #19906

Closed

rmcgibbo mentioned this pull request Dec 29, 2020

python3Packages.shapely: broken on darwin? #107842

Closed

MarcoPolo mentioned this pull request Jan 4, 2021

Fails on macOS Big Sur (?) input-output-hk/haskell.nix#982

Closed

This was referenced Jan 12, 2021

Nix Build Fails on MacOS Big Sur urbit/urbit#4214

Closed

MacOS installation build error nix-community/home-manager#1713

Closed

toonn mentioned this pull request Mar 14, 2021

Status of macOS #116341

Open

thefloweringash deleted the darwin-tapi-stdenv branch March 16, 2021 14:24

thefloweringash mentioned this pull request Mar 16, 2021

qtwebengine: use pure sandbox lib #116528

Merged

10 tasks

This was referenced Apr 27, 2021

Backport Darwin TBD Fixes to 20.09 #118103

Closed

darwin: Backport tbd stdenv (Big Sur support) to 20.09 #121040

Closed

Backport Big Sur support to awake-20.03 awakesecurity/nixpkgs#7

Merged

yipengsun mentioned this pull request Jun 25, 2021

Survey the build status of ROOT (curated) on macOS umd-lhcb/root-curated#2

Closed

georgefst added a commit to georgefst/cross-haskell-app that referenced this pull request Jul 15, 2021

Bump stuff

7eeea13

Needed on MacOS Big Sur due to NixOS/nixpkgs#98541.

georgefst mentioned this pull request Jul 16, 2021

Build error in macOS 11 dmjio/miso#639

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

darwin (Big Sur): tbd based stdenv #98541

darwin (Big Sur): tbd based stdenv #98541

thefloweringash commented Sep 23, 2020 •

edited

LnL7 Sep 23, 2020

thefloweringash Sep 24, 2020

angerman Nov 12, 2020

LnL7 Sep 23, 2020

LnL7 commented Oct 18, 2020

domenkozar commented Oct 20, 2020

thefloweringash commented Nov 9, 2020

thefloweringash commented Nov 9, 2020

worldofpeace Nov 9, 2020

thefloweringash Nov 9, 2020

LnL7 Nov 9, 2020

worldofpeace Nov 10, 2020

domenkozar Nov 10, 2020

domenkozar commented Nov 11, 2020

thefloweringash commented Nov 19, 2020

dhess commented Nov 19, 2020

angerman commented Nov 19, 2020

dhess commented Nov 19, 2020

FRidh commented Nov 29, 2020

angerman commented Nov 30, 2020

angerman commented Nov 30, 2020

matthewbauer commented Dec 2, 2020

davidtwco commented Dec 3, 2020

		@@ -41,6 +41,8 @@ in

		DarwinTools = callPackage ../os-specific/darwin/DarwinTools { };

		darwin-stubs = callPackage ../os-specific/darwin/darwin-stubs { };

darwin (Big Sur): tbd based stdenv #98541

darwin (Big Sur): tbd based stdenv #98541

Conversation

thefloweringash commented Sep 23, 2020 • edited

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LnL7 commented Oct 18, 2020

domenkozar commented Oct 20, 2020

thefloweringash commented Nov 9, 2020

thefloweringash commented Nov 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenkozar commented Nov 11, 2020

thefloweringash commented Nov 19, 2020

dhess commented Nov 19, 2020

angerman commented Nov 19, 2020

dhess commented Nov 19, 2020

FRidh commented Nov 29, 2020

angerman commented Nov 30, 2020

angerman commented Nov 30, 2020

matthewbauer commented Dec 2, 2020

davidtwco commented Dec 3, 2020

thefloweringash commented Sep 23, 2020 •

edited