Update macOS frameworks to 10.12 #47678

matthewbauer · 2018-10-02T19:25:26Z

Motivation for this change

Lots of updates to Darwin stuff are now possible following @copumpkin's update in #46704.

A few things I want to get to:

Bump libSystem/XNU to 10.12
Bump frameworks to 10.12 (or maybe 10.13 for consistency)
~~Update Security/SecurityTool~~

GrahamcOfBorg · 2018-10-02T19:38:13Z

Success on aarch64-linux (full log)

Attempted: coreutils, krb5, patch

The following builds were skipped because they don't evaluate on aarch64-linux: darwin.hfs, darwin.xnu

Partial log (click to expand)

  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


these paths will be fetched (0.84 MiB download, 3.07 MiB unpacked):
  /nix/store/nl80wx84gd0pl427niwgf47g580ywggv-krb5-1.15.2
copying path '/nix/store/nl80wx84gd0pl427niwgf47g580ywggv-krb5-1.15.2' from 'https://cache.nixos.org'...
/nix/store/8z916jw6wslyy089hr0s8mr0x75gjaxg-coreutils-8.30
/nix/store/nl80wx84gd0pl427niwgf47g580ywggv-krb5-1.15.2
/nix/store/a8ss5b93lpayri92gpzcnzxdbgqikdkn-patch-2.7.6

GrahamcOfBorg · 2018-10-02T19:38:36Z

Success on x86_64-linux (full log)

Attempted: coreutils, krb5, patch

The following builds were skipped because they don't evaluate on x86_64-linux: darwin.hfs, darwin.xnu

Partial log (click to expand)

  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


these paths will be fetched (0.95 MiB download, 3.24 MiB unpacked):
  /nix/store/y2baad5sx4s250b6i94smivgiak7k5p2-krb5-1.15.2
copying path '/nix/store/y2baad5sx4s250b6i94smivgiak7k5p2-krb5-1.15.2' from 'https://cache.nixos.org'...
/nix/store/hr4y6cdbg56hc8b6vjdim5p7247gkk8z-coreutils-8.30
/nix/store/y2baad5sx4s250b6i94smivgiak7k5p2-krb5-1.15.2
/nix/store/8yhhh8h46yz20g6kqsxymzav48942115-patch-2.7.6

LnL7 · 2018-10-02T19:48:08Z

pkgs/os-specific/darwin/apple-source-releases/Libsystem/system_kernel_symbols

@@ -790,6 +790,8 @@ _removexattr
 _rename
 _rename_ext
 _renameat
+_renamex_np
+_renameatx_np


Are these available on 10.12 and earlier?

It looks like they were introduced in xnu-3789.70.16 which corresponds to 10.12. gnulib uses it since this commit:

coreutils/gnulib@9abc084

and from updating to latest xnu:

https://github.com/apple/darwin-xnu/blob/0a798f6738bc1db01281fc08ae024145e84df927/bsd/sys/stdio.h#L47-L48

Here is the file that defines it in XNU:

http://newosxbook.com/src.jl?tree=xnu-3789.70.16&file=/libsyscall/wrappers/renamex.c

I wonder what people think about breaking support for 10.10 and 10.11? How common are they in the wild?

Well 10.12 is pretty important because I'm pretty sure most builders are still running that. We probably don't have to keep older versions, but we should officially drop support for those then.

I was working on automating the generation of those symbol lists as a step towards #23432
https://github.com/veprbl/libsystem_symbol_list/branches <- tool to collect lists
https://github.com/veprbl/nixpkgs/commits/libsystem_symbols <- branch intended for the PR, includes the lists and scripts to take intersection
I was never able to dedicate CPU time to bootstrap this before making a PR, but if someone would be interested in picking this up, feel free :)

@veprbl Were there any issues with this method? Might be worth putting in this PR.

There is a 10.12.6 branch in libsystem_symbol_list that was acquired from Travis machine. It would be nice to have a volunteer to contribute a list for 10.14.

@matthewbauer Not that I know of. In the end of the day you just get a list and the rest of the machinery stays the same. You are not able to easily edit it, but its content have a clear motivation. And you also recover some missing symbols that used to be skipped in manually maintained list.

Ok I just cherry-picked these and am doing a mass rebuild. We'll see how it works...

GrahamcOfBorg · 2018-10-02T20:22:19Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: coreutils, darwin.hfs, darwin.xnu, krb5, patch

Partial log (click to expand)

cannot build derivation '/nix/store/ph8x3mcvh3m4mr9c2if8zb5v3vviw4hc-flex-2.6.4.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/1z5g1ll0cx9dmi5f4xn9z4xfmh94ns7a-groff-1.22.3.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/w28hpijiiz5qpj579way0nzargmp8ayc-hfs-osx-10.13.3.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/yr6gpc7vjzl03s0lnzgv7fbnrk2c80rx-bootstrap_cmds-dev-tools-7.0.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/kn2nvhm50wkvv0rwgdbp5rlf819l3lsz-libkrb5-1.15.2.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/vggmja9f4j8hr30irra8m9lf1cdbz453-xnu-osx-10.13.3.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/2r78cs86yfhvrxl0y08v6jzm24pckjgk-cyrus-sasl-2.1.26.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/gsild20q7w89626gvhj4bbyaj0an76v5-openldap-2.4.46.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/bajl5c338drm7320l07r33v9fr2qlcbm-krb5-1.15.2.drv': 9 dependencies couldn't be built
error: build of '/nix/store/6yy6i76z598g0ffnrd65xvd911f4mj7r-coreutils-8.30.drv', '/nix/store/bajl5c338drm7320l07r33v9fr2qlcbm-krb5-1.15.2.drv', '/nix/store/kam7245pk2c9jr30mrdfj84jp06jwjnj-patch-2.7.6.drv', '/nix/store/vggmja9f4j8hr30irra8m9lf1cdbz453-xnu-osx-10.13.3.drv', '/nix/store/w28hpijiiz5qpj579way0nzargmp8ayc-hfs-osx-10.13.3.drv' failed

periklis · 2018-10-03T11:39:38Z

This will enable finally modern tools like chunkwm to be packaged 💯

LnL7 · 2018-10-03T16:49:07Z

Yeah, I think so.

copumpkin · 2018-10-09T18:30:40Z

This is mostly great, but it doesn't feel like nixpkgs is really the place to keep the archive of all libsystem symbol versions, especially when those files aren't actually used in the evaluation of nixpkgs (just as input to a script that produces the final intersected list). Perhaps we could just shove them into a separate repo in the NixOS org?

infinisil · 2018-10-12T12:51:39Z

Also, this should probably go to staging

copumpkin · 2018-10-12T13:11:11Z

Yeah, I'd probably send it to staging and maybe even trigger a PR job in Hydra for it, since it can have pretty far-reaching implications and it'd be nice to not be surprised by them.

LnL7 · 2018-10-12T20:49:23Z

This shouldn't go to staging before we run a wip jobset, I want to have an idea of the impact before polluting staging with potentially problematic changes.

LnL7 · 2018-11-22T22:12:01Z

What's the status here? The only regression I noticed when running a small set of builds was darwin.libutil.

matthewbauer · 2018-11-23T02:58:24Z

I would be happy to grt this merged. I haven’t had time to really clean this up and stuff. Mostly someone will need to be around if there are any regressions. I think the only notable change this brings is that macOS 10.11 will no longer work properly.

I am still working on getting securitytool to build. But it’s gotten a lot harder than i expected.

LnL7 · 2018-11-24T13:40:37Z

We should definitively move the symbols to a separate repo, but we don't have to wait for that to test it further. I would propose to rebase against a hydra eval (eg. 206a1c0 for trunk 1490895) and start off a darwin eval.

GrahamcOfBorg · 2018-11-30T03:34:56Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: xcbuild

Partial log (click to expand)

cannot build derivation '/nix/store/8v4hbx47269sw9i5fda035xjwcjal5r6-apple-framework-Security.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/kx60p2f7dnpgsrbyz28nl71mpy0fp133-apple-framework-Accelerate.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/9sx9w4nq2k5pz7pibw36i5lpc3zb6fbk-apple-framework-ServiceManagement.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/jfd60wy579m00xk4vg3pq8q4zkgmlwkr-apple-framework-SystemConfiguration.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/i1j139l79wv2jg0s9aq108l4j5kij4dy-apple-framework-CoreGraphics.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/mr7yjc81fsnk5vpcj892105aankzcd7c-apple-framework-CoreServices.drv': 12 dependencies couldn't be built
cannot build derivation '/nix/store/4bf2bxbangmms49hjncyknrmbmd29i7g-apple-framework-ImageIO.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/c549wg4pmkxj42qzgf0i15cqspma4qqn-xcbuild-0.1.2-pre.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/7wlv3pq4hdi5whi439zh30vyzpip3bk4-xcodebuild-0.1.2-pre.drv': 9 dependencies couldn't be built
error: build of '/nix/store/7wlv3pq4hdi5whi439zh30vyzpip3bk4-xcodebuild-0.1.2-pre.drv' failed

GrahamcOfBorg · 2018-11-30T03:46:02Z

Failure on x86_64-linux (full log)

Attempted: xcbuild

The following builds were skipped because they don't evaluate on x86_64-linux: darwin.dtrace, darwin.libpthread

Partial log (click to expand)

Call Stack (most recent call first):
  CMakeLists.txt:10 (project)


-- Configuring incomplete, errors occurred!
See also "/build/source/build/CMakeFiles/CMakeOutput.log".
See also "/build/source/build/CMakeFiles/CMakeError.log".
builder for '/nix/store/i7dr5xx4gn01s5vg7qcz62f04w0y2zjp-xcbuild-0.1.2-pre.drv' failed with exit code 1
cannot build derivation '/nix/store/kwbwqxsv85cca4vz47faiywmvdhsh115-xcodebuild-0.1.2-pre.drv': 1 dependencies couldn't be built
error: build of '/nix/store/kwbwqxsv85cca4vz47faiywmvdhsh115-xcodebuild-0.1.2-pre.drv' failed

GrahamcOfBorg · 2018-11-30T04:07:08Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: darwin.dtrace, darwin.libpthread, xcbuild

Partial log (click to expand)

cannot build derivation '/nix/store/9sx9w4nq2k5pz7pibw36i5lpc3zb6fbk-apple-framework-ServiceManagement.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/jfd60wy579m00xk4vg3pq8q4zkgmlwkr-apple-framework-SystemConfiguration.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/i1j139l79wv2jg0s9aq108l4j5kij4dy-apple-framework-CoreGraphics.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/mr7yjc81fsnk5vpcj892105aankzcd7c-apple-framework-CoreServices.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/4bf2bxbangmms49hjncyknrmbmd29i7g-apple-framework-ImageIO.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/6gkmxvl6dpac5fp6bf8gwlnvs6qhysch-xcbuild-0.1.2-pre.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/7d2cdaz560zm3aynvkykbg7yn3v8yngz-xcodebuild-0.1.2-pre.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/z5c2px4a28kwlfq772vkqjc7qqb9nirs-hook.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/lmdv6vsbqfcixlwh7v8mfldkwg6z6319-dtrace-osx-10.12.6.drv': 11 dependencies couldn't be built
error: build of '/nix/store/7d2cdaz560zm3aynvkykbg7yn3v8yngz-xcodebuild-0.1.2-pre.drv', '/nix/store/90qarrc4jprs36znmh5qzy1369n1f4h0-libpthread-osx-10.12.6.drv', '/nix/store/lmdv6vsbqfcixlwh7v8mfldkwg6z6319-dtrace-osx-10.12.6.drv' failed

matthewbauer · 2018-12-01T20:02:51Z

@GrahamcOfBorg eval

matthewbauer · 2018-12-03T18:43:28Z

@LnL7 this should be ready to go! I haven't looked at it in a while, so will need someone to go through this again just to make sure it doesn't mess anything up. I was originally planning to include fixes for "securitytool" in here - but haven't gotten it to work. But otherwise this should be safe to go to staging I think.

LnL7 · 2018-12-03T19:10:55Z

Both coreutils and patch are failing for me, but this looks promising Homebrew/homebrew-core@440b9e7. Looks like we didn't run into that yet because we keep the sdk versions stable.

Undefined symbols for architecture x86_64:
  "_renameatx_np", referenced from:
      _renameat2 in libpatch.a(renameat2.o)
ld: symbol(s) not found for architecture x86_64

matthewbauer · 2018-12-03T19:23:18Z

Does that happen on 10.12??? I have a 10.11 machine that it breaks on but I was hoping that would be the only one broken. I don't get it on 10.14

LnL7 · 2018-12-03T19:36:12Z

Yeah, this is 10.12. Does this not happen on newer versions?

matthewbauer · 2018-12-03T19:47:01Z

Yeah - I'm surprised it happens on 10.12. The symbol was added by Apple recently:

http://newosxbook.com/src.jl?tree=xnu-3789.70.16&file=/bsd/sys/stdio.h

But it should be there for 10.12. Maybe it needs it listed in the symbols list?

matthewbauer · 2018-12-15T21:46:52Z

Sorry the original commit was missing the new symbols to libSystem. That error should be fixed in 716f9fb. Going to do a rebuild and then merge.

Lots of stuff has gotten moved around. Many security libraries have been merged into the Security monorepo. I’ve cleared them out for now, we will need to modify Security to build them! This also moves some things around to more clearly separate bootstrapping the stdenv from everything else. We want the “normal” mode to be the non-bootstrapped version. When you ask for “Security”, you want the actual built software, not a crippled one. - Add TARGET_OS_OSX to darwin.libSystem. Looks like something introduced in 10.12. TARGET_OS_MAC is only set when building for desktop (iOS will have TARGET_OS_MAC set) - Bump darwin.dtrace - Bump darwin.libpthread - Remove SmartCardServices, libsecurity*, etc. - Install some more headers for darling.

LnL7 · 2018-12-19T22:46:16Z

Like I mentioned before, this should be tested on hydra before merging to staging. It causes 8600 breakages and the stdenv is still broken on 10.12 and thus will block the new staging-next until fixed or reverted.

https://hydra.nixos.org/eval/1496326#tabs-now-fail

Undefined symbols for architecture x86_64:
  "_fs_snapshot_root", referenced from:
     -reexported_symbols_list command line option
  "_mk_timer_arm_leeway", referenced from:
     -reexported_symbols_list command line option
ld: symbol(s) not found for architecture x86_64

matthewbauer · 2018-12-19T23:43:08Z

Sorry about that!

It's very weird that this is not happening locally. Most likely we can remove those entries from system_kernel_symbols safely. The symbols should be available on 10.12.6 but perhaps not on earlier 10.12.

LnL7 · 2018-12-20T18:35:34Z

The VM was installed with 10.12.2, looks like it's much better after updating which is a bit unfortunate but not as bad as I thought. Can you look into the binutils issue? I'll start another eval when that's fixed to see what remains.

hfs was introduced in e6f7f29 (NixOS#47678), but it seems nobody will needs its pure headers. We drop it from Libsystem.

GrahamcOfBorg added 6.topic: darwin Running or building packages on Darwin 10.rebuild-darwin-stdenv 8.has: package (new) 10.rebuild-darwin: 501+ 10.rebuild-linux: 1-10 labels Oct 2, 2018

LnL7 reviewed Oct 2, 2018

View reviewed changes

matthewbauer requested review from basvandijk, peti and ryantm as code owners October 6, 2018 18:43

GrahamcOfBorg added 6.topic: haskell 2.status: merge conflict labels Oct 6, 2018

matthewbauer force-pushed the landmarks branch 2 times, most recently from fdbf496 to 9b9046f Compare November 4, 2018 16:21

LnL7 mentioned this pull request Nov 17, 2018

stress-ng: 0.06.14 -> 0.09.46 #50506

Merged

9 tasks

matthewbauer force-pushed the landmarks branch from 9b9046f to 09399d1 Compare November 24, 2018 19:48

matthewbauer requested review from edolstra, Ericson2314 and FRidh as code owners November 24, 2018 19:48

matthewbauer mentioned this pull request Nov 30, 2018

macOS: ranlib timestamp resolution bugs break parallel builds? #51221

Closed

matthewbauer added this to the 19.03 milestone Dec 1, 2018

zimbatm mentioned this pull request Dec 4, 2018

Fails when /etc/nix missing nix-community/linuxkit-nix#12

Open

matthewbauer force-pushed the landmarks branch from 83d4a73 to 716f9fb Compare December 15, 2018 21:46

matthewbauer force-pushed the landmarks branch 3 times, most recently from 1ad60a8 to a7bf961 Compare December 16, 2018 04:57

matthewbauer force-pushed the landmarks branch from a7bf961 to e6f7f29 Compare December 17, 2018 06:08

matthewbauer merged commit d1dfc92 into NixOS:staging Dec 18, 2018

LnL7 mentioned this pull request Dec 20, 2018

installer: update macOS version check to 10.12.2 NixOS/nix#2594

Merged

matthewbauer mentioned this pull request Jan 2, 2019

Darwin staging branch #53230

Closed

matthewbauer deleted the landmarks branch February 22, 2019 04:16

LnL7 mentioned this pull request Sep 25, 2019

Can't install in macOS 10.11 : "dyld: Symbol not found: _clock_gettime" LnL7/nix-darwin#164

Closed

holymonson mentioned this pull request Jan 31, 2021

darwin.hfs: remove from Libsystem and mark broken #111468

Draft

10 tasks

holymonson added a commit to holymonson/nixpkgs that referenced this pull request Feb 2, 2021

darwin.hfs: remove from Libsystem and mark broken

c7c9843

hfs was introduced in e6f7f29 (NixOS#47678), but it seems nobody will needs its pure headers. We drop it from Libsystem.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update macOS frameworks to 10.12 #47678

Update macOS frameworks to 10.12 #47678

matthewbauer commented Oct 2, 2018 •

edited

GrahamcOfBorg commented Oct 2, 2018

GrahamcOfBorg commented Oct 2, 2018

LnL7 Oct 2, 2018

matthewbauer Oct 2, 2018

LnL7 Oct 2, 2018

veprbl Oct 3, 2018

This comment was marked as outdated.

matthewbauer Oct 3, 2018

veprbl Oct 3, 2018

veprbl Oct 3, 2018 •

edited

matthewbauer Oct 6, 2018

GrahamcOfBorg commented Oct 2, 2018

periklis commented Oct 3, 2018

LnL7 commented Oct 3, 2018

copumpkin commented Oct 9, 2018

infinisil commented Oct 12, 2018

copumpkin commented Oct 12, 2018

LnL7 commented Oct 12, 2018

LnL7 commented Nov 22, 2018

matthewbauer commented Nov 23, 2018

LnL7 commented Nov 24, 2018

GrahamcOfBorg commented Nov 30, 2018

GrahamcOfBorg commented Nov 30, 2018

GrahamcOfBorg commented Nov 30, 2018

matthewbauer commented Dec 1, 2018

matthewbauer commented Dec 3, 2018

LnL7 commented Dec 3, 2018

matthewbauer commented Dec 3, 2018 •

edited

LnL7 commented Dec 3, 2018

matthewbauer commented Dec 3, 2018

matthewbauer commented Dec 15, 2018 •

edited

LnL7 commented Dec 19, 2018

matthewbauer commented Dec 19, 2018

LnL7 commented Dec 20, 2018

Update macOS frameworks to 10.12 #47678

Update macOS frameworks to 10.12 #47678

Conversation

matthewbauer commented Oct 2, 2018 • edited

Motivation for this change

GrahamcOfBorg commented Oct 2, 2018

GrahamcOfBorg commented Oct 2, 2018

LnL7 Oct 2, 2018

Choose a reason for hiding this comment

matthewbauer Oct 2, 2018

Choose a reason for hiding this comment

LnL7 Oct 2, 2018

Choose a reason for hiding this comment

veprbl Oct 3, 2018

Choose a reason for hiding this comment

This comment was marked as outdated.

matthewbauer Oct 3, 2018

Choose a reason for hiding this comment

veprbl Oct 3, 2018

Choose a reason for hiding this comment

veprbl Oct 3, 2018 • edited

Choose a reason for hiding this comment

matthewbauer Oct 6, 2018

Choose a reason for hiding this comment

GrahamcOfBorg commented Oct 2, 2018

periklis commented Oct 3, 2018

LnL7 commented Oct 3, 2018

copumpkin commented Oct 9, 2018

infinisil commented Oct 12, 2018

copumpkin commented Oct 12, 2018

LnL7 commented Oct 12, 2018

LnL7 commented Nov 22, 2018

matthewbauer commented Nov 23, 2018

LnL7 commented Nov 24, 2018

GrahamcOfBorg commented Nov 30, 2018

GrahamcOfBorg commented Nov 30, 2018

GrahamcOfBorg commented Nov 30, 2018

matthewbauer commented Dec 1, 2018

matthewbauer commented Dec 3, 2018

LnL7 commented Dec 3, 2018

matthewbauer commented Dec 3, 2018 • edited

LnL7 commented Dec 3, 2018

matthewbauer commented Dec 3, 2018

matthewbauer commented Dec 15, 2018 • edited

LnL7 commented Dec 19, 2018

matthewbauer commented Dec 19, 2018

LnL7 commented Dec 20, 2018

matthewbauer commented Oct 2, 2018 •

edited

veprbl Oct 3, 2018 •

edited

matthewbauer commented Dec 3, 2018 •

edited

matthewbauer commented Dec 15, 2018 •

edited