stdenv/darwin: integrate a new CoreFoundation #46704

copumpkin · 2018-09-15T16:16:35Z

This also updates the bootstrap tool builder to LLVM 5, but not the ones we actually use for bootstrap. I'll make that change in a subsequent commit so as to provide traceable provenance of the bootstrap tools.

Motivation for this change

We have a horribly out of date CoreFoundation in our stdenv and this should modernize it a bit, allowing us to finally advance some other packages and also remove some workarounds.

Things done

Here's an evaluation of the last state of this prior to me rebasing onto staging. I didn't feel like running a whole new eval after rebasing. Note that I also added the CFFileDescriptor.h stuff to support watchman properly since that eval, so if it complains that watchman is now failing, it's actually healthy again.

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

copumpkin · 2018-09-15T16:45:04Z

pkgs/development/libraries/libuv/default.nix

@@ -40,7 +40,7 @@ stdenv.mkDerivation rec {
        "tcp_open" "tcp_write_queue_order" "tcp_try_write" "tcp_writealot"
        "multiple_listen" "delayed_accept"
        "shutdown_close_tcp" "shutdown_eof" "shutdown_twice" "callback_stack"
-        "tty_pty"
+        "tty_pty" "condvar_5"


This was failing a bunch for me locally so I just turned it off. Seems unlikely that I broke that test given how flaky their other tests seem to be.

copumpkin · 2018-09-15T17:14:50Z

My plan is to merge this, trigger a staging eval, then submit a PR against staging for my LTO-enabled stdenv, merge that, then regenerate bootstrap tools make a new PR using those.

This also updates the bootstrap tool builder to LLVM 5, but not the ones we actually use for bootstrap. I'll make that change in a subsequent commit so as to provide traceable provenance of the bootstrap tools.

copumpkin · 2018-09-15T20:44:17Z

I've merged it but will be around for another week and will happily back it out if someone critical breaks as a result. I think qt4 will break for a bit due to its use of a long-deprecated API in CF, but I opened #46719 in response to that. I'd rather just kill qt4 honestly.

I still welcome feedback on this PR!

matthewbauer · 2018-10-04T17:54:30Z

@copumpkin This ends up breaking building on macOS 10.11:

Undefined symbols for architecture x86_64:
  "_os_unfair_lock_lock", referenced from:
      __CFSuddenTerminationDisable in CFUtilities.c.o
      __CFSuddenTerminationEnable in CFUtilities.c.o
      __CFSuddenTerminationExitIfTerminationEnabled in CFUtilities.c.o
      __CFSuddenTerminationExitWhenTerminationEnabled in CFUtilities.c.o
      __CFMachPortCreateWithPort2 in CFMachPort.c.o
      ____CFMachPortCreateWithPort2_block_invoke in CFMachPort.c.o
      ___CFMachPortDeallocate in CFMachPort.c.o
      ...
  "_os_unfair_lock_unlock", referenced from:
      __CFSuddenTerminationDisable in CFUtilities.c.o
      __CFSuddenTerminationEnable in CFUtilities.c.o
      __CFSuddenTerminationExitIfTerminationEnabled in CFUtilities.c.o
      __CFSuddenTerminationExitWhenTerminationEnabled in CFUtilities.c.o
      __CFMachPortCreateWithPort2 in CFMachPort.c.o
      ____CFMachPortCreateWithPort2_block_invoke in CFMachPort.c.o
      ___CFMachPortDeallocate in CFMachPort.c.o
      ...
ld: symbol(s) not found for architecture x86_64

10.12 seems to be fine. Would it be okay to bump our minimum version to 10.12 everywhere?

copumpkin · 2018-10-09T18:37:55Z

Probably fine for now, maybe with a note that the major blocker is those locks so if someone cares enough and can put in a bit of work, we could probably keep it working.

LnL7 · 2018-10-09T18:46:45Z

There's also an issue with wxmac (CFURLGetFSRef) https://hydra.nixos.org/build/82300955. It sounds familiar, but I can't remember if it's something you already knew about or fixed in another build.

copumpkin · 2018-10-10T12:50:19Z

I've seen it but haven't done anything about it. If absolutely necessary, we could probably manually resurrect the shitty *FSRef stuff by manually transplanting and adjusting the code from the old CF releases into the new swift one, but I'd really prefer not to if we can avoid it. Apple doesn't want to support it, and I certainly don't if they don't 😄

LnL7 · 2018-10-24T10:41:50Z

Looks like I have a solution using cf-private, not ideal but better than creating and maintaining a whole bunch of patches.

copumpkin · 2018-10-24T19:50:54Z

Sounds good, thanks! I think I left a long note on cf-private asking people who use it to say what they need it for, so we can progressively rip it out as the situation changes.

LnL7 · 2018-10-24T23:39:26Z

I added some comments to clarify why it's there, #49081.

copumpkin added the 6.topic: darwin Running or building packages on Darwin label Sep 15, 2018

copumpkin requested review from matthewbauer and LnL7 September 15, 2018 16:16

copumpkin requested a review from FRidh as a code owner September 15, 2018 16:16

GrahamcOfBorg added 6.topic: python 6.topic: stdenv Standard environment labels Sep 15, 2018

copumpkin force-pushed the new-cf-staging branch from af0fa32 to 7e7a27a Compare September 15, 2018 16:44

copumpkin commented Sep 15, 2018

View reviewed changes

copumpkin force-pushed the new-cf-staging branch from 7e7a27a to dce0e1d Compare September 15, 2018 16:51

copumpkin mentioned this pull request Sep 15, 2018

darwin.apple_sdk: update to 10.13 #46699

Closed

9 tasks

GrahamcOfBorg added 10.rebuild-darwin-stdenv 8.has: package (new) 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Sep 15, 2018

copumpkin force-pushed the new-cf-staging branch from dce0e1d to 2fff161 Compare September 15, 2018 17:10

stdenv/darwin: integrate a new CoreFoundation

4efd405

This also updates the bootstrap tool builder to LLVM 5, but not the ones we actually use for bootstrap. I'll make that change in a subsequent commit so as to provide traceable provenance of the bootstrap tools.

copumpkin force-pushed the new-cf-staging branch from 2fff161 to 4efd405 Compare September 15, 2018 20:07

copumpkin merged commit b91ad0e into NixOS:staging Sep 15, 2018

orivej-nixos pushed a commit that referenced this pull request Sep 15, 2018

Fix comment location after #46704

06c6312

matthewbauer mentioned this pull request Oct 2, 2018

Update macOS frameworks to 10.12 #47678

Merged

veprbl added this to Infrastructure (stdenv, sandboxing, etc.) in Darwin Oct 31, 2019

LnL7 moved this from Stdenv to Done in Darwin Oct 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stdenv/darwin: integrate a new CoreFoundation #46704

stdenv/darwin: integrate a new CoreFoundation #46704

copumpkin commented Sep 15, 2018

copumpkin Sep 15, 2018

copumpkin commented Sep 15, 2018

copumpkin commented Sep 15, 2018

matthewbauer commented Oct 4, 2018

copumpkin commented Oct 9, 2018

LnL7 commented Oct 9, 2018

copumpkin commented Oct 10, 2018

LnL7 commented Oct 24, 2018

copumpkin commented Oct 24, 2018

LnL7 commented Oct 24, 2018

stdenv/darwin: integrate a new CoreFoundation #46704

stdenv/darwin: integrate a new CoreFoundation #46704

Conversation

copumpkin commented Sep 15, 2018

Motivation for this change

Things done

copumpkin Sep 15, 2018

Choose a reason for hiding this comment

copumpkin commented Sep 15, 2018

copumpkin commented Sep 15, 2018

matthewbauer commented Oct 4, 2018

copumpkin commented Oct 9, 2018

LnL7 commented Oct 9, 2018

copumpkin commented Oct 10, 2018

LnL7 commented Oct 24, 2018

copumpkin commented Oct 24, 2018

LnL7 commented Oct 24, 2018