darwin: fix Libsystem compatibility for macOS 10.14 #43140

LnL7 · 2018-07-07T10:05:21Z

Motivation for this change

Similar to coretls, libsystem_networking.dylib has been removed in 10.14 so we shouldn't reexport it. Our current bootstrap tarball still references it but that's not a problem in most cases (currently doesn't work 10.13 either) and the changes are already in place to build a new one that will.

Also updated stdenvBootstrapTools.unpack since that's what is tested on hydra.

@copumpkin Why do we reconstruct libSystem.B.dylib after unpacking instead of including it in the tarball?

Things done

Fixes NixOS#42719

matthewbauer · 2018-07-26T21:58:30Z

@LnL7 Unless there is any serious issue, I would strongly suggest merging this into staging ASAP. It can take up to a week to merge staging & it is definitely possible that Apple release macOS Mojave very soon.

LnL7 · 2018-07-27T06:15:08Z

Another issue was mentioned for more recent betas, I don't thinks this is ready yet.

kirelagin · 2018-08-03T23:04:27Z

I don’t think there were any issues with more recent betas compared to the first one. What are you referring to specifically? I’ve been using a branch with this throughout all the betas up to the current one (fourth).

LnL7 · 2018-08-08T20:38:10Z

@GrahamcOfBorg build stdenv

GrahamcOfBorg · 2018-08-08T22:47:13Z

Failure on aarch64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/r8bf63l8pmlk3g2zpykz686amdvnsckl-binutils-wrapper-2.30.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/dlx8s2bigxfn5dvvh4xxcr6l6ny9iwpc-diffutils-3.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/ik355xisz4czp29lzplfwg262r21r9ng-ed-1.14.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/bfkcjhfk08jq33283n2zk72ld78fkppc-findutils-4.6.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/gz86cspcdi8zm02hbjcg8154v94cayci-libtool-2.4.6.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/2ibgx3fdckqa4k2yhi5v1s8rx1h8l0jp-gcc-wrapper-7.3.0.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/5xy8xbqc5x3pb5di30cnvgip8h39wcqg-hook.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/2sdfyrjyv72gkkdvvzsbi8a7mjja94sn-patch-2.7.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/4q4lm6425mzlckhd3glcy968wbcr233i-stdenv-linux.drv': 26 dependencies couldn't be built
error: build of '/nix/store/4q4lm6425mzlckhd3glcy968wbcr233i-stdenv-linux.drv' failed

GrahamcOfBorg · 2018-08-08T23:18:39Z

Failure on x86_64-darwin (full log)

Attempted: stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/736ipzmq75hvl9kbpk5a8bhdycf07hj6-cctools-port-895.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/6r2855c2xzm9kpwrjglkaj7abfhfx4rd-hook.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/vlq5bg91gmrkx8ixv08lran73221swzz-ICU-osx-10.10.5.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/rmpwd2kgjj5kncdrr9vx29s7c97wmn5i-cctools-binutils-darwin.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/0zb5fv29g2xgrw16gd0vq1hbdrinnppm-gnutar-1.30.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/rqm51d6xvzbkz5dnv647dwbsl9yvgsfm-CF-osx-10.10.5.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/03906kad6d6dvr8103rr1m3rvngis1i2-cctools-binutils-darwin-wrapper.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/7xfwp6zh0q5cjvq4zvh1ax131dzf2sj4-clang-wrapper-5.0.2.drv': 11 dependencies couldn't be built
cannot build derivation '/nix/store/9wszbnfybg7s9cg5pw08pv336hby0apy-stdenv-darwin.drv': 36 dependencies couldn't be built
error: build of '/nix/store/9wszbnfybg7s9cg5pw08pv336hby0apy-stdenv-darwin.drv' failed

GrahamcOfBorg · 2018-08-08T23:32:09Z

Timed out, unknown build status on x86_64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/w11inn5hhnpj4dhrlni4vclshiba3x45-binutils-wrapper-2.30.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/rj7n9187xy95g6sy6qn1lzs0fw2dj3bs-diffutils-3.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/y1p9hi8z2mw54m1107p7k4qprf0skspl-ed-1.14.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/v3bs2bvzlr389iji0nbxily2ajyvh1bv-findutils-4.6.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/8rw4xnd8qjbar84p3fdknv7b786g3glz-libtool-2.4.6.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/bwkhnbk5qr2pxdwr10klr345m03b3dfi-gcc-wrapper-7.3.0.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/ycc73vnyckly3yz5szk527p7l9vn0g4g-hook.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/xiphwmh6pyasbpn862hl8yiqvxsd119m-patch-2.7.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/gf7bv4d4pygwmgg4f27fmfdlimcn0ha3-stdenv-linux.drv': 24 dependencies couldn't be built
error: build of '/nix/store/gf7bv4d4pygwmgg4f27fmfdlimcn0ha3-stdenv-linux.drv' failed

GrahamcOfBorg · 2018-08-08T23:47:18Z

Failure on aarch64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/r8bf63l8pmlk3g2zpykz686amdvnsckl-binutils-wrapper-2.30.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/dlx8s2bigxfn5dvvh4xxcr6l6ny9iwpc-diffutils-3.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/ik355xisz4czp29lzplfwg262r21r9ng-ed-1.14.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/bfkcjhfk08jq33283n2zk72ld78fkppc-findutils-4.6.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/gz86cspcdi8zm02hbjcg8154v94cayci-libtool-2.4.6.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/2ibgx3fdckqa4k2yhi5v1s8rx1h8l0jp-gcc-wrapper-7.3.0.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/5xy8xbqc5x3pb5di30cnvgip8h39wcqg-hook.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/2sdfyrjyv72gkkdvvzsbi8a7mjja94sn-patch-2.7.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/4q4lm6425mzlckhd3glcy968wbcr233i-stdenv-linux.drv': 26 dependencies couldn't be built
error: build of '/nix/store/4q4lm6425mzlckhd3glcy968wbcr233i-stdenv-linux.drv' failed

GrahamcOfBorg · 2018-08-09T00:43:30Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: stdenv

Partial log (click to expand)

Scanning dependencies of target LLVMGlobalISel
[ 18%] Building CXX object lib/CodeGen/GlobalISel/CMakeFiles/LLVMGlobalISel.dir/CallLowering.cpp.o
[ 18%] Building CXX object lib/CodeGen/GlobalISel/CMakeFiles/LLVMGlobalISel.dir/IRTranslator.cpp.o
[ 18%] Building CXX object lib/CodeGen/GlobalISel/CMakeFiles/LLVMGlobalISel.dir/InstructionSelect.cpp.o
building of '/nix/store/9y0adcgmk53xfmxs936x5wd694f80dj7-llvm-5.0.2.drv' timed out after 1200 seconds
cannot build derivation '/nix/store/3n2l2a541fwmbbl8gxnaqfxa8zd784rf-clang-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/s8819pnh3bia3hj243p6ixmn80gcsbwc-compiler-rt-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/7xfwp6zh0q5cjvq4zvh1ax131dzf2sj4-clang-wrapper-5.0.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/9wszbnfybg7s9cg5pw08pv336hby0apy-stdenv-darwin.drv': 4 dependencies couldn't be built
error: build of '/nix/store/9wszbnfybg7s9cg5pw08pv336hby0apy-stdenv-darwin.drv' failed

GrahamcOfBorg · 2018-08-09T00:51:51Z

Timed out, unknown build status on x86_64-linux (full log)

Attempted: stdenv

Partial log (click to expand)

cannot build derivation '/nix/store/w11inn5hhnpj4dhrlni4vclshiba3x45-binutils-wrapper-2.30.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/rj7n9187xy95g6sy6qn1lzs0fw2dj3bs-diffutils-3.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/y1p9hi8z2mw54m1107p7k4qprf0skspl-ed-1.14.2.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/v3bs2bvzlr389iji0nbxily2ajyvh1bv-findutils-4.6.0.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/8rw4xnd8qjbar84p3fdknv7b786g3glz-libtool-2.4.6.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/bwkhnbk5qr2pxdwr10klr345m03b3dfi-gcc-wrapper-7.3.0.drv': 5 dependencies couldn't be built
cannot build derivation '/nix/store/ycc73vnyckly3yz5szk527p7l9vn0g4g-hook.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/xiphwmh6pyasbpn862hl8yiqvxsd119m-patch-2.7.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/gf7bv4d4pygwmgg4f27fmfdlimcn0ha3-stdenv-linux.drv': 24 dependencies couldn't be built
error: build of '/nix/store/gf7bv4d4pygwmgg4f27fmfdlimcn0ha3-stdenv-linux.drv' failed

copumpkin

Looks good, thanks! I haven't tried building it though and it seems like borg is having trouble 😦

LnL7 · 2018-08-09T18:25:00Z

I downloaded the latest beta and will try to test this properly during the weekend.

@copumpkin The ofborg builds where intentional, I was testing the new 'Timed out' detection. 😄

kirelagin · 2018-08-10T20:17:33Z

I am still getting

HandshakeFailed (Error_Protocol ("certificate has unknown CA",True,UnknownCa))

when trying to setup a TLS connection from Haskell code and right before it I see this:

dyld: Symbol not found: ___CFObjCIsCollectable
  Referenced from: /nix/store/wkqjqwyfi1nn2hg9k0as03lpcmalqxzj-libsecurity_utilities-osx-10.7.5/lib/libsecurity_utilities.0.dylib
  Expected in: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
 in /nix/store/wkqjqwyfi1nn2hg9k0as03lpcmalqxzj-libsecurity_utilities-osx-10.7.5/lib/libsecurity_utilities.0.dylib

I am not sure if it is related to Mojave to be honest, i.e. I have a vague memory of having some TLS-related issues even on High Sierra.

matthewbauer · 2018-08-10T20:22:13Z

Do you have an example program? I think I made a few changes to libsecurity recently that hopefully are not causing that issue. I can verify if it is an issue in High Sierra.

It's possible we have too old of a libsecurity. We can try to update it.

kirelagin · 2018-08-11T11:01:25Z

I had this issue with http-client but for simplicity here is a program using req:

{-# LANGUAGE OverloadedStrings #-}

import Control.Monad
import Data.Default.Class
import Network.HTTP.Req

main :: IO ()
main = runReq def $
  void $ req GET (https "api.github.com" /: "meta") NoReqBody ignoreResponse mempty

› nix-shell -p 'haskellPackages.ghcWithPackages (hp: [ hp.req ])' --run 'runhaskell Main.hs'
dyld: Symbol not found: ___CFObjCIsCollectable
  Referenced from: /nix/store/wkqjqwyfi1nn2hg9k0as03lpcmalqxzj-libsecurity_utilities-osx-10.7.5/lib/libsecurity_utilities.0.dylib
  Expected in: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
 in /nix/store/wkqjqwyfi1nn2hg9k0as03lpcmalqxzj-libsecurity_utilities-osx-10.7.5/lib/libsecurity_utilities.0.dylib
Main.hs: VanillaHttpException (HttpExceptionRequest Request {
  host                 = "api.github.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/meta"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (InternalException (HandshakeFailed (Error_Protocol ("certificate has unknown CA",True,UnknownCa)))))

The expected response is a 403 because User-Agent is not set:

$ nix-shell -p 'haskellPackages.ghcWithPackages (hp: [ hp.req ])' --run 'runhaskell Main.hs'
Main.hs: VanillaHttpException (HttpExceptionRequest Request {
  host                 = "api.github.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/meta"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (StatusCodeException (Response {responseStatus = Status {statusCode = 403, statusMessage = "Forbidden"}, responseVersion = HTTP/1.0, responseHeaders = [("Cache-Control","no-cache"),("Connection","close"),("Content-Type","text/html")], responseBody = (), responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}) "Request forbidden by administrative rules. Please make sure your request has a User-Agent header (http://developer.github.com/v3/#user-agent-required). Check https://developer.github.com for other possible causes.\n"))

These no longer exist on 10.14 Fixes NixOS#42719

lukateras · 2018-08-13T23:32:12Z

@LnL7 Can this be merged into staging as is? This seems to be a major improvement over current Nixpkgs situation on macOS Mojave.

LnL7 · 2018-08-14T06:57:15Z

I'm still running some builds, but my plan is to merge this once those are finished.

LnL7 · 2018-08-14T21:01:45Z

This should be ready for staging now, but I'll wait with the backport until we have some results for that in case there are some side-effects.

@kirelagin Feel free to open a separate issue for that libsecurity/CF problem, looks like that's probably something separate.

copumpkin · 2018-08-15T02:26:17Z

Yay! thanks 😄

domenkozar · 2018-09-27T12:16:12Z

@kirelagin did you manage to resolve your issue? macOS user is reporting the same issue using latest nixpkgs master: cachix/cachix#134 (comment)

domenkozar · 2018-09-27T17:07:55Z

Aha seems like it's still broken: #45042

matthewbauer · 2018-09-27T18:59:55Z

@domenkozar That's a little bit of a different issue that effects only things using darwin.Security. I think it might be time to just use the impure darwin.apple_sdk.frameworks.Security. It's unfortunate because it's using Apple-built stuff but I think it would lead to less issues like this.

LnL7 added 2 commits July 7, 2018 11:42

Libsystem: remove libsystem_network.dylib from reexport list

52b8b68

Fixes NixOS#42719

stdenv-bootstrap-tools: update unpack to use $reexportedLibrariesFile

5024e4a

LnL7 added the 9.needs: port to stable A PR needs a backport to the stable release. label Jul 7, 2018

LnL7 requested a review from copumpkin July 7, 2018 10:05

GrahamcOfBorg added 6.topic: darwin Running or building packages on Darwin 6.topic: stdenv Standard environment 10.rebuild-darwin-stdenv 10.rebuild-darwin: 501+ 10.rebuild-linux: 0 labels Jul 7, 2018

LnL7 added the 2.status: work-in-progress label Jul 10, 2018

matthewbauer approved these changes Jul 13, 2018

View reviewed changes

copumpkin approved these changes Aug 9, 2018

View reviewed changes

Libsystem: remove version symbols

54f901d

These no longer exist on 10.14 Fixes NixOS#42719

LnL7 removed the 2.status: work-in-progress label Aug 14, 2018

LnL7 merged commit ea1542e into NixOS:staging Aug 14, 2018

LnL7 deleted the macos-10.14 branch August 14, 2018 21:02

kirelagin mentioned this pull request Aug 14, 2018

macOS 10.14 Mojave #42719

Closed

LnL7 mentioned this pull request Aug 21, 2018

darwin: macOS 10.14 compatibility #45437

Merged

9 tasks

domenkozar mentioned this pull request Sep 27, 2018

Broken on macOS mojave cachix/cachix#134

Closed

matthewbauer mentioned this pull request Sep 27, 2018

try-reflex broken on macOS Mojave reflex-frp/reflex-platform#374

Closed

samueldr removed the 9.needs: port to stable A PR needs a backport to the stable release. label Apr 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

darwin: fix Libsystem compatibility for macOS 10.14 #43140

darwin: fix Libsystem compatibility for macOS 10.14 #43140

LnL7 commented Jul 7, 2018

matthewbauer commented Jul 26, 2018

LnL7 commented Jul 27, 2018

kirelagin commented Aug 3, 2018

LnL7 commented Aug 8, 2018 •

edited

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 9, 2018

GrahamcOfBorg commented Aug 9, 2018

copumpkin left a comment

LnL7 commented Aug 9, 2018

kirelagin commented Aug 10, 2018

matthewbauer commented Aug 10, 2018 •

edited

kirelagin commented Aug 11, 2018 •

edited

lukateras commented Aug 13, 2018

LnL7 commented Aug 14, 2018

LnL7 commented Aug 14, 2018

copumpkin commented Aug 15, 2018

domenkozar commented Sep 27, 2018

domenkozar commented Sep 27, 2018

matthewbauer commented Sep 27, 2018

darwin: fix Libsystem compatibility for macOS 10.14 #43140

darwin: fix Libsystem compatibility for macOS 10.14 #43140

Conversation

LnL7 commented Jul 7, 2018

Motivation for this change

Things done

matthewbauer commented Jul 26, 2018

LnL7 commented Jul 27, 2018

kirelagin commented Aug 3, 2018

LnL7 commented Aug 8, 2018 • edited

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 8, 2018

GrahamcOfBorg commented Aug 9, 2018

GrahamcOfBorg commented Aug 9, 2018

copumpkin left a comment

Choose a reason for hiding this comment

LnL7 commented Aug 9, 2018

kirelagin commented Aug 10, 2018

matthewbauer commented Aug 10, 2018 • edited

kirelagin commented Aug 11, 2018 • edited

lukateras commented Aug 13, 2018

LnL7 commented Aug 14, 2018

LnL7 commented Aug 14, 2018

copumpkin commented Aug 15, 2018

domenkozar commented Sep 27, 2018

domenkozar commented Sep 27, 2018

matthewbauer commented Sep 27, 2018

LnL7 commented Aug 8, 2018 •

edited

matthewbauer commented Aug 10, 2018 •

edited

kirelagin commented Aug 11, 2018 •

edited