Xcode integration for nix #1048

pikajude · 2016-08-31T21:55:31Z

After several months of hacks, weird runtime bugs, and frustrations over files missing from Apple's open-source releases, I've become increasingly convinced that this is the most viable option for anyone who wishes to build Darwin-specific projects that extensively use system frameworks.

This fork adds constants xcodeSDKRoot and xcodeVersion, which are detected during evaluator initialization based on the current Xcode install.

In addition to this I have a fork https://github.com/pikajude/darwinixpkgs, which I don't think is ready to PR yet, but which features:

A new darwin stdenv that integrates with Xcode
A sandbox profile that allows access to a subset of SDK headers (since they include headers for Python, Ruby, MySQL, etc)
First-class support for framework dependencies

cstrahan · 2016-08-31T22:39:23Z

Looks interesting!

Are you planning to put this into a state such that it can be merged (rather than maintained as a separate fork)?

How does this relate to the pure stdenv? Do you envision that the pure stdenv should be abandoned in favor of this?

pikajude · 2016-08-31T23:03:53Z

@cstrahan I think the pure stdenv should be an alternative but this should be the default.

I'll work on the merge conflict later tonight, but I wanted to get this PR opened to get some people's opinions first.

veprbl · 2016-08-31T23:34:53Z

@pikajude
In that case you will have to have hydra builds for every supported OSX target, or you will have to throw the purity out of the window, I suppose. You also get occasional broken builds after SDK gets updated (like brew has).

pikajude · 2016-08-31T23:36:28Z

@veprbl Correct, I'd like to have a Hydra instance for each of the most recent 2 or 3 Xcode versions.

veprbl · 2016-08-31T23:50:04Z

@pikajude
There also will be people wanting to use Command Line Tools instead of Xcode.

For completeness of this PR could you please name some applications you can get working this way and which closed-source frameworks they depend on?

pikajude · 2016-09-01T07:32:04Z

@veprbl xhyve (Kernel and hypervisor frameworks), anything touching NSString, emacs with proper Cocoa support.

The problem is not whether we can compile the programs but whether their runtime behavior will be predictable and correct. The headers that pure-darwin includes occasionally are inconsistent with the actual object files in ways that are very difficult to predict and debug—for example, compiling phantomjs2 with pure-darwin's CF headers would cause a runtime failure whenever NSString was used—and, more importantly, these inconsistencies vary between OS versions, meaning that we need some way of determining what libc version, at least, the end user is on.

I decided that the line of purity was blurry enough that I was willing to give it up a little more in exchange for a set of libraries and headers that are guaranteed to work together.

pikajude · 2016-09-01T07:34:34Z

@veprbl For the reason given above, I can't provide a fallback for people who want to use the CLT since it doesn't include the standard headers. They would need to use the pure-darwin stdenv instead.

veprbl · 2016-09-01T07:43:41Z

@pikajude
This makes sense. But I don't understand the CLT problem. It can compile things so there are headers somewhere, maybe not presented as an SDK, but they are there, right?

copumpkin · 2016-09-01T11:49:37Z

To be clear, I think there's still some disagreement out there (at least with me) as to whether purity is worth throwing out just yet. I don't think we've really exhausted our options in that regard, and although I appreciate the contribution, I don't personally want darwinixpkgs to become the default. I don't have a strong opinion about this particular PR, since it's enabling darwinixpkgs as opposed to changing nixpkgs to switch stdenv.

Anyway, I acknowledge the difficulties you list as motivations, but as mentioned on IRC, I do think there are sensible ways to fix those while not throwing out the current approach. Getting people to do the work has been hard, but I don't think adding another stdenv for them to divide their attention with is going to fix that.

edolstra · 2016-09-01T12:08:59Z

Requiring Xcode seems like a distinct step back... I'm very happy that Nix on Darwin no longer requires Xcode and things (mostly) just work out of the box. I don't think our build machines even have Xcode installed at the moment.

cstrahan · 2016-09-06T00:03:13Z

I think the pure stdenv is pretty awesome. If we can get it to work in all cases, that's a huge win.

But I can also see the utility in having something that closes the gap, while sacrificing some purity. If we can increase adoption, I think that'll lead to more power to push along a pure approach (more people / bigger names to pester Apple when necessary, and more nixpkgs contributions), and I'd bet that having this impure option would help us get those users that we'd otherwise lose due to frustrations with missing/broken packages.

As long as this doesn't break the pure stdenv, and it doesn't bloat the code too much, I think it makes sense to merge something like this.

copumpkin · 2016-09-06T00:17:54Z

Mostly agree with Charles here: I could definitely see the impure one being useful but I'm also afraid that it'll become infectious and people will start using it by default if they run into issues with the other one, which makes it harder for us to know about problems and improve the main one. Perhaps we could use the new lib.warn function to emit a little warning telling people that they're using the impure stdenv mkDerivation and that there could be more system-dependent issues. Do you think that would be too annoying for end users?

On Sep 5, 2016, at 20:03, Charles Strahan notifications@github.com wrote:

I think the pure stdenv is pretty awesome. If we can get it to work in all cases, that's a huge win.

But I can also see the utility in having something that closes the gap, while sacrificing some purity. If we can increase adoption, I think that'll lead to more power to push along a pure approach (more people / bigger names to pester Apple when necessary, and more nixpkgs contributions), and I'd bet that having this impure option would help us get those users that we'd otherwise lose due to frustrations with missing/broken packages.

As long as this doesn't break the pure stdenv, and it doesn't bloat the code too much, I think it makes sense to merge something like this.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

pikajude · 2017-01-07T06:59:29Z

@edolstra Understood that this seems like a step back. Two things:

I intend the Xcode stdenv to be totally separate and opt-in, so the build machines wouldn't need Xcode installed in the first place—this could be seen as a "Darwin power users" feature.
Evaluation won't actually require xcode unless the value of one of the xcode* builtins is forced, so we could just check for their existence without causing a runtime error. (At least, I think that's the case. I'll check once I have some spare time to spend on this).

edolstra · 2017-01-09T15:27:54Z

src/libexpr/primops.cc

+ * Xcode integration
+ *************************************************************/
+
+std::string exec(const char* cmd) {


Please use runProgram() from util.cc which already does this.

edolstra · 2017-01-09T15:31:30Z

src/libexpr/primops.cc

+    addConstant("__xcodeVersion", v);
+
+    mkString(v, lookupSDKRoot());
+    addConstant("__xcodeSDKRoot", v);


This is not lazy, so it will run two child processes at startup time, and throw an exception if Xcode is not installed. I think the easiest way to make this lazy is to turn it into a primop with one dummy argument, and then use mkApp to pass the dummy argument.

edolstra · 2017-01-09T15:31:46Z

src/libstore/build.cc

@@ -1680,7 +1680,7 @@ void DerivationGoal::startBuilder()
    {
        string x = settings.get("build-use-sandbox",
            /* deprecated alias */
-            settings.get("build-use-chroot", string("false")));
+            settings.get("build-use-chroot", string("relaxed")));


Changing the default sandbox mode probably belong in a separate PR.

edolstra · 2017-01-09T15:33:32Z

src/libstore/build.cc

+    tmpDirInSandbox =
+#if !__APPLE__
+      useChroot ? canonPath("/tmp", true) + "/nix-build-" + drvName + "-0" :
+#endif


Maybe add a comment explaining why this is conditional on !__APPLE__?

edolstra · 2017-01-09T15:34:20Z

src/resolve-system-dependencies/resolve-system-dependencies.cc

@@ -41,6 +43,11 @@ std::string findDylibName(bool should_swap, ptrdiff_t dylib_command_start) {
 }

 std::set<std::string> runResolver(const Path & filename) {
+    if(getFileType(filename) != DT_REG) {
+        printMsg(lvlError, format("%1%: not a regular file") % filename);


=> printError("%s: not a regular file", filename).

edolstra · 2017-01-09T15:35:13Z

src/resolve-system-dependencies/resolve-system-dependencies.cc

@@ -157,6 +185,10 @@ int main(int argc, char ** argv) {
    return handleExceptions(argv[0], [&]() {
        initNix();

+        if(argc < 2) {
+            throw Error(format("missing derivation argument"));


format is not needed here (or indeed almost anywhere).

edolstra · 2017-01-09T15:35:44Z

release.nix

@@ -27,7 +27,7 @@ let
          [ curl bison flex perl libxml2 libxslt bzip2 xz
            pkgconfig sqlite libsodium boehmgc
            docbook5 docbook5_xsl
-            autoconf-archive
+            autoconf-archive libyamlcpp


This should be conditional on stdenv.isDarwin since we don't need libyamlcpp elsewhere.

edolstra · 2017-01-09T15:36:38Z

README.md

+
+``` bash
+$ nix-env -f release.nix -iA build.x86_64-darwin
+```


We shouldn't duplicate the manual here, so please don't add installation instructions here.

edolstra · 2017-01-09T15:37:33Z

README.md

 purity aspects, a lot of issues found in traditional package managers don't
 appear with Nix.

+*Darwinix* is an OSX-specific port of nix that integrates more tightly with Xcode,
+avoiding a host of problems introduced by the fact that the vast majority of OSX's system
+libraries are closed-source.


IMHO it's just confusing to introduce a separate name for Nix on Darwin.

edolstra · 2017-01-09T15:37:58Z

corepkgs/buildenv.nix

@@ -25,19 +25,16 @@ derivation {
  # Also don't bother substituting.
  allowSubstitutes = false;

+  __impureHostDeps = [
+    "/usr/lib/libSystem.dylib"
+  ];


This should be conditional on Darwin.

Jude Taylor added 16 commits August 13, 2016 15:34

add xcode constants

b13dfa5

disable tmpdir canonicalization on unsupported platforms

f23c8b0

replace old script

402dbec

fix cache path

a929026

add missing require

e641348

xcode detection

9459eb3

cleanup

5ae888d

fix two compilation problems

8f133c6

add /Applications to allowed impure host deps for Xcode

e8fd204

update readme

b34ff31

fix (most) tests

dc3efba

fix script list

e764f76

implement in c++

b8999e4

implement in C++

830b95f

this now needs to be canonicalized

c33c607

fix indentation and capitalization

6c70d35

pikajude force-pushed the master branch from 6c70d35 to 3dc2888 Compare September 1, 2016 07:25

add xcode constants

8897656

Jude Taylor added 10 commits September 17, 2016 14:09

fix (most) tests

0fc18c4

implement in c++

b083adf

implement in C++

ce7d4a9

this now needs to be canonicalized

2f01330

add installation to readme

799322c

Merge branch 'master' of https://github.com/pikajude/darwinix

d2db957

disable installCheck

16398d9

Merge branch 'master' of https://github.com/NixOS/nix

a20c568

darwin has no execvpe

0eb1d3a

Merge branch 'master' of https://github.com/NixOS/nix

e9bc018

pikajude force-pushed the master branch from 3dc2888 to e9bc018 Compare October 21, 2016 20:35

domenkozar added the macos Nix on macOS, aka OS X, aka darwin label Nov 3, 2016

Jude Taylor added 4 commits November 9, 2016 09:45

Merge branch 'master' of https://github.com/NixOS/nix

749d908

Merge branch 'master' of https://github.com/NixOS/nix

7c0f63f

Merge branch 'master' of https://github.com/NixOS/nix

85ffb90

assign environ to a temp variable to ensure liveness

24834f3

edolstra reviewed Jan 9, 2017

View reviewed changes

shlevy added the backlog label Apr 1, 2018

shlevy assigned copumpkin Apr 1, 2018

pikajude closed this Jan 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Xcode integration for nix #1048

Xcode integration for nix #1048

pikajude commented Aug 31, 2016

cstrahan commented Aug 31, 2016

pikajude commented Aug 31, 2016

veprbl commented Aug 31, 2016

pikajude commented Aug 31, 2016

veprbl commented Aug 31, 2016

pikajude commented Sep 1, 2016 •

edited

pikajude commented Sep 1, 2016

veprbl commented Sep 1, 2016

copumpkin commented Sep 1, 2016

edolstra commented Sep 1, 2016

cstrahan commented Sep 6, 2016

copumpkin commented Sep 6, 2016

pikajude commented Jan 7, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

edolstra Jan 9, 2017

Xcode integration for nix #1048

Xcode integration for nix #1048

Conversation

pikajude commented Aug 31, 2016

cstrahan commented Aug 31, 2016

pikajude commented Aug 31, 2016

veprbl commented Aug 31, 2016

pikajude commented Aug 31, 2016

veprbl commented Aug 31, 2016

pikajude commented Sep 1, 2016 • edited

pikajude commented Sep 1, 2016

veprbl commented Sep 1, 2016

copumpkin commented Sep 1, 2016

edolstra commented Sep 1, 2016

cstrahan commented Sep 6, 2016

copumpkin commented Sep 6, 2016

pikajude commented Jan 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pikajude commented Sep 1, 2016 •

edited