We should run a full hydra build for darwin before merging this, IIRC I already tried to build a basic set do you remember if that was ok?

I don't remember, sorry :(. If it was a hydra job would it be on an instance somewhere perhaps?

Well, staging is completely broken.

D'oh re:staging being broken. Are there issues tracking this?

EDIT: by "this" I mean "staging being broken" :)

Should be fixed now, let me start another build.

Well, it was for a day. Unknown option: -DCMAKE_CXX_COMPILER=c++. I'll look into it again.

@LnL7: https://github.com/NixOS/nixpkgs/pull/40539/files#r188388605 fwiw, hopefully fixed soon :).

Beep boop! 😁

Updated to latest staging (conflicts resolved)!

Conflicts resolved!

hmm not sure why eval breaks, do you know why this is happening @Ericson2314 ? No worries if not but since you're poked at these things lately thought I'd ask before diving in too deep :3.

Yes, it's because the Darwin bootstrapping recently was changed referred to llvmPackages_5 directly. Unfortunately this hard-coding of version was necessary because we need to override llvmPackages too, and don't want to break llvmPackages == llvmPackages_5.

I've fixed this and rebased the PR.

Okay, thank you! I think I oops accidentally missed your change... but I think ended up in the same place.

Will try moving back to your update shortly...

(EDIT: this was done almost immediately after posting this, there's just no visual indication of such :))

Maybe we'll update the default by the time LLVM 7.0.1 is released... 😛 😇

Anything blocking this if, let's say, we would target this for 18.09? Any inconvenience could be expected to be caused by merging this?

There's not been a bunch of discussion, so either everything's fine (which I think it probably is) or there hasn't been enough eyes on this.

This has a large impact on darwin builds, similar to a gcc upgrade for the linux side. There's no fundamental problem but it needs more rigorous testing compared to other changes.

1. Make sure stdenv bootstrapping still works, I cleaned some stuff up recently so that might be simpler now.
2. Build a full darwin eval on hydra to get a sense of the amount of newly broken packages, I've used my wip jobset for this in the past.

I'll look into conflicts shortly, but this is long overdue! 😁

They're cutting RC's for 7, haha!

@dtzWill You should be able to drop the overrides in all-packages.nix, that stuff is done in the stdenv phases now.

@dtzWill You should be able to drop the overrides in all-packages.nix, that stuff is done in the stdenv phases now.

Oooo very nice, makes this sort of thing much easier! Resolved commits, haha at this point it's almost entirely s/llvmPackages_5/llvmPackages_6/g 😁. Let's see.

Hmm, I think the llvm6 expressions might have diverged a bit.

output '/nix/store/a04j9abvhgbr59z379n89dcsqi1az8h9-stdenv-darwin' is not allowed to refer to the following paths:
     /nix/store/4ks1n2bvbvpr9gkds5fb65pggfk3m8k8-libxml2-2.9.8

Hmm, I think the llvm6 expressions might have diverged a bit.

After some poking I found that llvm_6 (or clang_6, one of them) has a reference to libxml2 on Linux as well, where the llvm/clang 5 variant does not. Investigating, there are some options set on 6 that don't really make sense while bootstrapping anyway...

Oh I just noticed, this stuff might be important https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/darwin/default.nix#L256

Oh I just noticed, this stuff might be important https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/darwin/default.nix#L256

I may be mistaken but I think that's already handled, and in fact is the only reason this PR isn't a tiny trivial change in top-level :).

I'm not sure what role libxml2 plays but I'm wondering if answer might be to just libxml2 = null in Darwin's bootstrap LLVM? Or maybe mark it allowed if that's preferable?

Unfortunately I'm not in a good position to iterate/test/explore this, thoughts and testing from Darwin folks appreciated :).

@dtzWill I don't think we want to depend on libxml2 (same with ncurses actually), but it's a required build dependency IIRC. libxml2 isn't a runtime dependency now.

$ nix why-depends /nix/store/fv2l2w3w8931jcsv150c1pkb4l315gas-stdenv-darwin /nix/store/k4hyqqki1r4xfaja94r4hrnjlgnxh7v1-bootstrap-stage3-libxml2-2.9.8
/nix/store/fv2l2w3w8931jcsv150c1pkb4l315gas-stdenv-darwin
╚═══setup: …e-epoch-to-latest.sh /nix/store/wm35lw0kgpnjfybj1b1cp78s2ys3rhww-clang-wrapper-6.0.1".defaultBui…
    => /nix/store/wm35lw0kgpnjfybj1b1cp78s2ys3rhww-clang-wrapper-6.0.1
    ╚═══bin/clang: …k disable=SC2193.[[ "/nix/store/nbbznkx8imy3qx2934fwvdspl0h0vl2n-clang-6.0.1/bin/clang" = *++ ]]…
        => /nix/store/nbbznkx8imy3qx2934fwvdspl0h0vl2n-clang-6.0.1
        ╚═══lib/cmake/clang/ClangTargets-release.cmake: …RTED_SONAME_RELEASE "/nix/store/5d98z1k98ajwjgld86mcz3v0sclls7qj-llvm-6.0.1-lib/lib/libclang.dyl…
            => /nix/store/5d98z1k98ajwjgld86mcz3v0sclls7qj-llvm-6.0.1-lib
            ╚═══lib/libLLVM.dylib: ….x.................../nix/store/k4hyqqki1r4xfaja94r4hrnjlgnxh7v1-bootstrap-stage3-libxml2-2.9.8/…
                => /nix/store/k4hyqqki1r4xfaja94r4hrnjlgnxh7v1-bootstrap-stage3-libxml2-2.9.8

$ otool -L /nix/store/5d98z1k98ajwjgld86mcz3v0sclls7qj-llvm-6.0.1-lib/lib/libLLVM.dylib
/nix/store/5d98z1k98ajwjgld86mcz3v0sclls7qj-llvm-6.0.1-lib/lib/libLLVM.dylib:
        /nix/store/5d98z1k98ajwjgld86mcz3v0sclls7qj-llvm-6.0.1-lib/lib/libLLVM.dylib (compatibility version 1.0.0, current version 6.0.1)
        /nix/store/rxdymjw4v1y5hwffdvxwazpk7cfxch8c-libffi-3.2.1/lib/libffi.6.dylib (compatibility version 7.0.0, current version 7.4.0)
        /nix/store/nsk3c08c1k7443kmghjmm1bm9bhf301y-zlib-1.2.11/lib/libz.dylib (compatibility version 1.0.0, current version 1.2.11)
        /nix/store/v588bjpzfd0kqaln1fzvw5v84yk6cz1d-ncurses-6.1/lib/libncursesw.6.dylib (compatibility version 6.0.0, current version 6.0.0)
        /nix/store/ry3vw1brlkhl1dbn3d0j4kxzj66apqvi-Libsystem-osx-10.11.6/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
        /nix/store/k4hyqqki1r4xfaja94r4hrnjlgnxh7v1-bootstrap-stage3-libxml2-2.9.8/lib/libxml2.2.dylib (compatibility version 12.0.0, current version 12.8.0)
        /nix/store/msjrcrk2kv31bd3lghg1v9003lch56y3-libc++-6.0.1/lib/libc++.1.0.dylib (compatibility version 1.0.0, current version 1.0.0)

libxml2 is an optional dependency that is only used to support Windows .manifest files.

I must have been thinking of ncurses, not libxml2. Removing it seems to work, should we do that in the stdenv or can we just drop it everywhere?

diff --git a/pkgs/development/compilers/llvm/6/llvm.nix b/pkgs/development/compilers/llvm/6/llvm.nix
index 787a48416ab..78583906273 100644
--- a/pkgs/development/compilers/llvm/6/llvm.nix
+++ b/pkgs/development/compilers/llvm/6/llvm.nix
@@ -4,7 +4,6 @@
 , python
 , libffi
 , libbfd
-, libxml2
 , ncurses
 , version
 , release_version
@@ -36,7 +35,7 @@ in stdenv.mkDerivation (rec {
   nativeBuildInputs = [ cmake python ]
     ++ stdenv.lib.optional enableManpages python.pkgs.sphinx;
 
-  buildInputs = [ libxml2 libffi ];
+  buildInputs = [ libffi ];
 
   propagatedBuildInputs = [ ncurses zlib ];

I'd enable it for the sake of completeness, and if LLVM is rebuilt after the stdenv bootstrap is complete regardless of libxml2, there are no downsides.

There has been a recent bug report about a miscompilation with -O2 in Clang 6 and later: https://bugs.llvm.org/show_bug.cgi?id=38786 . I'd rather not update without a patch.

If it now depends on libxml2, that isn't a huge deal, we can just add it to the stdenv whitelist.

Edit: just saw the .manifest thing, let's just remove the optional dependency for now to minimize changes

That's not the problem. This is a bootstrap version of libxml2 so we'd need to build llvm/clang twice or pull in python and all it's dependencies at runtime.

Given the darwin bootstrapping and runtime implications I'd argue moving libxml2 to an overridable flag that's disabled by default is a better than increasing the darwin stdenv or adding conditionals.

The final tag for 7 is in.

Closing in favor of #49402 (which attempts to dodge the Darwin bootstrap issue/touchups so it can be solved separately).