swift: init at 3.1 #22098

dtzWill · 2017-01-24T16:28:12Z

Also add dependency 'libblocksruntime'.

#11463

Motivation for this change

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Review/testing/feedback requested!

Short-term TODO:

Check what value is appropriate for 'license'
Is it okay to put a copy of build-presets.ini in the tree?
Revisit packaging 3.0.2 (or a different release)
Fix use of paxctl on machines without grsecurity enabled

Things to pay attention to, and other thoughts:
Misc notes:

Does not build nicely on Linux (non-NixOS) without sandbox, but I don't think that's a serious problem.
swiftc, swift-pm, and the swift REPL seem to work in my limited testing.
Tests are not executed, might be nice to fix and enable as many of these as possible.
A number of problems occur due to swift projects wanting everything to be prefixed with "/usr", to the point that this is hard-coded in a number of places. I patch around the ones that broke functionality I tested but there's probably more.
Normally we use a cc-wrapper to fixup clang invocations but here I kludge them directly into the compiler O:). This is done so the just-built clang can be used to build the rest of the code while still being able to find the paths to their build dependencies. Even if this is okay, maybe the installed clang should be unmangled and wrapped with cc-wrapper?

dtzWill · 2017-01-24T17:16:44Z

For the curious, working on a variant of this using the 3.0.2 release on this branch: https://github.com/dtzWill/nixpkgs/tree/feature/swift-3.0.2

Don't even know if that branch builds yet though, but thought I'd share :).

dtzWill · 2017-01-24T19:23:58Z

3.0.2 needs multiple patches to avoid crashing during compilation, I've patched a few in that branch but giving up on making that work for the time being. Hopefully 3.1+ is suitable and has an official release soon enough :).

copumpkin · 2017-02-04T03:23:57Z

pkgs/development/compilers/swift/default.nix

+    cp -r ${sources.llbuild} llbuild
+    cp -r ${sources.pm} swiftpm
+    cp -r ${sources.xctest} swift-corelibs-xctest
+    cp -r ${sources.foundation} swift-corelibs-foundation


Does the build automatically build all this stuff? I'm surprised to see these going in.

Yep! What's built is described by the preset we're using, which is a modified version of what they use on the linux buildbots:
https://github.com/NixOS/nixpkgs/pull/22098/files#diff-207758b47bdae6fde307317858e26f47R715
(which references the mixin_linux_installation preset: https://github.com/NixOS/nixpkgs/pull/22098/files#diff-207758b47bdae6fde307317858e26f47R681).

It's possible we can break these apart eventually but I don't think it would be easy or even worth it--AFAICT the components are all very much meant to be built/tested together.

Also this does take a while to build... :).

benley · 2017-02-26T05:46:26Z

How much remains to be done on this PR before it's suitable for merging to master?

dtzWill · 2017-02-28T19:37:38Z

I think a good start would be having one or more people express interest and confirm this builds and works for them/their use cases. Any takers? :)

benley · 2017-02-28T19:57:04Z

I've been using it over the last few days to do some intro-to-swift exercises, and can confirm that the basic functionality does seem to work: swift build and swift test on simple packages, and the swift lldb REPL. I don't yet know how to use the swift repl in a way that makes things like import Foundation work.

dtzWill · 2017-03-06T15:03:32Z

@benley great to hear! Thanks for reporting the bit about 'import Foundation', I can confirm that it doesn't work here either. This appears to be a known issue upstream: https://bugs.swift.org/browse/SR-3794

One of the proposed workarounds there is to add -I$swift/lib/swift/clang/include, which fixes it for me:

$ swift -I /nix/store/pa0gkjxvzzyyq1rqld0b1klnbk0lg3zn-swift-3.1-2017-03-03/lib/swift/clang/include

(your hash might be different; I forget which tree I installed swift from)

Not sure if this is something they need to fix or if it would make sense to auto-add it with a wrapper or some source patching.

dtzWill · 2017-03-14T14:09:08Z

squasshed + rebased, updated to 3.1-2017-03-13.

puffnfresh · 2017-03-28T08:25:49Z

Works for me 👍

We should probably bump to 3.1 final and ship it

dtzWill · 2017-03-28T14:25:09Z

Final has been released? Woohoo! Pushed updated expression, still rebuilding to test locally though :).

matthewbauer · 2017-03-28T15:11:44Z

Looks good! One nitpick though: is there a way we can get around build-presets.ini? I see the file might also be in utils/build-presets.ini, can we just use that?

copumpkin · 2017-03-28T15:54:07Z

Yeah, that build-presets.ini seems like a maintenance nightmare, especially if upstream evolves the format or options available. If we're making changes to it, can we just bundle a patch to the upstream file to make it clearer what we care about and what doesn't matter?

Apart from that it looks good

dtzWill · 2017-03-28T17:07:35Z

Sure thing, I'll take care of this later today (meetings and such :)).

copumpkin · 2017-03-28T17:08:59Z

pkgs/development/compilers/swift/default.nix

+      sha256 = "0820mx1ghfnk4p5595r1f313y1699jwi61zghfbwak5wgwgy7n4x";
+    };
+    llvm = fetch {
+      repo = "swift-llvm";


Do you know if it really needs its own custom llvm or if it can use our pre-existing packages? I fear that we'll be missing patches and it'll reach out to all sorts of impure locations as a result.

I was unable to convince their build scripts to use an external LLVM. I believe their versions of these components are very similar but haven't tried sorting out differences (especially since I don't think they're based on a particular fixed release). Without more insight into the differences I'd be reluctant to use anything other than their vendor'd LLVM due to concerns about possible subtle behavioral differences from other "Swift 3.1" implementations.

That said, even breaking this huge build into pieces would be awfully nice :).

copumpkin · 2017-03-28T17:11:58Z

pkgs/development/compilers/swift/default.nix

+
+  buildPhase = ''${builder}'';
+
+  dontStrip = true;


Why not (just curious)?

I believe I set this early after looking at this bit from Arch's AUR PKGBUILD for Swift 3.02 and never revisited seeing what broke with it disabled. I'm not sure how to test whether this is still needed, although I can at least confirm whether stripping severely breaks things :).

Not a problem, was just curious. I'll probably tinker with it a bit to get a macOS version running after you push this.

copumpkin · 2017-03-28T17:12:18Z

pkgs/development/compilers/swift/default.nix

+
+  doCheck = false;
+
+  buildPhase = ''${builder}'';


''${builder}'' should be equivalent to builder

copumpkin · 2017-03-28T17:13:17Z

pkgs/development/compilers/swift/default.nix

+  installPhase = ''
+    mkdir -p $out
+
+    # Extract the generated tarball into the store


Does the build process have to generate a tarball? Feels kind of convoluted, but I guess such is life sometimes 😄

I wouldn't go so far as to say it /has/ to generate a tarball, but it's the best I could find for installing all the bits neatly. Unfortunately given the very long build time this is a bit tainted by being satisfied with what works vs trying other things. I agree it's unfortunate, for sure.

copumpkin · 2017-03-28T17:15:28Z

pkgs/development/libraries/libblocksruntime/default.nix

+  buildInputs = [ clang ];
+
+  configurePhase = ''
+    export CC=clang


I'd actually use the clangStdenv here, since that'll get you proper llvm-flavored libraries like libc++ and so on, and will be more standard I think.

Hmm. So it's a bit tricky I think, here's what I know/why things are why this way:

libblocksruntime isn't C++ (so I don't think it matters if we use libc++)

their build script checks for gcc, clang, then 'cc' but later uses the name it found to determine what CFLAGS to use. Using a clang-based stdenv doesn't actually make clang available by that name (just like linux stdenv doesn't make gcc available by that name), so clang would need to be a buildInput or this behavior patched around.

It turns out that hardening flags add -O2 which I think may make the previous point moot but I don't think that should be relied upon.

I'm not saying this is the best solution, but I'm not sure it's better to use clangStdenv. Thoughts? Incidentally I tried using clangStdenv just now and the resulting archive was bit-identical but that doesn't really mean much just a fun fact :).

dtzWill · 2017-03-28T19:52:00Z

(Thank you very much for the comments and review, there were more dark corners in this than I remembered! Working to address them now..)

dtzWill · 2017-03-28T20:16:38Z

I can squash in a bit, but thought I'd make it easier to spot the new changes for now.

Experimentally removed 'dontStrip', we'll see how this goes (go go gadget compiler!). If it doesn't break my use of Swift I'm thinking that we ship this way--if and when someone encounters breakage we can add the 'dontStrip' workaround.

Sound good?

dtzWill · 2017-03-28T20:18:36Z

Related: SourceKit apparently builds on Linux but currently requires building Swift /twice/:

apple/swift#3594 (comment)

Just an FYI so if anyone wants to add it AFAIK it shouldn't be difficult just a lot more building, at least until they do the necessary build refactoring upstream.

copumpkin · 2017-03-28T20:22:42Z

@dtzWill cool, I think the "just FYI" kind of thing is more likely to get noticed as a comment in the expression source though. Otherwise you just know someone will see the expression and try it out 😄

copumpkin · 2017-03-28T20:27:02Z

(the only thing I really care about is the .ini file at this point)

Add dependency 'libblocksruntime'.

dtzWill · 2017-03-28T22:34:33Z

Added FYI to the expression per suggestion, everything else should be addressed in the latest commit! :)

matthewbauer · 2017-03-28T23:24:22Z

pkgs/development/compilers/swift/default.nix

+    homepage = "https://github.com/apple/swift";
+    maintainers = with maintainers; [ jb55 dtzWill ];
+    license = licenses.asl20;
+    platforms = platforms.linux;


Have you tried it on macOS?

Nope! But since, for example, the current buildPhase explicitly refers to the linux build configuration I'm pretty sure it won't work :(. Hopefully it's not too much trouble to make it work on Darwin, but I really don't know.

Okay I can look into it but this PR is definitely good to merge even without macOS

copumpkin

Looks good to me!

LnL7 added the 8.has: package (new) label Jan 24, 2017

dtzWill mentioned this pull request Jan 25, 2017

WIP: Add support for building Swift packages with deps #22138

Closed

7 tasks

dtzWill changed the title ~~swift: init at DEVELOPMENT-SNAPSHOT-2017-01-23-a~~ swift: init at 3.1-2017-01-22 Jan 27, 2017

dtzWill mentioned this pull request Jan 27, 2017

Packaging Swift #11463

Closed

copumpkin reviewed Feb 4, 2017

View reviewed changes

dtzWill changed the title ~~swift: init at 3.1-2017-01-22~~ swift: init at 3.1-2017-02-09 Feb 10, 2017

dtzWill changed the title ~~swift: init at 3.1-2017-02-09~~ swift: init at 3.1-2017-02-13 Feb 14, 2017

dtzWill changed the title ~~swift: init at 3.1-2017-02-13~~ swift: init at 3.1-2017-02-20 Feb 21, 2017

dtzWill mentioned this pull request Feb 23, 2017

Package Request: swift-3.0.2 #23110

Closed

dtzWill changed the title ~~swift: init at 3.1-2017-02-20~~ swift: init at 3.1-2017-02-24 Feb 27, 2017

dtzWill changed the title ~~swift: init at 3.1-2017-02-24~~ swift: init at 3.1-2017-02-27 Feb 28, 2017

dtzWill changed the title ~~swift: init at 3.1-2017-02-27~~ swift: init at 3.1-2017-03-03 Mar 6, 2017

dtzWill changed the title ~~swift: init at 3.1-2017-03-03~~ swift: init at 3.1-2017-03-05 Mar 6, 2017

dtzWill force-pushed the feature/swift branch from a02a275 to 3516797 Compare March 14, 2017 14:08

dtzWill changed the title ~~swift: init at 3.1-2017-03-05~~ swift: init at 3.1-2017-03-13 Mar 14, 2017

dtzWill changed the title ~~swift: init at 3.1-2017-03-13~~ swift: init at 3.1 Mar 28, 2017

dtzWill force-pushed the feature/swift branch from 3516797 to 90f4450 Compare March 28, 2017 14:24

copumpkin reviewed Mar 28, 2017

View reviewed changes

dtzWill force-pushed the feature/swift branch from 3959ba5 to c1b154d Compare March 28, 2017 20:32

swift: init at 3.1

ecae31a

Add dependency 'libblocksruntime'.

dtzWill force-pushed the feature/swift branch from c1b154d to ecae31a Compare March 28, 2017 20:34

matthewbauer reviewed Mar 28, 2017

View reviewed changes

copumpkin approved these changes Mar 29, 2017

View reviewed changes

copumpkin merged commit cea6bfa into NixOS:master Mar 29, 2017

dtzWill deleted the feature/swift branch March 29, 2017 01:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

swift: init at 3.1 #22098

swift: init at 3.1 #22098

dtzWill commented Jan 24, 2017 •

edited

dtzWill commented Jan 24, 2017

dtzWill commented Jan 24, 2017

copumpkin Feb 4, 2017

dtzWill Feb 5, 2017

benley commented Feb 26, 2017

dtzWill commented Feb 28, 2017

benley commented Feb 28, 2017 •

edited

dtzWill commented Mar 6, 2017

dtzWill commented Mar 14, 2017

puffnfresh commented Mar 28, 2017

dtzWill commented Mar 28, 2017

matthewbauer commented Mar 28, 2017

copumpkin commented Mar 28, 2017

dtzWill commented Mar 28, 2017

copumpkin Mar 28, 2017

dtzWill Mar 28, 2017

copumpkin Mar 28, 2017

dtzWill Mar 28, 2017

copumpkin Mar 28, 2017

copumpkin Mar 28, 2017

copumpkin Mar 28, 2017

dtzWill Mar 28, 2017

copumpkin Mar 28, 2017

dtzWill Mar 28, 2017

dtzWill commented Mar 28, 2017

dtzWill commented Mar 28, 2017

dtzWill commented Mar 28, 2017

copumpkin commented Mar 28, 2017

copumpkin commented Mar 28, 2017

dtzWill commented Mar 28, 2017

matthewbauer Mar 28, 2017

dtzWill Mar 29, 2017

matthewbauer Mar 29, 2017

copumpkin left a comment

swift: init at 3.1 #22098

swift: init at 3.1 #22098

Conversation

dtzWill commented Jan 24, 2017 • edited

Motivation for this change

Things done

dtzWill commented Jan 24, 2017

dtzWill commented Jan 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benley commented Feb 26, 2017

dtzWill commented Feb 28, 2017

benley commented Feb 28, 2017 • edited

dtzWill commented Mar 6, 2017

dtzWill commented Mar 14, 2017

puffnfresh commented Mar 28, 2017

dtzWill commented Mar 28, 2017

matthewbauer commented Mar 28, 2017

copumpkin commented Mar 28, 2017

dtzWill commented Mar 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dtzWill commented Mar 28, 2017

dtzWill commented Mar 28, 2017

dtzWill commented Mar 28, 2017

copumpkin commented Mar 28, 2017

copumpkin commented Mar 28, 2017

dtzWill commented Mar 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

copumpkin left a comment

Choose a reason for hiding this comment

dtzWill commented Jan 24, 2017 •

edited

benley commented Feb 28, 2017 •

edited