Init ldc at 1.3.0, fix bootstrap dmd build, fix dtools test, run dmd-testsuite in checkPhase and fix Foundation framework #28635

ThomasMader · 2017-08-28T11:38:53Z

Motivation for this change

I wanted to add the D programming language compiler ldc to nixpkgs.

Things done

DMD wasn't building on Linux anymore because the default compiler was switched from gcc5 to gcc6 and the bootstrap compiler doesn't seem to build with that version anymore.
Fixed by using gcc5 for the bootstrap compiler only on Linux.
dtools rdmd test wasn't successful anymore because std.stdiobase isn't available anymore.
Added ldc package which fulfills it's testsuite (except cppa.d test of dmd-testsuite because it doesn't work for now)
Added dmd-testsuite run to dmd compiler as an additional test. (except cppa.d test because it doesn't work for now)
Fixed Foundation framework in apple_sdk. (see also fatal error: 'CoreFoundation/CFAttributedString.h' file not found #13778)
The dmd testsuite uses this on OSX and it complained about not finding the header file:

... runnable/testmodule.d ()
runnable/extra-files/objc_objc_msgSend.m:1:9: fatal error: 'Foundation/Foundation.h' file not found
#import <Foundation/Foundation.h>
^~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
failed to execute 'clang -m64 -c runnable/extra-files/objc_objc_msgSend.m -o test_results/runnable/objc_objc_msgSend.m.o'
make[1]: *** [Makefile:152: test_results/runnable/objc_objc_msgSend.d.out] Error 1

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Remove cppa test for now because it isn't working.

mention-bot · 2017-08-28T11:39:03Z

@ThomasMader, thanks for your PR! By analyzing the history of the files in this pull request, we identified @nckx, @pikajude and @vlstill to be potential reviewers.

Mic92 · 2017-08-28T11:52:41Z

pkgs/development/compilers/ldc/default.nix

+    description = "The LLVM-based D compiler";
+    homepage = https://github.com/ldc-developers/ldc;
+    # from https://github.com/ldc-developers/ldc/blob/master/LICENSE
+    license = with licenses; [ bsd3 boost mit ncsa gpl2Plus ];


Do you want to maintain it?

I thought about that.
Not sure what it means in reality though.
In fact I plan to keep all D related development stuff going as time permits.
I don't like it either that there is no maintainer for all D related stuff.
So i guess the difference is just a formality right? (I know about adding my name into maintainers list, I already did it in a PR a few year back which was never merged fully)

You might be consulted in pull requests as a reviewer and you get emails from hydra, when the build of this compiler fails. How much time you want to spent is up to you. Being a package maintainer in nixpkgs has usually less implications compared to other linux/package distributions.

Sounds nice.
I added myself to the maintainers. Thanks.

Mic92 · 2017-08-28T11:55:02Z

pkgs/development/compilers/ldc/default.nix

+    cmake \
+    -DCMAKE_INSTALL_PREFIX=$out \
+    -DINCLUDE_INSTALL_DIR=$out/include/dlang/ldc \
+    ..


This should not be necessary. Try to add cmake to buildInputs and set cmakeFlagsArray=("-DINCLUDE_INSTALL_DIR=$out/include/dlang/ldc") in preConfigure instead.

Thanks, done.

Mic92 · 2017-08-28T11:57:09Z

pkgs/development/compilers/ldc/default.nix

+
+  installPhase = ''
+    make install
+  '';


this is the default of installPhase

Thanks, done.

ThomasMader · 2017-08-28T13:37:01Z

@Mic92 Do you happen to know what to do about the timeout on Travis?

The checkPhases happen to be very long now but they are producing regular output to stdout and I haven't witnessed a stall for multiple minutes on my machines.

matthewbauer · 2017-08-28T19:11:15Z

pkgs/os-specific/darwin/apple-sdk/frameworks.nix

@@ -46,7 +46,7 @@ with frameworks; with libs; {
  ExceptionHandling       = [];
  FWAUserLib              = [];
  ForceFeedback           = [ CF IOKit ];
-  Foundation              = [ CF libobjc Security ApplicationServices SystemConfiguration ];
+  Foundation              = [ cf-private CF libobjc Security ApplicationServices SystemConfiguration ];


It's been a while since I looked at this but I'm a little concerned at what this does. This may end up working, but I think it's a little hacky. "ApplicationServices" will already add cf-private as a framework and I think the reason this works is because clang will just use the first "CF" it finds (in this case cf-private). I think we need to look at just including cf-private in CF as that distinction doesn't really exist outside Nixpkgs.

/cc @copumpkin

If you're going to add it, please comment why it was needed. In a slightly more ideal world, cf-private wouldn't really exist at all, so I want to keep track of what requires it and why (even just as a link to the other ticket).

I added a comment to link to this PR because in here it explains why it is needed for ldc and a further reference to the other bug exists anyway.

Mic92 · 2017-08-28T19:58:23Z

@ThomasMader yes, for big builds, travis is sometimes useless. There were actually plans to let hydra take over. Just ignore the build failure, we usually build manually before merging in this case.

…oesn't include std.stdiobase.d anymore.

…ecause CoreFoundation/CFAttributedString.h could not be found. The problem appeared when building ldc on Mac OSX. See also 13778.

ThomasMader · 2017-09-02T06:44:29Z

Ok from my side everything is finished and ready to be merged.

This bug also fixes the build of dmd referenced in #28643 and a further build problem with dtools not yet detected because dmd fails to build on master.

trishume · 2017-09-10T00:11:42Z

I'm looking forward to this fix. Might also be worth looking into using the dmd-cxx branch that they are maintaining a little bit. This thread mentions it in the context of the new GCC build issue: http://forum.dlang.org/post/mailman.4916.1500319853.31550.dlang-internal@puremagic.com

ThomasMader · 2017-09-10T12:54:31Z

Might also be worth looking into using the dmd-cxx branch that they are maintaining a little bit. This thread mentions it in the context of the new GCC build issue: http://forum.dlang.org/post/mailman.4916.1500319853.31550.dlang-internal@puremagic.com

Thanks, thats very valuable information.
I will try to fix it this way in a future PR. I also always wondered how they used 2.068.2 as the bootstrap version because it contains idgen in D. That information is in that thread too.

Mic92 · 2017-09-15T09:32:13Z

Thanks! I will also remove broken = true from dmd.

ThomasMader · 2017-09-15T20:07:37Z

@Mic92 Thank you for merging.

I saw that Hydra still isn't able to build the bootstrap dmd compiler. (https://hydra.nixos.org/build/61091111)
I looked into it and the problem is the getpwnam_r C lib call (https://linux.die.net/man/3/getpwnam_r) in https://github.com/dlang/phobos/blob/v2.067.1/std/path.d#L2991 .
This function returns the proper home directory for a given user but on Hydra this directory for user root is not "/root" but "/build" which is not standard.
The problem is that "/root" is expected in the unittest of dmd/phobos and so the build fails.

The problem can be reproduced on other machines if the home directory for user root is changed to something different. e.g.: usermod -m -d /build root

Now the question is if it is possible to fix the home directory for the root user on Hydra. I think this would be the cleaner configuration as "/root" is the default on Linux systems.

Mic92 · 2017-09-15T21:07:59Z

I think /etc/passwd in the nix sandbox just contains the build user and not the root user. Can just skip this test? If the root cause is known, this is ok. We also do it for some tests in go.

ThomasMader · 2017-09-15T21:26:52Z

I highly doubt that because getpwnam_r would fail for a non existing username and the assert output would be "~root". (See https://github.com/dlang/phobos/blob/v2.067.1/std/path.d#L3086)
The build on Darwin works too but that has a different configuration for sure.
(https://hydra.nixos.org/build/61092059)

I also have nix.useSandbox = true in my /etc/nixos/configuration.nix and that should enable the sandboxing if I build with nix-build no?

Mic92 · 2017-09-15T21:30:59Z

Hydra runs nixUnstable, which does provide additional sandboxing.

ThomasMader added 2 commits August 27, 2017 19:03

dmd: Fix bootstrap dmd compiler build. Doesn't work with gcc6.

e17fdb9

dmd: Run dmd-testsuite in checkPhase

5a35b07

Remove cppa test for now because it isn't working.

Mic92 reviewed Aug 28, 2017

View reviewed changes

ThomasMader force-pushed the ldc branch 2 times, most recently from 4c9901e to 4816b1a Compare August 28, 2017 14:29

ThomasMader changed the title ~~Init ldc at 1.3.0, fix bootstrap dmd build, run dmd-testsuite in checkPhase and fix Foundation framework~~ Init ldc at 1.3.0, fix bootstrap dmd build, fix dtools test, run dmd-testsuite in checkPhase and fix Foundation framework Aug 28, 2017

matthewbauer reviewed Aug 28, 2017

View reviewed changes

ThomasMader added 3 commits September 2, 2017 08:30

ldc: Init at 1.3.0

8a2a3da

dtools: Fix rdmd test when building with newer phobos version which d…

930b01a

…oesn't include std.stdiobase.d anymore.

apple-sdk: Move cf-private first in list, otherwise the build fails b…

2a0f1ec

…ecause CoreFoundation/CFAttributedString.h could not be found. The problem appeared when building ldc on Mac OSX. See also 13778.

ThomasMader force-pushed the ldc branch from 564bd2e to 2a0f1ec Compare September 2, 2017 06:36

ThomasMader mentioned this pull request Sep 2, 2017

dub: Updating (1.1.0 -> 1.5.0) and improving #28396

Merged

8 tasks

ThomasMader mentioned this pull request Sep 14, 2017

Zero Hydra Failures for 17.09 #28643

Closed

Mic92 merged commit 4198692 into NixOS:master Sep 15, 2017

ThomasMader deleted the ldc branch September 15, 2017 17:21

This was referenced Sep 15, 2017

Home directory for user root on Hydra/Linux seems to be /build #29443

Closed

dmd & ldc: Fix bootstrap dmd to build with sandboxing of nixUnstable and fix dmd to build on Darwin; Disable check phases of dmd and ldc because of sandboxing problem #29469

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Init ldc at 1.3.0, fix bootstrap dmd build, fix dtools test, run dmd-testsuite in checkPhase and fix Foundation framework #28635

Init ldc at 1.3.0, fix bootstrap dmd build, fix dtools test, run dmd-testsuite in checkPhase and fix Foundation framework #28635

ThomasMader commented Aug 28, 2017 •

edited

mention-bot commented Aug 28, 2017

Mic92 Aug 28, 2017

ThomasMader Aug 28, 2017

Mic92 Aug 28, 2017 •

edited

ThomasMader Aug 28, 2017

Mic92 Aug 28, 2017

ThomasMader Aug 28, 2017

Mic92 Aug 28, 2017

ThomasMader Aug 28, 2017

ThomasMader commented Aug 28, 2017

matthewbauer Aug 28, 2017

copumpkin Aug 28, 2017 •

edited

ThomasMader Sep 2, 2017

Mic92 commented Aug 28, 2017

ThomasMader commented Sep 2, 2017

trishume commented Sep 10, 2017

ThomasMader commented Sep 10, 2017

Mic92 commented Sep 15, 2017

ThomasMader commented Sep 15, 2017 •

edited

Mic92 commented Sep 15, 2017 •

edited

ThomasMader commented Sep 15, 2017 •

edited

Mic92 commented Sep 15, 2017 •

edited

Init ldc at 1.3.0, fix bootstrap dmd build, fix dtools test, run dmd-testsuite in checkPhase and fix Foundation framework #28635

Init ldc at 1.3.0, fix bootstrap dmd build, fix dtools test, run dmd-testsuite in checkPhase and fix Foundation framework #28635

Conversation

ThomasMader commented Aug 28, 2017 • edited

Motivation for this change

Things done

mention-bot commented Aug 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 Aug 28, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThomasMader commented Aug 28, 2017

Choose a reason for hiding this comment

copumpkin Aug 28, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Aug 28, 2017

ThomasMader commented Sep 2, 2017

trishume commented Sep 10, 2017

ThomasMader commented Sep 10, 2017

Mic92 commented Sep 15, 2017

ThomasMader commented Sep 15, 2017 • edited

Mic92 commented Sep 15, 2017 • edited

ThomasMader commented Sep 15, 2017 • edited

Mic92 commented Sep 15, 2017 • edited

ThomasMader commented Aug 28, 2017 •

edited

Mic92 Aug 28, 2017 •

edited

copumpkin Aug 28, 2017 •

edited

ThomasMader commented Sep 15, 2017 •

edited

Mic92 commented Sep 15, 2017 •

edited

ThomasMader commented Sep 15, 2017 •

edited

Mic92 commented Sep 15, 2017 •

edited