Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dmd & ldc: Fix bootstrap dmd to build with sandboxing of nixUnstable and fix dmd to build on Darwin; Disable check phases of dmd and ldc because of sandboxing problem #29469

Merged
merged 4 commits into from Sep 21, 2017

Conversation

ThomasMader
Copy link
Contributor

@ThomasMader ThomasMader commented Sep 16, 2017

Motivation for this change

Fixing the problems which appeared because of the merge of #28635.
The sandboxing problem is also discussed in #29443.

Disabled check phases of dmd and ldc to get them to build with the sandboxing of Nix 1.12.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • Linux
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

CC @Mic92

@vcunat
Copy link
Member

vcunat commented Sep 16, 2017

It might be a local problem, but I repeatedly get:

/nix/store/wn6slnv6m5glx8lm6gkpjr1h3bxrrs67-dmd-2.067.1.drv

[...]
 --- std.socket(521) test fails depending on environment ---
 (core.exception.AssertError@std/socket.d(523): Assertion failure
----------------
generated/linux/release/64/unittest/libphobos2-ut.so(+0x3bce724) [0x7ffff695a724]
generated/linux/release/64/unittest/libphobos2-ut.so(nothrow @safe void std.socket.__unittestL518_3().__lambda1()+0x3d) [0x7ffff6929375]
generated/linux/release/64/unittest/libphobos2-ut.so(@trusted void std.socket.softUnittest(@safe void delegate(), int)+0x23) [0x7ffff6928d5b]
generated/linux/release/64/unittest/libphobos2-ut.so(@safe void std.socket.__unittestL518_3()+0x18) [0x7ffff6929330]
generated/linux/release/64/unittest/libphobos2-ut.so(void std.socket.__modtest()+0x9) [0x7ffff695a621]
generated/linux/release/64/unittest/test_runner() [0x40141c]
generated/linux/release/64/unittest/libphobos2-ut.so(runModuleUnitTests+0xe1) [0x7ffff74322a9]
generated/linux/release/64/unittest/libphobos2-ut.so(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x17) [0x7ffff7454b17]
generated/linux/release/64/unittest/libphobos2-ut.so(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2d) [0x7ffff7454acd]
generated/linux/release/64/unittest/libphobos2-ut.so(_d_run_main+0x1ec) [0x7ffff7454a44]
generated/linux/release/64/unittest/test_runner() [0x40160c]
/nix/store/q0prbrw9n7h2sjqmla8ac32lfyq0w91z-glibc-2.25-49/lib/libc.so.6(__libc_start_main+0xf0) [0x7ffff22d4560])
No service for epmap.
****** FAIL release64 std.socket
core.exception.RangeError@std/socket.d(865): Range violation
----------------
generated/linux/release/64/unittest/libphobos2-ut.so(+0x3bce6f4) [0x7ffff695a6f4]
generated/linux/release/64/unittest/libphobos2-ut.so(@safe void std.socket.__unittestL860_5()+0x59) [0x7ffff6929bc9]
generated/linux/release/64/unittest/libphobos2-ut.so(void std.socket.__modtest()+0x13) [0x7ffff695a62b]
generated/linux/release/64/unittest/test_runner() [0x40141c]
generated/linux/release/64/unittest/libphobos2-ut.so(runModuleUnitTests+0xe1) [0x7ffff74322a9]
generated/linux/release/64/unittest/libphobos2-ut.so(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x17) [0x7ffff7454b17]
generated/linux/release/64/unittest/libphobos2-ut.so(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2d) [0x7ffff7454acd]
generated/linux/release/64/unittest/libphobos2-ut.so(_d_run_main+0x1ec) [0x7ffff7454a44]
generated/linux/release/64/unittest/test_runner() [0x40160c]
/nix/store/q0prbrw9n7h2sjqmla8ac32lfyq0w91z-glibc-2.25-49/lib/libc.so.6(__libc_start_main+0xf0) [0x7ffff22d4560]
make: *** [posix.mak:376: unittest/std/socket.d] Error 1

but it seems the same before this PR as well...

@ThomasMader
Copy link
Contributor Author

ThomasMader commented Sep 16, 2017

Haven't seen that one before.
The range violation error is the problem which is terminating the build.
Do you happen to have some special network settings on that machine? Can you elaborate about your environment and how you start the build with which versions and so on.
Maybe I am able to reproduce the problem.

My wild guess is that the call to gethostbyaddr (https://github.com/dlang/phobos/blob/v2.067.1/std/socket.d#L839) is failing.
It seems the test is trying to resolve the hostname for 127.0.0.1 but fails.
What is the output of this python code:

import socket
socket.gethostbyaddr('127.0.0.1')

@vcunat
Copy link
Member

vcunat commented Sep 16, 2017

I have NixOS with sandboxing on, version: roughly master/staging from two weeks ago. There's no unusual network AFAIK, but sandboxing is supposed to block/isolate networking anyway. Sandboxing is causing this failure, probably.

@vcunat
Copy link
Member

vcunat commented Sep 16, 2017

I forgot to write I'm using nixUnstable (1.12pre5511_c94f3d55 ATM).

@ThomasMader
Copy link
Contributor Author

I don't see how I install nixUnstable the most comfortable way.
I cloned the Nix repository and currently building from source but I doubt that this is the way to go.

@vcunat
Copy link
Member

vcunat commented Sep 16, 2017

You don't need to. It's in nixpkgs under attribute name nixUnstable. You can try it on NixOS via nix.package = pkgs.nixUnstable;

@ThomasMader
Copy link
Contributor Author

Ok didn't know that it's just a name inside nixpkgs.
So I was able to create a nix-shell with nix-shell -p nixUnstable executed from inside the nixpkgs directory.
Thought that's more complicated.

This way I was able to reproduce the problem.

@ThomasMader
Copy link
Contributor Author

I was right in #29469 (comment) gethostbyaddr returns null.

I decided to just disable the check phase for now. Everything is building with sandboxing of Nix 1.12 including ldc.

I came to the conclusion that the check phase isn't the right place to test the packages because of the minimal environment given by the sandbox.
Is it really needed to restrict the network in sandboxing in such a way?
Wouldn't it make sense to shutdown sandboxing for the check phase?
In the meantime it seems to me, that it would make more sense to add tests to nixos/tests for those packages but I don't have any clue on how that is working.

@ThomasMader ThomasMader changed the title dmd: Fix bootstrap dmd to build with sandboxing of nixUnstable and fix dmd to build on Darwin dmd & ldc: Fix bootstrap dmd to build with sandboxing of nixUnstable and fix dmd to build on Darwin; Disable check phases of dmd and ldc because of sandboxing problem Sep 16, 2017
@vcunat
Copy link
Member

vcunat commented Sep 17, 2017

The main intention of network sandboxing is to avoid impurities like trying to download additional stuff during the build. Working with "localhost" is not a problem theoretically, due to each build living in its own network namespace, but apparently something is currently too different in there.

BTW, nix sandboxing has no idea about phases – it just invokes the builder which can do anything. It's a matter of nixpkgs that we typically use bash with the stdenv scripts that do the phases.

gethostbyaddr: I'm afraid it's not clear to me from the documentation whether the function is supposed to try getting some name for the address. That might as simple as missing /etc/hosts with 127.0.0.1 localhost \n ::1 localhost.

@ThomasMader
Copy link
Contributor Author

BTW, nix sandboxing has no idea about phases – it just invokes the builder which can do anything. It's a matter of nixpkgs that we typically use bash with the stdenv scripts that do the phases.

gethostbyaddr: I'm afraid it's not clear to me from the documentation whether the function is supposed to try getting some name for the address. That might as simple as missing /etc/hosts with 127.0.0.1 localhost \n ::1 localhost.

The socket unittest of phobos is doing more than just this localhost test. That's also the reason I saw no point in trying to dig deeper for now because it clearly shows that check phase isn't what I need it to be.
I could be wrong but right now it looks as if the check phase should be run in it's own sandbox. Preventing it from doing any changes on the build or anything else. It only has permissions to modify state inside a check phase directory/chroot and can't break out of it. But it has read permissions on the build. Could also be a copy inside the new chroot environment. The sandbox environment should provide a standard environment which does not restrict at all to be able to run all possible tests.
Has somebody already had this idea or is there some problem I don't see right now?

@vcunat
Copy link
Member

vcunat commented Sep 17, 2017

I think there were some similar ideas at least, but nothing really got implemented. One could also simulate that simply by splitting into multiple derivations – first would do the build and output the intermediate files, then you could run the checks in another one. (That would also have the advantage of not having to rebuild if you just want to tweak something in the check phase.)

@ThomasMader
Copy link
Contributor Author

And why is one derivation build with another sandbox in this scenario? Or is it configurable per derivation input somehow?

It would also be very comfortable if it would be possible to disable the check phases when running a build via nix-build if one just needs a quick result.

@vcunat
Copy link
Member

vcunat commented Sep 17, 2017

Fixed-output derivations have full access to the network. The "testing derivation" may e.g. just output a single empty file and nothing else (or fail).

@ThomasMader
Copy link
Contributor Author

Ah that's already available that's cool thanks. Can you give me an example of such an fixed output derivation?
I will try to rebuild my packages this way in a future PR.

@vcunat
Copy link
Member

vcunat commented Sep 17, 2017

See the manual. The primary intended use is for downloaders, e.g. fetchurl, but nix does most things in a general way...

@ThomasMader
Copy link
Contributor Author

ThomasMader commented Sep 17, 2017

Ok since the CI infrastructure is days behind it makes no sense in waiting on it especially because the build is currently in a broken state anyway and can only get better with this PR.
This PR is ready to be merged.
It also needs to be cherry picked into the 17.09 release.

@joachifm joachifm merged commit ae4f2fb into NixOS:master Sep 21, 2017
@joachifm joachifm added this to the 17.09 milestone Sep 21, 2017
@ThomasMader ThomasMader deleted the fix-dmd branch September 22, 2017 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants