Run test to determine if sandboxing is available #3006

matthewbauer · 2019-07-25T13:39:23Z

adds smEnabledIfAvailable option that will run a check where possible
to determine if user namespaces are supported.

edolstra

What's the exact use case for this? If it's systems that lack CLONE_NEWUSER, it's probably better to disable only CLONE_NEWUSER, rather than all sandboxing. So you could check whether clone succeeds with and without CLONE_NEWUSER.

edolstra · 2019-07-25T15:38:37Z

src/libstore/build.cc

+
+            int flags = CLONE_NEWUSER | CLONE_NEWNS | CLONE_NEWIPC | CLONE_NEWUTS | CLONE_PARENT | SIGCHLD;
+            size_t stackSize = 1;
+            char* stack = malloc(stackSize);


Probably better to use std::make_unique<unsigned char[]>(stackSize);. Leaked mallocs pollute valgrind runs.

edolstra · 2019-07-25T15:39:52Z

src/libstore/build.cc

+            if (child == -1) {
+                printError("warning: test cloning failed, sandboxing is disabled");
+                useChroot = false;
+            }


Could this whole test be done inside an std::once? Now we're forking an extra child process for every build.

edolstra · 2019-07-25T15:43:51Z

src/libstore/globals.hh

@@ -11,7 +11,7 @@

 namespace nix {

-typedef enum { smEnabled, smRelaxed, smDisabled } SandboxMode;
+typedef enum { smEnabled, smEnabledIfAvailable, smRelaxed, smDisabled } SandboxMode;


Maybe this should be a separate flag, because you may want to use relaxed mode and disable the sandbox if unavailable.

edolstra · 2019-07-25T15:44:49Z

src/libstore/build.cc

+                kill(child, SIGKILL);
+
+            if (child == -1) {
+                printError("warning: test cloning failed, sandboxing is disabled");


There is a warn function since recently.

matthewbauer · 2019-07-25T16:25:35Z

What's the exact use case for this? If it's systems that lack CLONE_NEWUSER, it's probably better to disable only CLONE_NEWUSER, rather than all sandboxing. So you could check whether clone succeeds with and without CLONE_NEWUSER.

That sounds like a better idea, if it works right. As a plus that would avoid having an extra "sandbox" setting as sandboxing would still be enabled (just without CLONE_NEWUSER). There are at least 3 patches for this in the wild that I can find:

Debian & more: https://salsa.debian.org/kernel-team/linux/raw/master/debian/patches/debian/add-sysctl-to-disallow-unprivileged-CLONE_NEWUSER-by-default.patch
Alt Linux & more: https://lore.kernel.org/patchwork/patch/640897/
RedHat / CentOS: https://lore.kernel.org/patchwork/patch/700143/

~~It looks like they all only effect CLONE_NEWUSER.~~ Some systems also disable CLONE_NEWNS.

Some kernels disable "unpriveleged user namespaces". This is unfortunate, but we can still use mount namespaces. Anyway, since each builder has its own nixbld user, we already have most of the benefits of user namespaces.

When sandbox-fallback = true (the default), the Nix builder will fall back to disabled sandbox mode when the kernel doesn’t allow users to set it up. This prevents hard errors from occuring in tricky places, especially the initial installer. To restore the previous behavior, users can set: sandbox-fallback = false in their /etc/nix/nix.conf configuration.

edolstra · 2019-07-30T09:40:25Z

Looks good but I would be more comfortable with sandbox-fallback defaulting to false. The special handling for missing user namespaces is more important anyway.

matthewbauer · 2019-07-30T20:34:45Z

Looks good but I would be more comfortable with sandbox-fallback defaulting to false. The special handling for missing user namespaces is more important anyway.

My main concern with sandbox-fallback = false is that we continue to have a lot of users not able to install Nix because the default sandbox doesn't work on their computers. See #2662, #2636, #2632, #2318

startProcess does not appear to send the exit code to the helper correctly. Not sure why this is, but it is probably safe to just fallback on all sandbox errors.

grahamc · 2019-07-30T22:01:53Z

I also reported #3000 which is along the same lines. I'll run this PR through my nix install matrix.

matthewbauer · 2019-07-30T22:08:29Z

This looks like it should be handled (WSL2), but I'm not sure how:

error: while setting up the build environment: mounting /proc: Operation not permitted
./install: unable to install Nix into your default profile

grahamc · 2019-07-30T22:23:08Z

Note that #3000 actually isn't related to WSL.

grahamc · 2019-07-31T13:45:28Z

2.2.2's default to enable the sandbox broke installation for:

centos6
centos7
debian8
debian9
gentoo
ubuntu12.04

each of these failed with one of two messages:

centos-6-install    installing 'nix-2.2.2'
centos-6-install    error: cloning builder process: Invalid argument
centos-6-install    error: unable to start build process
centos-6-install    /tmp/nix-binary-tarball-unpack.0e2oQ1HL9U/unpack/nix-2.2.2-x86_64-linux/install: unable to install Nix into your default profile

or:

debian-8-install    installing 'nix-2.2.2'
debian-8-install    error: cloning builder process: Operation not permitted
debian-8-install    error: unable to start build process
debian-8-install    /tmp/nix-binary-tarball-unpack.N6AiK3XbCd/unpack/nix-2.2.2-x86_64-linux/install: unable to install Nix into your default profile

After this PR, neither of these errors come up, but all six now fail with another error:

centos-7-install    installing 'nix-2.3pre6828_9a0855b'
centos-7-install    warning: Nix search path entry '/home/vagrant/.nix-defexpr/channels' does not exist, ignoring
centos-7-install    error: while setting up the build environment: changing into '/tmp/nix-build-user-environment.drv-0': No such file or directory
centos-7-install    ./nix/install: unable to install Nix into your default profile

Here are the install matrix's reports:

grahamc · 2019-08-27T12:18:41Z

The new commit fixes many (all?) of the installations https://buildkite.com/organizations/grahamc/pipelines/nix-install-matrix/builds/61/jobs/ae342503-d31f-43e6-8434-a9136b377b11/artifacts/5cdb7828-2262-44bd-bc4f-4da5cfe8da2d (built at
5c06a8d )

matthewbauer force-pushed the test-sandboxing branch from a67b488 to fff1b7e Compare July 25, 2019 15:06

edolstra reviewed Jul 25, 2019

View reviewed changes

matthewbauer force-pushed the test-sandboxing branch from fff1b7e to 679273c Compare July 25, 2019 18:40

matthewbauer added 2 commits July 25, 2019 14:42

Disable CLONE_NEWUSER when it’s unavailable

d171090

Some kernels disable "unpriveleged user namespaces". This is unfortunate, but we can still use mount namespaces. Anyway, since each builder has its own nixbld user, we already have most of the benefits of user namespaces.

matthewbauer force-pushed the test-sandboxing branch from 679273c to 11d8534 Compare July 25, 2019 18:44

Don’t rely on EPERM

9a0855b

startProcess does not appear to send the exit code to the helper correctly. Not sure why this is, but it is probably safe to just fallback on all sandbox errors.

Reset tmpDirInSandbox for unsandboxed

5c06a8d

grahamc added this to the Nix 2.3 milestone Aug 27, 2019

edolstra merged commit 5c06a8d into NixOS:master Aug 27, 2019

edolstra mentioned this pull request Aug 28, 2019

Error: cloning builder process: Operation not permitted installing Nix 2.2.1 in (Arch) Linux #2633

Closed

matthewbauer mentioned this pull request Oct 13, 2019

NIX_BUILD_TOP not set correctly outside of nixos #3142

Open

flokli mentioned this pull request Apr 3, 2020

Improvements to the NixOS Hardened Profile NixOS/nixpkgs#73763

Merged

9 tasks

matthewbauer mentioned this pull request Jun 8, 2020

Issue installing nix on debian 9 (with solution) NixOS/nixpkgs#61271

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run test to determine if sandboxing is available #3006

Run test to determine if sandboxing is available #3006

matthewbauer commented Jul 25, 2019

edolstra left a comment

edolstra Jul 25, 2019

edolstra Jul 25, 2019

edolstra Jul 25, 2019

edolstra Jul 25, 2019

matthewbauer commented Jul 25, 2019 •

edited

edolstra commented Jul 30, 2019

matthewbauer commented Jul 30, 2019 •

edited

grahamc commented Jul 30, 2019

matthewbauer commented Jul 30, 2019 •

edited

grahamc commented Jul 30, 2019

grahamc commented Jul 31, 2019

grahamc commented Aug 27, 2019 •

edited

Run test to determine if sandboxing is available #3006

Run test to determine if sandboxing is available #3006

Conversation

matthewbauer commented Jul 25, 2019

edolstra left a comment

Choose a reason for hiding this comment

edolstra Jul 25, 2019

Choose a reason for hiding this comment

edolstra Jul 25, 2019

Choose a reason for hiding this comment

edolstra Jul 25, 2019

Choose a reason for hiding this comment

edolstra Jul 25, 2019

Choose a reason for hiding this comment

matthewbauer commented Jul 25, 2019 • edited

edolstra commented Jul 30, 2019

matthewbauer commented Jul 30, 2019 • edited

grahamc commented Jul 30, 2019

matthewbauer commented Jul 30, 2019 • edited

grahamc commented Jul 30, 2019

grahamc commented Jul 31, 2019

grahamc commented Aug 27, 2019 • edited

matthewbauer commented Jul 25, 2019 •

edited

matthewbauer commented Jul 30, 2019 •

edited

matthewbauer commented Jul 30, 2019 •

edited

grahamc commented Aug 27, 2019 •

edited