build-fhs-userenv: change to using bubblewrap over chrootenv #55973

illegalprime · 2019-02-18T01:31:30Z

Motivation for this change

this makes FHS envs use bubblewrap over Nix's custom chrootenv.c which might fix issues such as #33106 but more generally gives us the ability to add many new configuration options with bwrap's added power. Maybe using bwrap will be more stable as well since it seems widely used.

I have tested this by building Yocto images by running bitbake in an FHSUserEnv but I have yet to run steam which is likely the most complicated package that utilizes this.

This comes with fixes from a lot of experience using FHS env for Yocto and attempting to run it in various environments. Namely:

/host/etc is now /host-etc because systemd symlinks /etc/resolv.conf to ../run/systemd/resolve/resolv.conf so keeping /host-etc on the same level as /etc helps with that.
/etc/pki is now included so fedora users can get SSL within the env
Removing the mount of /host-etc (/host/etc) and directly bind-mounting all etc files and dirs, then allowing directories of etc from the generated rootfs to override the one from the host.

Applications that work:

Things done

pkgs/build-support/build-fhs-userenv/default.nix

illegalprime · 2019-02-19T15:40:21Z

The build error is from this line:
https://github.com/NixOS/nixpkgs/pull/55973/files#diff-0e0af17bff7614ce6822d64c6863bb1fR36
but I'm not sure what the fix is, it seems that I can't readDir "${fhs-env}" during build because that path doesn't exist yet? it's weird because this works on my test fhs-env. Do i do it at runtime or can i force this directory to exist?

hedning · 2019-02-19T16:32:53Z

ofborg only evaluates things and it's not able to evaluate the buildFHSUserEnv without building due to readDir. The easiest fix might be calculate mounts, ro_mounts and blacklist in bwrap_cmd instead.

illegalprime · 2019-02-20T15:26:27Z

did some more tests, most things didn't run because my host system's /etc/fonts/fonts.conf could not be read by these packages for some reason. Nothing that seems to be wrong with the FHS env itself.
also updated the docs.

matthewbauer · 2019-02-21T01:45:04Z

@GrahamcOfBorg build houdini lightworks conda

illegalprime · 2019-02-26T15:54:20Z

@hedning what do I need to merge this?

hedning · 2019-02-26T16:55:54Z

It would be good to get an OK from @yegortimoshenko or @abbradar first.

hedning · 2019-02-26T16:56:07Z

doc/functions/fhs-environments.xml

+    </term>
+    <listitem>
+      <para>
+        Any extra arguments tp pass to bubblewrap (which is now the underlying


typo: tp -> to

lukateras

I would prefer if bwrap replacement were to replace chrootenv without extending API.

lukateras · 2019-02-26T17:52:00Z

doc/functions/fhs-environments.xml

@@ -50,6 +50,58 @@
    </para>
   </listitem>
  </varlistentry>
+  <varlistentry>
+    <term>
+      <literal>automount</literal>


I don't think this should be an option, not for buildFHSUserEnv, that is: it should really be the only behavior, because isolation is not the goal (FHS compat is).

Later, we can get to sandboxing of individual derivations, XDG portals, etc. That said, it will certainly require a complete redesign of buildFHSUserEnv and will have to be carefully thought out.

lukateras · 2019-02-26T17:56:25Z

doc/functions/fhs-environments.xml

+    </term>
+    <listitem>
+      <para>
+        Set to <literal>"$(pwd)"</literal> by default, this option determines


This seems to be orthogonal to FHS emulation (and can be/is implemented in wrappers).

illegalprime · 2019-02-28T15:52:47Z

@yegortimoshenko if I wanted to edit the bwrap options for my own uses, how should I implement this to make that possible? Maybe I should include it as an arg in the derivation and use overrideAttrs? Or maybe I should just make my own custom FHSUserEnv, but I want to re-use as much upstream code as I can.

lukateras · 2019-05-28T09:13:49Z

It's just that I worry that we might need to switch back at some point, if we ever need anything bubblewrap doesn't implement or if it proves to be suboptimal in some regard. If we were to expose bubblewrap options, we will never be able to change underlying implementation without breaking reverse compat.

abbradar · 2019-05-28T09:18:24Z

Yeah, that's what I fear too so my current idea is to proceed with caution and 1) don't expose the implementation 2) look at bubblewrap's code to see if it's hackable enough.

…

On May 28, 2019 12:13:58 PM GMT+03:00, Yegor Timoshenko ***@***.***> wrote: It's just that I worry that we might need to switch back at some point, if we ever need anything bubblewrap doesn't implement. If we were to expose bubblewrap options, we will never be able to change underlying implementation without breaking reverse compat. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #55973 (comment)

-- Nikolay.

abbradar · 2019-05-30T13:12:23Z

I didn't yet review the PR itself yet, only bubblewrap -- it's quite more complicated but we should definitely try using it; the code is readable too.

One concern of mine is if nested bubblewrap would work okay, especially if it would blow /proc/mounts. This needs to be tested.

Another thing -- we explicitly don't want any kind of isolation that bubblewrap does, including no automount, any other namespaces etc, and we also don't want to expose it as an implementation. I think we could hit both birds by adding buildBwrapUserEnv upon which old buildFHSUserEnv resides. The former would have all the bubblewrap options exposed and the latter would enforce the old behaviour.

lukateras · 2019-05-30T17:19:55Z

Wrt naming: User seems to be redundant, maybe buildBwrapEnv? Sounds good otherwise.

abbradar · 2019-05-30T17:38:13Z

Maybe; don't have big preferences here. @illegalprime can choose the naming ~_^

vcunat · 2019-08-18T11:31:18Z

For reference, the relocation error messages from above are claimed to be gone since glibc 2.30, which has a WIP PR.

illegalprime · 2019-09-15T11:07:03Z

I'm back!
@yegortimoshenko I updated this branch to not introduce any new options and I also changed /host/etc to /host-etc because /etc/resolv.conf is symlinked to ../run/ in systemd. Fedora also stores its SSL certificates as a symlink to somewhere not included in the current etc package so that should be investigated too.

illegalprime · 2019-09-15T11:33:02Z

OK Fedora stores its SSL certs in /etc/pki so I've added that to the environment.

illegalprime · 2019-09-15T13:34:28Z

I added a commit which removes /host/etc (or /host-etc) and bind-mounts specific files or directories from the host's /etc to the chroot. It then will look inside the generated root filesystem's /etc and bind mount subdirectories of it. because this is all subdirectories of etc the generated etc and the host etc are "merged" somewhat at runtime.

This helps avoid weird symlink bugs when nesting chroots like in this commit 06f27dc and in general simplifies things (IMO)

zimbatm · 2020-02-05T11:07:16Z

👍 really cool work. What can I do to help finish this PR?

bachp · 2020-04-14T20:37:26Z

@illegalprime Are you still working on this. Looks very promising

Atemu · 2020-07-28T12:34:46Z

Rebased on top of nixos-unstable here: https://github.com/Atemu/nixpkgs/tree/chrootenv2bubblewrap

Mic92 · 2020-07-31T09:31:36Z

pkgs/build-support/build-fhs-userenv/default.nix

+      --unshare-all \
+      --share-net \
+      --die-with-parent \
+      --ro-bind /nix /nix \


Would this break single user nix?

raboof · 2020-07-31T10:53:03Z

Nice!

I like removing our dependency on that small custom C program
needs a simple rebase (did it in https://github.com/raboof/nixpkgs/pull/new/chrootenv-2-bwrap)
tested, verified it indeed works for my use cases (building things with sbt-protoc, building noisemaker)

(I don't really feel qualified to review the rest so this is just a comment ;) )

Atemu · 2020-07-31T11:14:01Z

Since the compatibility is not 100% yet, perhaps we should limit bubblewrap use to the programs which are known to work first and move over the rest in later PRs.

Making all of these work feels like too large of a scope to me.

Mic92 · 2020-07-31T12:03:12Z

We could call it v2 and migrate everything to it and at some point delete the old version.

Atemu · 2020-07-31T13:26:57Z

I did some rebase magic to do just that in a transparent manner https://github.com/Atemu/nixpkgs/tree/buildFHSUserEnvBw
I also ported three of the FHS packages that I know work fine.

Should I open a new PR?

Once we've ported all packages over, we can simply rename the function to the old name. This name should be temporary.

Mic92 · 2020-08-01T08:06:31Z

I did some rebase magic to do just that in a transparent manner Atemu/nixpkgs@buildFHSUserEnvBw
I also ported three of the FHS packages that I know work fine.

Should I open a new PR?

Once we've ported all packages over, we can simply rename the function to the old name. This name should be temporary.

bring it on.

xaverdh · 2020-09-08T15:04:37Z

With #94442 merged, should this be closed now?

siraben · 2021-04-10T17:46:08Z

#94442 seems to subsume this PR, closing.

hedning reviewed Feb 19, 2019

View reviewed changes

pkgs/build-support/build-fhs-userenv/default.nix Outdated Show resolved Hide resolved

illegalprime force-pushed the chrootenv-2-bwrap branch from 1b36ffb to 6eebb6a Compare February 19, 2019 16:22

illegalprime force-pushed the chrootenv-2-bwrap branch from 6eebb6a to 7d101f2 Compare February 19, 2019 16:49

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 11-100 labels Feb 19, 2019

illegalprime force-pushed the chrootenv-2-bwrap branch from 7d101f2 to 88812e6 Compare February 20, 2019 15:22

GrahamcOfBorg added the 8.has: documentation label Feb 20, 2019

matthewbauer requested review from lukateras and abbradar February 21, 2019 01:42

hedning reviewed Feb 26, 2019

View reviewed changes

lukateras requested changes Feb 26, 2019

View reviewed changes

lukateras reviewed Feb 26, 2019

View reviewed changes

lukateras mentioned this pull request May 28, 2019

Support nested user chrootenv environments #61740

Merged

10 tasks

matthewbauer added this to the 19.09 milestone May 30, 2019

illegalprime force-pushed the chrootenv-2-bwrap branch from 88812e6 to 1a77224 Compare September 15, 2019 11:02

ofborg bot removed the 8.has: documentation label Sep 15, 2019

illegalprime force-pushed the chrootenv-2-bwrap branch from 1a77224 to ef3373e Compare September 15, 2019 11:32

illegalprime added 3 commits September 15, 2019 13:57

build-fhs-userenv: change to using bubblewrap over chrootenv

33c1f6a

build-fhs-userenv: bind mount parts of host etc directly

7ea3ea7

bind mount host's devfs so we can talk to devices

6221036

illegalprime force-pushed the chrootenv-2-bwrap branch from 4fca62f to 6221036 Compare September 15, 2019 19:44

illegalprime requested a review from hedning September 16, 2019 06:24

This was referenced Jul 28, 2020

Setting CAP_SYS_NICE=eip on vrcompositor-launcher crashes SteamVR #92798

Closed

Can't run getcap in steam-run (symbol lookup error: getcap: undefined symbol: cap_get_nsowner) #92795

Closed

raboof mentioned this pull request Jul 31, 2020

Don't replace /etc in fhs-userenv #80457

Closed

Mic92 reviewed Jul 31, 2020

View reviewed changes

Atemu mentioned this pull request Aug 1, 2020

Introduce BuildFHSUserEnv with Bubblewrap as an alternative to chrootenv #94442

Merged

10 tasks

KamilaBorowska mentioned this pull request Aug 25, 2020

discord: use buildFHSUserEnv to avoid crashing #96246

Closed

10 tasks

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

siraben closed this Apr 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build-fhs-userenv: change to using bubblewrap over chrootenv #55973

build-fhs-userenv: change to using bubblewrap over chrootenv #55973

illegalprime commented Feb 18, 2019 •

edited

illegalprime commented Feb 19, 2019

hedning commented Feb 19, 2019

illegalprime commented Feb 20, 2019

matthewbauer commented Feb 21, 2019

illegalprime commented Feb 26, 2019

hedning commented Feb 26, 2019

hedning Feb 26, 2019

lukateras left a comment •

edited

lukateras Feb 26, 2019

lukateras Feb 26, 2019

illegalprime commented Feb 28, 2019

lukateras commented May 28, 2019 •

edited

abbradar commented May 28, 2019 via email

abbradar commented May 30, 2019

lukateras commented May 30, 2019 •

edited

abbradar commented May 30, 2019

vcunat commented Aug 18, 2019

illegalprime commented Sep 15, 2019

illegalprime commented Sep 15, 2019

illegalprime commented Sep 15, 2019

zimbatm commented Feb 5, 2020 •

edited

bachp commented Apr 14, 2020

Atemu commented Jul 28, 2020

Mic92 Jul 31, 2020

raboof commented Jul 31, 2020

Atemu commented Jul 31, 2020

Mic92 commented Jul 31, 2020

Atemu commented Jul 31, 2020 •

edited

Mic92 commented Aug 1, 2020

xaverdh commented Sep 8, 2020

siraben commented Apr 10, 2021

build-fhs-userenv: change to using bubblewrap over chrootenv #55973

build-fhs-userenv: change to using bubblewrap over chrootenv #55973

Conversation

illegalprime commented Feb 18, 2019 • edited

Motivation for this change

Things done

illegalprime commented Feb 19, 2019

hedning commented Feb 19, 2019

illegalprime commented Feb 20, 2019

matthewbauer commented Feb 21, 2019

illegalprime commented Feb 26, 2019

hedning commented Feb 26, 2019

hedning Feb 26, 2019

Choose a reason for hiding this comment

lukateras left a comment • edited

Choose a reason for hiding this comment

lukateras Feb 26, 2019

Choose a reason for hiding this comment

lukateras Feb 26, 2019

Choose a reason for hiding this comment

illegalprime commented Feb 28, 2019

lukateras commented May 28, 2019 • edited

abbradar commented May 28, 2019 via email

abbradar commented May 30, 2019

lukateras commented May 30, 2019 • edited

abbradar commented May 30, 2019

vcunat commented Aug 18, 2019

illegalprime commented Sep 15, 2019

illegalprime commented Sep 15, 2019

illegalprime commented Sep 15, 2019

zimbatm commented Feb 5, 2020 • edited

bachp commented Apr 14, 2020

Atemu commented Jul 28, 2020

Mic92 Jul 31, 2020

Choose a reason for hiding this comment

raboof commented Jul 31, 2020

Atemu commented Jul 31, 2020

Mic92 commented Jul 31, 2020

Atemu commented Jul 31, 2020 • edited

Mic92 commented Aug 1, 2020

xaverdh commented Sep 8, 2020

siraben commented Apr 10, 2021

illegalprime commented Feb 18, 2019 •

edited

lukateras left a comment •

edited

lukateras commented May 28, 2019 •

edited

lukateras commented May 30, 2019 •

edited

zimbatm commented Feb 5, 2020 •

edited

Atemu commented Jul 31, 2020 •

edited