nixos/qemu-vm: Option to use squashfs Nix store closure instead of virtfs access to host's store #72354

chkno · 2019-10-31T07:09:27Z

Motivation for this change

Allow VMs built with nixos-rebuild build-vm to work without any virtfs mounts into the host. In particular, allow them to use a nix store closure in a squashfs like the installer uses.

This allows better isolation of guests. A host that runs multiple VMs may not wish the guests to be able to see each others' derivations.

Things done

chkno · 2019-11-01T17:51:48Z

(Rebased on top of #72342 so the NixOS tests can run)

chkno · 2019-11-01T18:04:42Z

@GrahamcOfBorg test qemu-private-store

matthewbauer · 2019-11-03T17:01:48Z

@GrahamcOfBorg test qemu-private-store

matthewbauer

This all looks like a good idea, though I don't have enough familiarity to really judge whether this breaks anything. Requested a few more people to review.

chkno · 2020-01-29T19:01:55Z

Updated to rebase on top of #55645. I confirm that login, upnp, installer, virtualbox, and the new test pass on x86_64-linux.

matthewbauer

Looks good!

chkno · 2020-03-19T06:27:39Z

Updated test from perl test framework to python test framework.

nixos-discourse · 2020-05-08T04:31:18Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review-may-2019/3032/157

flokli · 2020-05-24T11:06:14Z

nixos/modules/virtualisation/qemu-vm.nix

+    virtualisation.shareExchangeDir =
+      mkOption {
+        default = true;
+        type = types.bool;
+        description =
+          ''
+            Share /tmp/xchg with the host.
+          '';
+      };
+
+    virtualisation.shareSharedDir =
+      mkOption {
+        default = true;
+        type = types.bool;
+        description =
+          ''
+            Share /tmp/shared with the host.
+          '';
+      };


What's the difference of these?

I don't know.

By default they point at the same directory in the host. The test framework points them at different directories by setting SHARED_DIR.

History:

Aug 2011 d75efe4 edolstra: Introduced /tmp/xchg: "For security, don't mount the entire host filesystem."

Aug 2012 a025e7e edolstra: Introduced /tmp/shared: "Provide a common share between VMs to allow easy communication"

May 2020 d85dc4f erikarvstedt: Mentions "xchg is advertised as a bidirectional exchange dir."

Current uses I could find:

Mount Purpose Location

xchg coverage-data nixos/modules/testing/test-instrumentation.nix nixos/lib/test-driver/test-driver.pl

xchg runInMachine nixos/lib/testing.nix

shared copy_from_host copy_from_vm nixos/lib/test-driver/test-driver.py

I just want a lever to turn them off. I guess it makes more sense to be one lever rather than two.

I combined these two options into one and added a note about what these directories are currently used for in the option description.

@erikarvstedt @edolstra can you provide some insight here?

We can probably get rid of coverage-data. AFAIK we're not doing coverage runs in VMs anymore (and IIRC it was only used in some toy examples like the Quake 3 test anyway).

We can probably also eliminate copy_from_* since it's only used in 3 places, which can be replaced by machine.execute("cat ...").

We can probably get rid of coverage-data. AFAIK we're not doing coverage runs in VMs anymore (and IIRC it was only used in some toy examples like the Quake 3 test anyway).

I updated the list in #72828, and filed #97231 removing the remaining coverage-data references - PTAL.

flokli · 2020-07-06T18:50:41Z

This also needs a rebase.

When the store is not shared, the closure used in the VM it is packaged up in a squashfs, just like the ISO installer does.

chkno · 2020-07-06T20:39:44Z

Rebased on top of #92205, #92122, and #90280.

nh2 · 2020-07-06T21:23:10Z

nixos/modules/virtualisation/qemu-vm.nix

+            Share /tmp/shared and /tmp/xchg with the host.  These are used
+            for runInMachine, copy_from_{host,vm}, and coverage-data.


May want to use <literal> markup for easier reading, e.g.

Suggested change

Share /tmp/shared and /tmp/xchg with the host. These are used

for runInMachine, copy_from_{host,vm}, and coverage-data.

Share <literal>/tmp/shared</literal> and <literal>/tmp/xchg</literal> with the host.

These are used for <literal>runInMachine</literal>, <literal>copy_from_{host,vm}</literal>,

and <literal>coverage-data</literal>.

(Please check the manual builds after this, one can accidentlaly screw it up like I did recently.)

@nh2: ... or go even further and use <filename/> and <function/>:

Suggested change

Share /tmp/shared and /tmp/xchg with the host. These are used

for runInMachine, copy_from_{host,vm}, and coverage-data.

Share <filename class="directory">/tmp/shared</filename> and

<filename class="directory">/tmp/xchg</filename> with the host.

These are used for <function>runInMachine</function>,

<literal>copy_from_{host,vm}</literal>, and

<literal>coverage-data</literal>.

nh2 · 2020-07-06T21:33:45Z

nixos/modules/virtualisation/qemu-vm.nix

+            this makes building the VM much faster requires less disk
+            space.  Disable to give the guest a separate, fresh store
+            in a squashfs that contains only the closure of the guest's
+            dependencies.


I think the wording that contains only the closure of the guest's dependencies is a bit misleading, because it may be understood as that if you shareNixStore = true, then all of the hosts nix store is shared with the guest:

Even if you share the nix store with the host, only the closure of the guest's dependencies are "available" in the nix store; all the store paths are present as files in /nix/store, but most of those are not valid (registered in the sqlite DB).

This recently tricked me: https://discourse.nixos.org/t/how-to-build-a-nixos-vm-with-nix-in-which-nixos-rebuild-is-a-no-op/7937

Because of this ambiguity, I think we should add an explicit:

Suggested change

dependencies.

dependencies.

Note that even with <literal>virtualisation.shareNixStore = true;</literal>,

not the entire host Nix store is available in the guest Nix store:

While all <literal>/nix/store</literal> files will be present, only the ones

pulled in by <option>virtualisation.pathsInNixDB</option>

will be registered as valid in the Nix store database.

nh2 · 2020-07-06T21:34:05Z

nixos/modules/virtualisation/qemu-vm.nix

-            appear in the guest Nix store as well, but are considered
-            garbage (because they are not registered in the Nix
-            database in the guest).
+            database in the VM.  If virtualisation.shareNixStore is true,


Suggested change

database in the VM. If virtualisation.shareNixStore is true,

database in the VM. If <option>virtualisation.shareNixStore</option> is true,

nh2 · 2020-07-06T21:35:49Z

nixos/tests/qemu-private-store.nix

+    shared.succeed("[[ $(mount | grep -c virt) -gt 0 ]]")
+    private.succeed("[[ $(mount | grep -c virt) -eq 0 ]]")
+  '';
+})


New tests are great!

Optionally, maybe also add a copy of this test that exercises the default virtualisation.shareNixStore = false;?

This isn't used anymore as per NixOS#72354 (comment).

stale · 2021-03-31T06:16:46Z

I marked this as stale due to inactivity. → More info

now it works

chkno · 2021-10-25T20:53:48Z

Since ec6c604 from #127933 , this can be accomplished via configuration; it doesn't need to be explicitly supported in the qemu-vm module. Thanks, @rnhmjoj!

The necessary configuration is now available at https://github.com/chkno/nixos-qemu-vm-isolation/blob/main/modules/qemu-vm-isolation.nix , which can be used as an example or directly as a library for folks that want this feature.

rnhmjoj · 2021-10-25T21:17:58Z

Nice! It's good to know my cleanup has been useful.

ofborg bot added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 31, 2019

chkno force-pushed the qemu-vm-isolation branch 2 times, most recently from 12cf6fe to 588aaba Compare November 1, 2019 17:50

matthewbauer requested review from aanderse, samueldr and infinisil November 3, 2019 17:01

matthewbauer reviewed Nov 3, 2019

View reviewed changes

matthewbauer requested review from bjornfor and cleverca22 November 3, 2019 17:02

chkno force-pushed the qemu-vm-isolation branch from 588aaba to 9a8668b Compare January 29, 2020 18:58

ofborg bot added 10.rebuild-linux: 1-10 and removed 10.rebuild-linux: 0 labels Jan 29, 2020

matthewbauer approved these changes Jan 29, 2020

View reviewed changes

chkno force-pushed the qemu-vm-isolation branch from 9a8668b to 6fc1303 Compare March 19, 2020 06:25

flokli reviewed May 24, 2020

View reviewed changes

chkno force-pushed the qemu-vm-isolation branch from 6fc1303 to 838cfa9 Compare July 3, 2020 18:15

ofborg bot added the 10.rebuild-linux: 1 label Jul 3, 2020

This was referenced Jul 3, 2020

qemu-vm device name cleanup #92205

Merged

qemu-vm: Fix useBootLoader, remove /boot read-only restriction #92122

Merged

chkno force-pushed the qemu-vm-isolation branch from 838cfa9 to 0c3bff3 Compare July 4, 2020 03:14

flokli requested a review from erikarvstedt July 6, 2020 18:49

flokli requested a review from nh2 July 6, 2020 18:50

chkno added 3 commits July 6, 2020 13:36

nixos/qemu-vm: Option to not share Nix store with host

f3df5ab

When the store is not shared, the closure used in the VM it is packaged up in a squashfs, just like the ISO installer does.

nixos/qemu-vm: New options for turning virtfs mounts off

345f1f8

nixos/qemu-vm: Test for shareNixStore = false

483393d

chkno changed the base branch from staging to master July 6, 2020 20:37

chkno force-pushed the qemu-vm-isolation branch from 0c3bff3 to 483393d Compare July 6, 2020 20:37

nh2 reviewed Jul 6, 2020

View reviewed changes

nh2 reviewed Jul 9, 2020

View reviewed changes

tfc mentioned this pull request Sep 5, 2020

nixos/tests: Leftover tasks towards unified Python integration tests #72828

Closed

flokli added a commit to flokli/nixpkgs that referenced this pull request Sep 5, 2020

nixos/testing: remove remaining coverage-data logic

98d6b55

This isn't used anymore as per NixOS#72354 (comment).

flokli mentioned this pull request Sep 5, 2020

nixos/testing: remove remaining coverage-data logic #97231

Merged

10 tasks

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 31, 2021

r2r-dev added a commit to r2r-dev/nixos-shell that referenced this pull request May 12, 2021

Add NixOS/nixpkgs#72354 to the mix

fb3f52f

now it works

chkno closed this Oct 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/qemu-vm: Option to use squashfs Nix store closure instead of virtfs access to host's store #72354

nixos/qemu-vm: Option to use squashfs Nix store closure instead of virtfs access to host's store #72354

chkno commented Oct 31, 2019

chkno commented Nov 1, 2019

chkno commented Nov 1, 2019

matthewbauer commented Nov 3, 2019

matthewbauer left a comment

chkno commented Jan 29, 2020 •

edited

matthewbauer left a comment

chkno commented Mar 19, 2020

nixos-discourse commented May 8, 2020

flokli May 24, 2020

chkno Jul 3, 2020

flokli Jul 3, 2020

edolstra Jul 7, 2020

edolstra Jul 7, 2020

flokli Sep 5, 2020

flokli commented Jul 6, 2020

chkno commented Jul 6, 2020

nh2 Jul 6, 2020

aszlig Sep 28, 2020

nh2 Jul 6, 2020

nh2 Jul 6, 2020

nh2 Jul 6, 2020

stale bot commented Mar 31, 2021

chkno commented Oct 25, 2021

rnhmjoj commented Oct 25, 2021

Mount	Purpose	Location
xchg	coverage-data	nixos/modules/testing/test-instrumentation.nix nixos/lib/test-driver/test-driver.pl
xchg	runInMachine	nixos/lib/testing.nix
shared	copy_from_host copy_from_vm	nixos/lib/test-driver/test-driver.py

		Share /tmp/shared and /tmp/xchg with the host. These are used
		for runInMachine, copy_from_{host,vm}, and coverage-data.

-            Share /tmp/shared and /tmp/xchg with the host.  These are used
-            for runInMachine, copy_from_{host,vm}, and coverage-data.
+            Share <literal>/tmp/shared</literal> and <literal>/tmp/xchg</literal> with the host.
+            These are used for <literal>runInMachine</literal>, <literal>copy_from_{host,vm}</literal>,
+            and <literal>coverage-data</literal>.

-            Share /tmp/shared and /tmp/xchg with the host.  These are used
-            for runInMachine, copy_from_{host,vm}, and coverage-data.
+            Share <filename class="directory">/tmp/shared</filename> and
+            <filename class="directory">/tmp/xchg</filename> with the host.
+            These are used for <function>runInMachine</function>,
+            <literal>copy_from_{host,vm}</literal>, and
+            <literal>coverage-data</literal>.

-            dependencies.
+            dependencies.
+            Note that even with <literal>virtualisation.shareNixStore = true;</literal>,
+            not the entire host Nix store is available in the guest Nix store:
+            While all <literal>/nix/store</literal> files will be present, only the ones
+            pulled in by <option>virtualisation.pathsInNixDB</option>
+            will be registered as valid in the Nix store database.

	database in the VM. If virtualisation.shareNixStore is true,
	database in the VM. If <option>virtualisation.shareNixStore</option> is true,

nixos/qemu-vm: Option to use squashfs Nix store closure instead of virtfs access to host's store #72354

nixos/qemu-vm: Option to use squashfs Nix store closure instead of virtfs access to host's store #72354

Conversation

chkno commented Oct 31, 2019

Motivation for this change

Things done

chkno commented Nov 1, 2019

chkno commented Nov 1, 2019

matthewbauer commented Nov 3, 2019

matthewbauer left a comment

Choose a reason for hiding this comment

chkno commented Jan 29, 2020 • edited

matthewbauer left a comment

Choose a reason for hiding this comment

chkno commented Mar 19, 2020

nixos-discourse commented May 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flokli commented Jul 6, 2020

chkno commented Jul 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stale bot commented Mar 31, 2021

chkno commented Oct 25, 2021

rnhmjoj commented Oct 25, 2021

chkno commented Jan 29, 2020 •

edited