@copumpkin, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @obadz and @domenkozar to be potential reviewers.

cc @rbvermaa who might also be interested, since my main motivator is to use EC2 for image building (and eventually, revamping "VM tests" to work nicely on there too)

Nice!

Does Amazon support UEFI? With it it might be simple enough to just manually do the bootloader installation, which AFAICT is the only thing the VM is used for.

Not really in favor of this. The VM implementation works fine and doesn't require tricks like fakechroot or cptofs.

The slowness comes from the fact that EC2 doesn't use nested virtualization. This is hurting me as well, having 1h cycles to deploy a change is painful. With this and some improvements to image generation we could get down to a few minutes.

I wonder what @obadz think about this :)

By the way, you may check in to using Packet.net for some of these workloads, as they provide bare metal servers via an API. I'm working on NixOps integration, and can also share the temporary tooling I've developed to deploy to them. They run the qemu tests mighty fast.

@grahamc we know that running vm build on non virtualized machine works. We already have Hetzner support. But it's REALLY inconvenient to build vms on other machines where everything you do is based on Amazon.

I know the pain :(

@grahamc that's good to know, but it's still a major complication to datacenter design for stuff like this. Networking within a VPC is a world apart from opening up a VPN connection across providers and so on. It also just feels weird that we normally go to great lengths to make arbitrary builds "pure" and not rely on unnecessary global things, unless it's code that we control that makes unnecessary global assumptions about owning a filesystem; in that case we wrap the whole thing in a VM and call it a day. It feels like a neat trick, but in many ways runInLinuxVM feels like a bit of a crutch that we should strive to avoid. It (arguably) simplifies one thing and massively complicates others, like running a virtualized datacenter 😄

@shlevy just suggested that I might be able to use user namespaces to do a proper native chroot inside the build. I'll give that a go later.

Does Amazon support UEFI? With it it might be simple enough to just manually do the bootloader installation, which AFAICT is the only thing the VM is used for.

@dezgeg you mean using systemd-boot? I don't think EC2 supports UEFI yet, unfortunately, but we probably also don't need a full grub (since a boot menu is useless on EC2). Perhaps something like extlinux could be plugged in easily, but that seems like a bigger change.

@copumpkin The main issue is that I don't want make-disk-image to duplicate nixos-install since that's a maintenance headache. Now, we can probably get most of nixos-install to work fine using a user namespace without too much trouble. (In fact, with Nix 1.12, nixos-install probably won't even need to set up a chroot anymore since it can just do nix-env --store local?root=/mnt ... etc.). However, the real problem is installing the bootloader. I don't see a clean way to do that outside of a VM without tricks. (Can grub-install install to a regular file instead of a block device?)

@edolstra fully agree that duplicating nixos-install logic is shitty, and wasn't planning on leaving the duplication in place for long. My original plan was to factor the code a bit differently to split up things that are universal and things that will only work if "you own the machine". I've already started sketching that out locally but it's not ready yet. The idea would then be to have nixos-install call out to this shared "filesystem setup" script, and then so would the image builder machinery. The filesystem setup script would have a clear contract that it can't assume anything about the surrounding system and must only change things inside the directory/mount point it's been assigned. Does that sound reasonable? I could also add more branching into nixos-install itself, to support being called from an image builder vs. on a native system, but that feels a bit more error-prone and tougher to test, so I'd rather factor it into logical self-contained blocks of functionality.

As you say, the bootloader will probably be more of a pain, as will the activation script. I need to look into whether grub-install can be put directly into an image file, but if not, I might split up the logic a bit and e.g., not pass NIXOS_INSTALL_BOOTLOADER=1 into switch-to-configuration, instead setting up the bootloader separately myself. I do know that extlinux/syslinux support direct writing to a disk image file, so perhaps grub will be similarly friendly 😄 Also, TBC, I'm fine with keeping a minimal amount of VM in place, as long as most of the work can happen outside of it. As I mention above, even with the two remaining VM calls I currently have in this PR, the standard EC2 image build (i.e., the one you have in your AMI builder script) goes from 25 minutes to ~1 minute on EC2. I estimate I could probably get it down to about ~30s with no VM at all, but marginal benefit vs. marginal cost is small.

My main goal is to make sure that you're roughly on the same page before putting more time in this thing. I'm happy to be responsible for making it good (and not duplicating logic, and keeping it simple, and so on) but it'll be frustrating if I put a bunch of time into it only to have you disagree with the overall effort.

I'm also in the position to speed up these ami builds, but I see that it's going to be harder to maintain this code compared to current state.

Could we just have make-disk-image-vmless.nix for those that need to build AMIs on amazon?

@domenkozar eh, the VM-based stuff also breaks inside VMWare and other things. None of today's cloud environments support nested virtualization and they're becoming more and more mainstream. I haven't had a chance to refactor this to share code across nixos-install and this yet, but I think it'll be quite a bit prettier than this. Dismissing it as inherently harder to maintain seems premature.

Scenario: I want to enable somewhat seamless Linux Nix builds inside macOS by running a VM in the background, the same way we run Docker containers on macOS today. Keeping the VM-centric path by default will just be confusing and off-putting to folks who want to try this sort of thing out and who are tempted to press one of the big EC2 buttons on http://nixos.org/nixos/download.html, which is easier by far (if you're used to EC2) than any of the other "try it out" mechanisms on that page.

@edolstra @domenkozar I'm almost ready to push my refactoring of this and nixos-install (I'll probably make a separate PR for it). I think it makes more sense than the current one and will allow this image builder (and others we might want to write!) to share all the core NixOS filesystem logic. Here's a quick sketch of it, just in case you have early feedback:

nixos-prepare-root is a new script that lives adjacent to nixos-install. It's designed to work fully rootless, and explicitly deals with no host Nix-isms. So it doesn't try to build things, doesn't talk to binary caches, or any of that. Its sole purpose is to set up a NixOS root filesystem (I just cut & pasted most of the current nixos-install logic, amending it where relevant), populate the store on it from a set of closures provided on the command line, and set up the system and channel profile links. As such it's basically the subset of nixos-install that works in a Nix builder, minus any commands that might lead to a Nix build (it expects the caller to have built things already and only be passing in pre-existing store paths and closures).

In this world, the new nixos-install takes care of dealing with the host store, running any Nix builds that are necessary (to build the system closure, etc.), and then just passes them into nixos-prepare-root, then invokes the activation scripts.

This image builder, on the other hand, does the same thing but uses Nix's internal build machinery (mostly just exportReferencesGraph) to pre-build the closures that nixos-prepare-root uses.

The end result is that we've factored out the "pure" aspects of nixos-install (setting up filesystems and the Nix store) from things that assume stuff about the host, and there's no logic duplication between image builders and nixos-install. Other nice benefits is that I can get rid of the whole copy-from-other-stores.pl substituter machinery in nixos-install, as well as the work the current image builder does to amend the builder VM's own store.

Thoughts? I actually think it makes more sense than the current installer, should be more maintainable (i.e., testable in isolation, clearer separation of concerns), and allows for this image builder to work with almost no logic duplication.

Edit: I'm not married to the name nixos-prepare-root, and welcome suggestions there too 😄
Edit 2: Also, testing the new image builder, it's quite a bit faster even on KVM-enabled systems. So don't go with me for my sake! It'll make everyone's life more pleasant!

For anyone following this thread, I've created PR #23026 with the nixos-install refactor. I'll then use nixos-prepare-root from there in this image builder and everything will be perfect. I hope 😄

Scenario: I want to enable somewhat seamless Linux Nix builds inside macOS by running a VM in the background

This has been on my back burner ever since I heard about xhyve (and my interest was subsequently renewed after Docker's HyperKit work). Sadly, I lack the bandwidth to take such a project on at the moment. Nonetheless, I'm a strong proponent for any work that gets us closer to achieving that goal. It would go a long ways towards making a case for Nix/OS adoption at $EMPLOYER (present and future), where Macs represent the lion's share of dev environments.

Closing in favor of #24964, which is rebased on top of the (still unmerged) #23026. Please comment/follow there if you're interested in this development.