nixos/digital-ocean-image: init #58464

eamsden · 2019-03-28T00:53:27Z

Motivation for this change

A just-works NixOS image that can be uploaded to Digital Ocean, hopefully to be used as a basis for better Digital Ocean support in NixOps.

Things done

colemickens · 2019-03-28T02:14:42Z

Can you share expected example user data?

eamsden · 2019-03-28T03:50:55Z

@colemickens if you mean username/password: it's just nixos/nixos. It should be able to sudo without a password.

I got into a console and looked at the systemd journal. I can't copy-paste from the console so I apologize for the screenshot of text instead of copy-paste:

It looks like cloud-init isn't very robust and doesn't like having "nixos" as the distro. I'll look into that more tomorrow.

colemickens · 2019-03-28T03:53:27Z

@eamsden I meant example user-data you're passing into the Droplet config, or are you not specifying any and just expecting to see the SSH keys?

The problem isn't there. If you scroll further, you see that the network portion of the "vendor-data" config fails to apply. It then doesn't process the user-data or ssh keys, I suspect.

But this just makes me question all the more if it's worth the fun of cloud-init just to get ssh keys when they're accessible in at least one or two easy ways.

colemickens · 2019-03-28T03:56:55Z

I may have been looking at the wrong log, this is the most recent -u cloud-init log I got: https://gist.github.com/colemickens/98dee80a8f5d08188d0e932cdc1004a1

eamsden · 2019-03-28T13:36:05Z

I added the template file it was complaining about (see 37d3a52) and now it's not able to write /etc/hosts because NixOS makes it read-only. Is there an idiomatic way to override that?

Log:
https://gist.github.com/eamsden/bfb5153844f800241549695b48606f11

colemickens · 2019-03-28T16:57:57Z

I don't know. I'm still stuck on the last line of the log which reminds the same: handlers.py[DEBUG]: finish: init-network: FAIL: searching for network datasources. I'm not an expert, but the previous lines make me think the /etc/hosts issue is non-fatal: util.py[WARNING]: Running module update_etc_hosts (<module 'cloudinit.config.cc_update_etc_hosts' from '/nix/store/lfb4vx0fl1v2n30hs79bsy5pwfk622ip-cloud-init-0.7.9/lib/python2.7/site-packages/cloudinit/config/cc_update_etc_hosts.pyc'>) failed since it only says WARNING ? Again, cloud-init could have provided slightly more useful logging and this would be less of a guessing exercise.

To be honest, the time already spent reading cloud-init logs reminded me why I've gone to lengths to avoid this in the past, especially on NixOS where it's only real utility is better served with a few lines of bash.

In case it's useful -- I did write up a nixos/maintainers script that will auto-build and upload the image to Digital Ocean. I got the iteration cycle down to a few minutes. I also opened another PR that reduces the size of the image by about ~230MB by changing the cloud-init->cloud-utils packages. You can find it in here: https://github.com/colemickens/nixpkgs/blob/digitalocean/nixos/maintainers/scripts/digitalocean/upload-image.sh

eamsden · 2019-03-28T17:32:23Z

@colemickens Perhaps I should go ahead and build a utility that can parse the digital ocean metadata and set things properly. The only thing that wouldn't work for would be resetting the root password, since (for obvious reasons) they don't expose that in the metadata, but it isn't too hard to do from the nix config anyway if you really want a root password set.

nlewo

Oh, I just realized this is a draft PR. So, my comments might be not relevant at all!

nixos/modules/virtualisation/digital-ocean-config.nix

eamsden · 2019-03-29T01:50:20Z

@nlewo the stuff you mentioned in the review is to help me debug the nixos image on digital ocean, and will most certainly be removed before the draft tag comes off the PR

eamsden · 2019-04-02T12:00:26Z

It looks like the best approach is going to be a very optional module that can load bits and pieces of configuration from the config drive. Digital Ocean uses an OpenStack config drive so maybe we should just have a generic OpenStack service module? This would let us set (optionally according to the NixOS config)

root password
root SSH keys
network interface configurations
DNS resolvers
hostname
entropy seed

from the Digital Ocean metadata

Further, we could do as the Amazon AMI image does and allow loading of a NixOS module from the user data, so users of the NixOS image could quickly spin up a custom NixOS system directly from a cloud console.

The goal with this is not to enforce compliance with the host provider's metadata, but to make starting with NixOS in a Digital Ocean droplet as painless as possible, while also not interfering with the operation of e.g. NixOps or other deployment tools.

arianvp · 2019-04-10T16:16:22Z

Note that you can just directly curl the digitalocean metadata API for public keys and userdata. No need for bringing in cloud-init and openstack

https://developers.digitalocean.com/documentation/metadata/

You could put this in a systemd unit that starts up at boot:

curl http://169.254.169.254/metadata/v1/public-keys > /root/.ssh/authorised_keys

and:

curl http://169.254.169.254/metadata/v1/user-data > /etc/nixos/configuration.nix
nixos-rebuild switch

This is similar to what the NixOS amazon, GCE and Azure images do

eamsden · 2019-04-10T18:41:02Z

@arianvp I am working on something similar to this. The issue is that we cannot depend on networking being up when we start to configure the system.

Fortunately, there is a config drive available in a standard location, with the same JSON metadata available. I'm working on making sure the instance can configure its networking etc from this.

arianvp · 2019-04-10T18:51:12Z

Why can't we depend on the network being up?
Because networkd / networking isn't configured yet? You can make that part of the base image config. Then the metadata retrieval systemd unit just blocks until it can reach the metadata server and you make it a RequiredBy=metadata.target to make components that wait on the metadata start in the right order.

This is how the coreos digital ocean image works too.

eamsden · 2019-04-11T00:18:08Z

Interesting. Thank you! It wasn't clear from the Digital Ocean documentation that we could depend on DHCP being enabled and configure networking that way. Since it appears that we can. I'll go with your suggestion as soon as I get time to hack on it again.

arianvp · 2019-04-14T21:29:00Z

I was able to successfully boot an image, set up ssh, and hostname, with this config:
https://github.com/arianvp/nixos-stuff/blob/master/modules/digitalocean/config.nix

Feel free to take inspiration from it, or else I'm also happy to provide a patchset myself on top of this PR

eamsden · 2019-04-15T19:12:30Z

@arianvp I'm certainly not going to complain if someone else does work. :)

That said, my plan this week is to take the config you wrote and wrap the systemd services in options, so a user can turn them off in an updated system config, and add something similar to the Amazon/GCE setup where a user can put a NixOS configuration module in the user data and have the machine rebuild itself to that config on startup. I'd also like to make sure the default config is present on the system so that the user can use /etc/nixos/configuration.nix to configure the system without having to manually replicate everything the image does for Digital Ocean compatibility.

arianvp · 2019-04-15T20:30:05Z

Yeh good idea. we should probably disable the hostname fetching when someone sets the hostname in the NixOS config too. Such that nixos-rebuild doesn't race with the metadata daemon

eamsden · 2019-04-16T13:00:34Z

@arianvp I'm testing an image now and then I'll push, then I have to go to $DAYJOB. Some things I'd like to do before I take off the draft label.

Duplicate the support from Amazon/GCE images for putting a NixOS config in the user data.
MIME-decode the vendor data and run the RNG initialization script therein.
Figure out why we get a complete GNOME install in the image (wallpapers, Pango/Cairo, kitchen sink, etc) and stop that.

eamsden · 2019-04-16T13:39:43Z

@arianvp (In the latest push I did make systemd set the hostname from metadata only if it is not set in the NixOS config)

eamsden · 2019-04-16T23:19:02Z

A bit of hunting reveals that cloud-utils is pulled in by nixos/modules/system/boot/grow-partition.nix, via the option boot.growPartition. That seemingly innocuous option pulls in cloud-utils, but just for the growpart utility, which in turn pulls in qemu, which pulls in GTK3. Which means we really need #58469 to de-bloat the image.

eamsden · 2019-04-16T23:36:43Z

I'm hoping the bloat will be fixed separately when #58471 is merged, so I'm going to make this into a real PR now.

@arianvp @colemickens do either of you want to be in the maintainers list for this image? Or should I leave it as just me?

…onfig.nix from NixOS/nixpkgs#58464

arianvp · 2019-04-17T08:32:14Z

You can add me to the maintainers list. I'll be actively using this module do I don't mind maintaining it. Good job on the size hunting.

nixos/modules/virtualisation/digital-ocean-init.nix

nixos/modules/virtualisation/digital-ocean-config.nix

nixos/modules/virtualisation/digital-ocean-image.nix

nixos/modules/virtualisation/digital-ocean-config.nix

… exists before writing it from user data

eamsden · 2019-04-17T12:37:47Z

@arianvp I wasn't able to find you in https://github.com/NixOS/nixpkgs/blob/master/maintainers/maintainer-list.nix. Am I searching the wrong handle perhaps? I was grepping for 'arianvp'.

arianvp · 2019-04-17T12:43:15Z

@eamsden I don't maintain any packages so far, so I'm not in that list yet =)
You can add this entry if you want to:

"arianvp" = {
    email = "arian.vanputten@gmail.com";
    name = "Arian van Putten";
    github = "arianvp";
};

nixos/modules/virtualisation/digital-ocean-init.nix

eamsden · 2019-04-17T13:20:02Z

nixos/modules/virtualisation/digital-ocean-config.nix

+          set -e
+          TEMPDIR=$(mktemp -d)
+          curl --retry-connrefused http://169.254.169.254/metadata/v1/vendor-data | munpack -C $TEMPDIR
+          $TEMPDIR/entropy-seed


There is an error sometimes when this script runs, which causes issues in config switching when importing this module.

is it the curl part that is failing or is it the $TEMPDIR/entropy-seed part that is failing?

I think it is the curl not getting data, which is causing munpack to fail. But I'll have to try it again after work.

It didn't seem to be stopping the base system from coming up from the image, but if I build a system closure that imports digital-ocean-config.nix (I was trying this for my website) and I don't turn off the entropy seed fetcher, it gives an error when switching to it.

Actually this is a known issue with munpack, since 2005 at least! https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=211472

That bug report from 2005 is talking about the same version of munpack packaged by nixpkgs now. I think we need an alternate solution, if we are going to grab the entropy data at all.

For what it's worth, I'm using the following variant:

curl http://169.254.169.254/metadata/v1/vendor-data | munpack -tC $TEMPDIR ENTROPY_SEED=$(grep -rl "DigitalOcean Entropy Seed script" $TEMPDIR) ${pkgs.runtimeShell} $ENTROPY_SEED

eamsden · 2019-04-17T14:27:11Z

nixos/modules/virtualisation/digital-ocean-init.nix

+        echo "attempting to fetch configuration from Digital Ocean user data..."
+        export HOME=/root
+        export NIX_PATH=/nix/var/nix/profiles/per-user/root/channels/nixos:nixos-config=/etc/nixos/configuration.nix:/nix/var/nix/profiles/per-user/root/channels
+        userData=$(mktemp)


I copied this from the Amazon init module and changed obvious things, but I haven't tested it yet.

eamsden · 2019-04-18T02:27:30Z

nixos/modules/virtualisation/digital-ocean-init.nix

+            fi
+
+            echo "setting configuration from Digital Ocean user data"
+            cp "$userData" /etc/nixos/configuration.nix


I'm not entirely happy about this part. I'd like to find a way to make sure that digital-ocean-config is in the module list without having to explicitly include it. That way if a user provides a config in the user data they can assume still that the digital ocean stuff is in place, and disable it via the options if necessary.

cc @arianvp thoughts?

Usually the pattern that people are used to , is:

# configuration.nix { imports = [ ./hardware-configuration.nix ] }

right? At least, that is the standard /etc/nixos/configuration.nix that is being generated.

then just put the DO specific things in /etc/nixos/hardware-configuration.nix

It's not ideal because if people forget, the config won't work... but it is the same as on their laptops.

Other thing we could do is:

cp $userData /etc/nixos/do-userdata.nix

And then hardcode /etc/nixos/configuration.nix to be:

{ imports = [./do-userdata.nix <nixpkgs....blah../do-config.nix> ]; }

Then people don't have to remember to include the hardware-configuration manually

I was thinking along those lines too. But maybe we should stick to what the AWS/GCE images do, which is to put the user data in /etc/nixos/configuration.nix, and then make an issue to discuss this. Or we could just lead the way in doing things in a bit nicer way.

Actually it would be nice to get @infinisil's take as the code owner on this question.

What @arianvp suggested sounds much better, I don't think there's much to discuss there. I'd just go ahead and do it this way, which can serve as an example for others.

@eamsden shall we go with my suggestion? I'd love to see this merged. Been using it for quite a while already on my own nixpkgs fork

@arianvp Yeah go with your suggestion. I too really want to see this merged

…sh keys when users.mutableUsers is disabled

arcnmx · 2019-07-07T04:45:02Z

A few notes from trying this out.

a switch fails because /dev/sda doesn't exist, root appears to be /dev/vda instead
digitalocean-entropy-seed service seems to fail due to munpack tempdesc.txt: File exists

Besides these minor issues it seems to be working pretty well though!

flokli · 2019-07-07T08:27:10Z

nixos/modules/virtualisation/digital-ocean-config.nix

+        };
+      };
+
+      /* Fetch the ssh keys for root from Digital Ocean */


Can you reuse nixos/modules/virtualisation/{openstack-config.nix,ec2-data,amazon-init.nix} here? There's already a lot of metadata server fetching and applying…

No not really. They all are very specific to their platforms and I'm not sure how I could get a lot of code reuse out of them

The openstack-init from openstack-config.nix uses ec2-metadata-fetcher.nix, which creates a service fetching metadata, which is picked up by the service defined in ec2-data.nix (which is imported).

This looks pretty similar to what we're trying to do here (and what we should do in brightbox-image.nix too, btw.)

I made more than one attempt to use openstack to fetch metadata, and eventually settled on just using DigitalOcean's documented metdata service.

nixos/modules/virtualisation/digital-ocean-config.nix

arcnmx

I've been using this for the past month or so and would like to see or help it get pushed to completion if possible! So, I'm using this review to organize what I believe are the remaining three issues that have been brought up regarding this PR, and I think it should be in a pretty good shape once they're addressed?

arcnmx · 2019-07-31T14:58:05Z

nixos/modules/virtualisation/digital-ocean-config.nix

+        script = ''
+          set -e
+          TEMPDIR=$(mktemp -d)
+          curl --retry-connrefused http://169.254.169.254/metadata/v1/vendor-data | munpack -C $TEMPDIR


As has been noted, this fails due to issues with munpack being unable to name the files properly; we can avoid this with the -t argument and find the file by its header comment:

Suggested change

curl --retry-connrefused http://169.254.169.254/metadata/v1/vendor-data | munpack -C $TEMPDIR

curl --retry-connrefused http://169.254.169.254/metadata/v1/vendor-data | munpack -tC $TEMPDIR

ENTROPY_SEED=$(grep -rl "DigitalOcean Entropy Seed script" $TEMPDIR)

${pkgs.runtimeShell} $ENTROPY_SEED

(also remove the following line that executes $TEMPDIR/entropy-seed)

arcnmx · 2019-07-31T15:04:05Z

nixos/modules/virtualisation/digital-ocean-config.nix

+        initrd.kernelModules = [ "virtio_scsi" ];
+        kernelModules = [ "virtio_pci" "virtio_net" ];
+        loader = {
+          grub.device = "/dev/sda";


Suggested change

grub.device = "/dev/sda";

grub.device = "/dev/vda";

All of my droplets have vda rather than sda so this is required for switch to succeed when installing bootloader updates. Not sure if there are machines where sda exists on DO or if this just wasn't tested?

it probably just wasn't tested

arcnmx · 2019-07-31T15:16:31Z

nixos/modules/virtualisation/digital-ocean-init.nix

+            fi
+
+            echo "setting configuration from Digital Ocean user data"
+            cp "$userData" /etc/nixos/configuration.nix


As was previously suggested by arianvp:

Suggested change

cp "$userData" /etc/nixos/configuration.nix

cp "$userData" /etc/nixos/do-userdata.nix

echo '{ modulesPath, ... }: {

imports = [

./do-userdata.nix

(modulesPath + "/virtualisation/digital-ocean-config.nix")

];

}' > /etc/nixos/configuration.nix

(not necessarily suggesting that the template config be inlined like that but within the constraints of github review ui...)

arianvp · 2019-08-06T19:38:41Z

@arcnmx would you want to make a PR based on this PR with your suggested changes? I think this will then be ready to merge

arcnmx · 2019-08-19T19:04:41Z

@arianvp sorry was away for a bit but can do!

infinisil · 2019-10-27T16:17:22Z

Closing in favor of #66978

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) labels Mar 28, 2019

eamsden force-pushed the init_digital_ocean_image branch from 150c3ac to f7991b8 Compare March 28, 2019 00:59

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Mar 28, 2019

colemickens mentioned this pull request Mar 28, 2019

cloud-utils includes qemu; package qemu-utils? #58469

Closed

eamsden changed the title ~~digital-ocean-image: init~~ nixos/digital-ocean-image: init Mar 28, 2019

nlewo reviewed Mar 28, 2019

View reviewed changes

infinisil mentioned this pull request Apr 1, 2019

Use custom images for DigitalOcean instead of nixos-infect NixOS/nixops#1017

Open

nixos/digital-ocean-image: init

79a5a96

eamsden force-pushed the init_digital_ocean_image branch from 0812e66 to 79a5a96 Compare April 16, 2019 23:35

eamsden marked this pull request as ready for review April 16, 2019 23:36

eamsden requested a review from infinisil as a code owner April 16, 2019 23:36

eamsden pushed a commit to eamsden/website that referenced this pull request Apr 17, 2019

Initial commit: can build and deploy server. Requires digital-ocean-c…

e90dff7

…onfig.nix from NixOS/nixpkgs#58464

arianvp requested changes Apr 17, 2019

View reviewed changes

digital-ocean-image: Check that /etc/nixos/configuration.nix does not…

0b98a3f

… exists before writing it from user data

arianvp reviewed Apr 17, 2019

View reviewed changes

nixos/modules/virtualisation/digital-ocean-init.nix Outdated Show resolved Hide resolved

Edward Amsden added 4 commits April 17, 2019 08:57

digital-ocean-image: use modulesPath in default configuration file

4f7d286

nixos/digital-ocean-image: fix string quoting in digital-ocean-init.nix

46ea943

nixos/digital-ocean-image: fix misleading comment on hostname default

5b63fcf

nixos/digital-ocean-image: add arianvp to maintainer list

f36ec61

eamsden commented Apr 17, 2019

View reviewed changes

eamsden commented Apr 18, 2019

View reviewed changes

digital-ocean-image: add assertions about setting root password and s…

1b3f6f5

…sh keys when users.mutableUsers is disabled

flokli reviewed Jul 7, 2019

View reviewed changes

nixos/modules/virtualisation/digital-ocean-config.nix Show resolved Hide resolved

arcnmx suggested changes Jul 31, 2019

View reviewed changes

arcnmx mentioned this pull request Aug 19, 2019

nixos/digital-ocean-image: init (rebase) #66978

Merged

10 tasks

infinisil closed this Oct 27, 2019

eamsden deleted the init_digital_ocean_image branch March 6, 2020 18:36

-          curl --retry-connrefused http://169.254.169.254/metadata/v1/vendor-data | munpack -C $TEMPDIR
+          curl --retry-connrefused http://169.254.169.254/metadata/v1/vendor-data | munpack -tC $TEMPDIR
+          ENTROPY_SEED=$(grep -rl "DigitalOcean Entropy Seed script" $TEMPDIR)
+          ${pkgs.runtimeShell} $ENTROPY_SEED

-            cp "$userData" /etc/nixos/configuration.nix
+            cp "$userData" /etc/nixos/do-userdata.nix
+            echo '{ modulesPath, ... }: {
+              imports = [
+                ./do-userdata.nix
+                (modulesPath + "/virtualisation/digital-ocean-config.nix")
+              ];
+            }' > /etc/nixos/configuration.nix

nixos/digital-ocean-image: init #58464

nixos/digital-ocean-image: init #58464

Conversation

eamsden commented Mar 28, 2019 • edited

Motivation for this change

Things done

colemickens commented Mar 28, 2019

eamsden commented Mar 28, 2019

colemickens commented Mar 28, 2019

colemickens commented Mar 28, 2019

eamsden commented Mar 28, 2019

colemickens commented Mar 28, 2019

eamsden commented Mar 28, 2019

nlewo left a comment

Choose a reason for hiding this comment

eamsden commented Mar 29, 2019

eamsden commented Apr 2, 2019 • edited

arianvp commented Apr 10, 2019 • edited

eamsden commented Apr 10, 2019

arianvp commented Apr 10, 2019 • edited

eamsden commented Apr 11, 2019

arianvp commented Apr 14, 2019 • edited

eamsden commented Apr 15, 2019

arianvp commented Apr 15, 2019 • edited

eamsden commented Apr 16, 2019 • edited

eamsden commented Apr 16, 2019

eamsden commented Apr 16, 2019

eamsden commented Apr 16, 2019

arianvp commented Apr 17, 2019

eamsden commented Apr 17, 2019

arianvp commented Apr 17, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arcnmx commented Jul 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arcnmx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arcnmx Jul 31, 2019 • edited

Choose a reason for hiding this comment

arianvp commented Aug 6, 2019

arcnmx commented Aug 19, 2019

infinisil commented Oct 27, 2019

eamsden commented Mar 28, 2019 •

edited

eamsden commented Apr 2, 2019 •

edited

arianvp commented Apr 10, 2019 •

edited

arianvp commented Apr 10, 2019 •

edited

arianvp commented Apr 14, 2019 •

edited

arianvp commented Apr 15, 2019 •

edited

eamsden commented Apr 16, 2019 •

edited

arcnmx Jul 31, 2019 •

edited