Why have networkd by default? This is an experimental feature that still has some known bugs. I'd prefer not enabling it by default on installations. Especially because it behaves differently from what people expect

Turns out the answer is: Because I added it there. Which was a stupid mistake on my side! :D. we should remove it

Hm, it's been working well for me... From initial testing turning it off it causes problems for the metadata init services and neither hostname nor ssh keys are making it to the VM :(

That sounds like a (serious) bug in the NixOS networking module. By default networking module should listen to DHCP requests on all interfaces. investigating...

New theory network-online.target works well on systemd-networkd but maybe not with the default networking module. Workaround: add a Restart=always to digitalocean-init such that it keeps restarting until the network is available instead of it failing immediately...

This is me purely speculating though; I'll spin up a digitalocean box and do some actual debugging later tonight

I just did this on my digitalocean image and it seems to work fine! curl will then do the retries with exponential backoff and whatnot

Weird, depending on which network target (link a diff if possible)? I tried connrefused but it did not work for me because curl seems to give up if the error specifically isn't ECONNREFUSED, which is generally the case when the network isn't available or the metadata server isn't reachable/routable.

Perhaps we could use it to unify all our cloud image metadata fetching steps? (Just a thought; and probably for a later PR).

Definitely worth looking into! But yeah, not necessarily for this PR.

They have a service called coreos-metadata.service

Cool, yeah, I think in general that's the way forward here. Whether built on top of afterburn or a bash loop around curl or anything else, it gives us a central point to place the init/detection logic. I'll at least restructure the units around this approach, then even if we kept the network-online dependency for now, it should be easy to make follow-up improvements. I'll also look into writing a retry loop, I'm just slightly wary about making sure that I choose the appropriate retry logic.

Oh I'm running this on networkd I just realised. That's probably why it works.

If you're wary about the retry-loop, we can also let systemd handle that, which is maybe more elegant:

[Service]
Restart=on-failure
RestartSec=zzz
StartLimitIntervalSec=yyy
StartLimitBurst=xxx

I think either way works. Systemd doesn't do exponential backoff though, but im not sure we require that. the link-local server isn't gonna crash from hammering. Restarting every 2 seconds for like 10 times is probably sufficient.

I'll see if I can package afterburner tonight

If you're wary about the retry-loop, we can also let systemd handle that, which is maybe more elegant.

Though I wanted to, it requires that the service type be changed to simple, which then makes it unusable as a dependency for other units. It does work okay for the individual units, but yeah the lack of backoff is slightly annoying... A loop will work fine.

the link-local server isn't gonna crash from hammering. Restarting every 2 seconds for like 10 times is probably sufficient

Probably, my concern is more that we don't want it to spam journald forever, but we do want to make sure it doesn't give up too early if the network is slow to start or whatever, because these services are important for making sure you can access the machine! While also being fairly responsive because if you wait 2 seconds, a bunch of services will start in the meantime.

yeh not having it a oneshot is annoying. I didn't realise oneshot didn't support Restart=

So, pushed a few things:

1. metadata.service added with a retry loop that can be used as a dependency so that services that need metadata can just wait on it before doing their thing. The service is ordered After dhcpcd or systemd-networkd (if you have them enabled) and thus the retry loop isn't likely to be needed except in unusual circumstances, as the service should just start as soon as an ip/route is assigned to the interface.
2. sethostname and entropy services are wantedBy and Before network.target, so that any services that are ordered After network or network-online will wait for them. This does have the effect of delaying the startup of certain services until the droplet has an ip, but that's also kind of the point. networkd is pretty instantaneous, dhcpcd takes about 6 seconds.
3. Some fixes to the userdata init service. It remains part of / After network-online.
4. the password/sshkey services are just part of mult-user but ordered to take place before sshd so you can't have racy scenarios where the server is online and reachable but are unable to login.
5. Some changes to image/init to allow them to share the config file because it seemed kind of silly to duplicate?

I'm considering changing this default to false because:

1. It's a pretty insecure option, setting the password to something accessible from the metadata server to any unauthenticated user on the system that can access the network.
2. It currently resets the root password on every boot rather than first boot. I should maybe add a path condition to make sure it only ever runs once?
3. DO always tells me my custom images don't support root password setting, so I can't even get it to email me the password to test this. Maybe I'm using the wrong Distribution setting when uploading it?

Want to remove these defaults+assertions because mutableUsers=false does not imply that /root/.ssh is readonly, nor even /etc/shadow. In fact, enabling these settings is a likely motivation/reason for a server to use mutableUsers=false in the first place, so that it can provide auth details in some way other than via the nixos config.

Should I be worried that this isn't using https? This is used for root passwords and such..

Also it would be nice to download all files in this unit and store them somewhere to be accessed by the others (in a location only readable by root).

I'm more worried that unless you explicitly firewall it, any process can retrieve the same data. Though, the root password service in particular I think needs some changes and should probably also be disabled by default.

Also it would be nice to download all files in this unit and store them somewhere to be accessed by the others (in a location only readable by root).

I'd considered that but decided against it just because I didn't feel like deciding on a proper path for it or dealing with the corner cases:

1. The hostname can be changed while the server is running, so you can restart that service to update it. Making it depend on data from the first makes that dependency relationship (and the meaning of the systemd service states) a bit more complicated?
2. Each service was interested in a different endpoint, so downloading them all when most services never even run seemed like an unnecessary extra second of boot time.

Though, I believe the json blob contains everything... so each service can probably just use jq to pick out the pieces they need from that one file. I believe the hostname one is the only mutable piece of data so it could just continue to use the network to simplify things.

Nope this is a Link-Local address so should be safe.

"should be safe" doesn't make me feel safe! Can we check that it's a link-local address?

It is by definition a link local address because it's in the link-local private ip range. Ipv4 mandates that the IP-range isn't next-hopped https://tools.ietf.org/html/rfc3927. Only people on the same physical link can talk over the link-local ip range

Oh nice! I feel safe now

I'd be interested to see an example of all these downloaded files to see what they look like and what other info they contain.

I'd be interested to see an example of all these downloaded files to see what they look like and what other info they contain.

Example here.

How about doing something like nix-instantiate --eval '<nixpkgs/nixos>' -A config.system.stateVersion to find the desired channel too?

I'm a bit wary of making significant changes to the logic here, and it comes straight from other modules. I guess ideally there should some be some common module or place for this to go where cleanup can be applied to nixos on all relevant cloud providers?

This should be done by default already, or is there another reason this is added? In addition it would have to be services.openssh.listenAddresses.*.port

This needs to use the xml syntax <literal>...</literal>, same for other option descriptions

Needs githubId since recently

Hi! @arcnmx, I am planning to use this custom image. But I am confused about where is this digitalOceanImage located? Because in the file digital-ocean-image.nix, I only found digitalOcean.

You'll have to manually import the module file before you can use it.

This seems to be a convention all images use. Not sure if it's a good one.

E.g.. https://github.com/nix-community/nixos-generators/blob/master/formats/gce.nix

@arianvp thank you! Yes, I think I have misunderstanding about the use of cfg = config.virtualisation.digitalOceanImage here.