CC @basvandijk @aszlig @fpletz

Good stuff!

I see one last issue in journalctl when the server boots:

Oct 28 20:00:14 builder systemd[1]: Starting Address configuration of eth0...
Oct 28 20:00:14 builder network-addresses-eth0-start[910]: adding address 2a01:123:123:123::/64... done
Oct 28 20:00:14 builder network-addresses-eth0-start[910]: adding route 1.2.3.4/27... failed
Oct 28 20:00:14 builder systemd[1]: network-addresses-eth0.service: Main process exited, code=exited, status=1/FAILURE
Oct 28 20:00:14 builder systemd[1]: network-addresses-eth0.service: Failed with result 'exit-code'.
Oct 28 20:00:14 builder systemd[1]: Failed to start Address configuration of eth0.

@nh2: Did you test whether this works on a server with additional subnets?

@aszlig I think the

    "interfaces": {
      "eth0": {
        "ipv4": {
          "routes": [
            {
              "via": "1.2.3.1",
              "prefixLength": 27,
              "address": "1.2.3.0"
            }
          ]
        },

that we're generating now is unnecessary.

It gets placed into network-addresses-eth0 and results in the adding route 1.2.3.4/27... failed from above with RTNETLINK answers: Network is unreachable:

[root@builder:~]# cat $(systemctl cat network-addresses-eth0.service | grep ExecStart | cut -d= -f2)
#! /nix/store/lw7xaqhakk0i1c631m3cvac3x4lc5gr5-bash-4.4-p12/bin/bash -e
state="/run/nixos/network/addresses/eth0"
mkdir -p $(dirname "$state")

echo "1111:1111:1111:1111::/64" >>  $state
echo -n "adding address 1111:1111:1111:1111::/64... "
if out=$(ip addr add "1111:1111:1111:1111::/64" dev "eth0" 2>&1); then
  echo "done"
elif ! echo "$out" | grep "File exists" >/dev/null 2>&1; then
  echo "failed: $out"
  exit 1
fi


state="/run/nixos/network/routes/eth0"
mkdir -p $(dirname "$state")

echo "1.2.3.0/27" >> $state
echo -n "adding route 1.2.3.0/27... "
if out=$(ip route add "1.2.3.0/27"  via "1.2.3.1" dev "eth0" proto static 2>&1); then
  echo "done"
elif ! echo "$out" | grep "File exists" >/dev/null 2>&1; then
  echo "failed: $out"
  exit 1
fi

This is because network-addresses-eth0.service runs before network-setup.service can set up the direct route to the gateway -- it does that already in https://github.com/NixOS/nixpkgs/blob/6f07d27625f28219095c273b97ded3cc416e9d46/nixos/modules/tasks/network-interfaces-scripted.nix#L120.

@rbvermaa, @edolstra: Do we care about supporting NixOS versions older than 18.03? Or what is the general policy on which NixOS versions we support?

@nh2: Also, what's missing here (and what I wanted to implement when testing this) is a migration function that migrates the old net_info into the new one. This of course would need additional testing.

It gets placed into network-addresses-eth0 and results in the adding route 1.2.3.4/27... failed from above with RTNETLINK answers: Network is unreachable:

I've pushed a fix for that:

Only generate the route if it's not for the device that's the default gateway.

Also, what's missing here (and what I wanted to implement when testing this) is a migration function that migrates the old net_info into the new one.

@aszlig I've written an ad-hoc one that goes via sqlite3 and jq that I used to migrate my servers.

Perhaps the logic in there is useful to some extent for making the proper migration script.

https://gist.github.com/nh2/7707e0069df1d4bebc6572c40c85b673

@nh2 about IPv4, can we get NixOps and NixOS networking to configure something like this instead?

ip a add $server_ipv4/32 peer $gateway_ipv4 dev $interface
ip r add default via $gateway_ipv4

We arent supposed to reach anything on the same L2 except the gateway anyhow, this avoids having to configure a separate route for whatever netmask might be set - I saw both /26 and /27 so far…

@flokli In general I agree that there's no point not to set netmask /32 given what Hetzner writes.

I considered doing that, but then opted to first make the minimal change in this PR that fixes the issue.

I would recommend implementing the /32 chance as a separate PR on top of this, unless anybody can come up with whether /32 would make any problems.

@flokli This PR doesn't do migrations, I'd probably add that with a separate PR as well.

I care first about that new Hetzner deployments work, and second that old ones can be migrated.

I am hesitant to auto-migrate nixops Hetzner users from /27 to /32 because who knows what their network config is.

There must be some reason why Hetzner doens't just set it up as /32 -- perhaps it's for the case where you buy multiple IPs for multiple servers, and put a separate switch in between (they have a service for that)?

Again, I'm all for the /32 approach if nobody comes up with a counter case, but it would ideally not hold back this PR just so that we have some solution that works with NixOS out of the box without losing network connectivity now and then, as soon as possible.

I am hesitant to auto-migrate nixops Hetzner users from /27 to /32 because who knows what their network config is.

Same should apply for not auto-migrating existing deployments without asking back ;-)

There must be some reason why Hetzner doens't just set it up as /32 -- perhaps it's for the case where you buy multiple IPs for multiple servers, and put a separate switch in between (they have a service for that)?
I guess it a) grew historically b is easy to setup on some systems, and even when not adding the /26 or /27 route, most of the internet still works 😆

Additional IPv4 adresses are routed via the main IPv4 adress, so you can either add it with netmask /32 on any interface, or route it somewhere else internally (for example, as a peer route via a tunnel link)

Again, I'm all for the /32 approach if nobody comes up with a counter case, but it would ideally not hold back this PR just so that we have some solution that works with NixOS out of the box without losing network connectivity now and then, as soon as possible.

Fine, so let's merge this, and add some migration assistance and the /32 routing as two separate PRs.

@flokli: I tested this with a new server just now and routing to the same subnet doesn't work as it's missing the gateway. So I guess I'll add the /32 prefix to make sure that the machine always uses the gateway instead of sending directly, that way we shouldn't need the extra route.

@flokli: Actually, never mind about the doesn't work... because I've tested this against another host that was misconfigured.