🐉🔥💀 (draft) (WIP) wireguard: allow whole-internet VPN configuration (do not merge) 💀🔥🐉 #66300

grahamc · 2019-08-07T22:21:02Z

Mimick wg-quick's fwmark business.

wg-quick enables loose reverse path checking, so we do the same here.

To setup a whole-internet VPN with Wireguard after this commit, on
your "client" peer, add these settings:

networking.nameservers = [ "4.2.2.2" "4.2.2.3" ];
networking.wireguard.wg0.fwmark = 51820;
networking.wireguard.wg0.table = "51820";

and still on your client's configuration, set your "server" peer's
entry to:

networking.wireguard.wg0.peers = [
  ...
  {
    ...
    allowedIPs = [ "0.0.0.0/0" ];
  }
];

On your server, you need:

boot.kernel.sysctl."net.ipv4.ip_forward" = "1";

and to run (not sure how to best represent it in NixOS config):

sudo iptables -t nat -A POSTROUTING -j MASQUERADE

Motivation for this change

Things done

Notify maintainers

cc @

grahamc · 2019-08-07T22:25:49Z

nixos/modules/services/networking/wireguard.nix

+          ${optionalString (values.fwmark != null) ''
+            wg set ${name} fwmark "${toString values.fwmark}"
+            ip rule add not fwmark "${toString values.fwmark}" table "${toString values.table}"
+            ip rule add table "${values.table}" suppress_prefixlength 0


I can't stress enough how I have no idea what I'm doing.

Fancy table stuff should be conditionalized on /0 in allowed-ips, not on fwmark being set, since there are many uses of fwmark. If you're going to go this route, just copy the exact logic wg-quick(8) uses for these options.

Suggested change

ip rule add table "${values.table}" suppress_prefixlength 0

ip rule add table main suppress_prefixlength 0

That line should be main - we want to suppress the default route in the main routing table.

This seems to also be missing IPv6 support.

I agree with @zx2c4 this shouldn't be conditionalized on fwmark being set, but we probably shouldn't conditionalize on "allowed-ips contain a default route" either.

Users might set allowedIPsAsRoutes to false, configure peers with ::/0, 0.0.0.0/0, and add routes manually, or have a routing daemon doing it - in that case, we don't really have control about whether a default route is inserted.

Still, we should provide an option to enable the "fancy table stuff" and configure the routing table name and fwmark number being used.

andir · 2019-08-07T22:26:30Z

You probably want to use networking.nat for the masquerade stuff: https://github.com/NixOS/nixpkgs/blob/release-19.03/nixos/modules/services/networking/nat.nix#L38

ajs124

There are multiple ways to achieve 0.0.0.0/0 VPNs besides fwmark, e.g. setting a separate route to your endpoint or using network namespaces, why did you decide on this one?

All of this looks mostly fine to me, but I would be interested in your reasoning.

nixos/modules/services/networking/wireguard.nix

ajs124 · 2019-08-07T22:27:10Z

nixos/modules/services/networking/wireguard.nix

@@ -391,6 +414,7 @@ in
          message = "networking.wireguard.interfaces.${interfaceName} peer «${peer.publicKey}» has both presharedKey and presharedKeyFile set, but only one can be used.";
        }) all_peers;

+    networking.firewall.checkReversePath = "loose";


Would every wireguard user want this or is this only relevant to people trying to do a 0.0.0.0/0 setup?

A good question.

Reverse path filtering ensures some routing symmetry, meaning you can only receive packets from a src addr on an interface, when you would also use that interface to send to that address.

This is not strictly necessary for every wireguard setup.

It is probably necessary in this case, because you have two default routes in place.

loose reverse path filters are the default for most systemd-based distros. I guess we can drop that line, and instead let #66482 handle this.

grahamc · 2019-08-07T22:32:33Z

cc @zx2c4

Mimick wg-quick's fwmark business. wg-quick enables loose reverse path checking, so we do the same here. To setup a whole-internet VPN with Wireguard after this commit, on your "client" peer, add these settings: networking.nameservers = [ "4.2.2.2" "4.2.2.3" ]; networking.wireguard.interfaces.wg0.fwmark = 51820; networking.wireguard.interfaces.wg0.table = "51820"; and still on your client's configuration, set your "server" peer's entry to: networking.wireguard.interfaces.wg0.peers = [ ... { ... allowedIPs = [ "0.0.0.0/0" ]; } ]; On your server, you need: boot.kernel.sysctl."net.ipv4.ip_forward" = "1"; and to run (not sure how to best represent it in NixOS config): sudo iptables -t nat -A POSTROUTING -j MASQUERADE

…n startup fails again.

grahamc · 2019-08-07T22:51:06Z

I chose this route because it was about 24 lines of diff to add it to the existing NixOS config, and it is what wg-quick does, which is good since I don't know anything about networking. ;)

mweinelt · 2019-08-08T06:38:08Z

nixos/modules/services/networking/wireguard.nix

+        example = 51820;
+        type = types.nullOr types.int;
+        description = ''
+          Mark packets with this value in fwmark.


Which packets does it mark? Overlay or underlay?

I don't know.

lheckemann · 2019-08-08T11:00:26Z

#52411 related (issue for network-namespace-based implementation, including an implementation by @anderspapitto)

WilliButz

Apart from what has already been commented, I don't really have anything to add 🙂

For myself I put toghether a 'works-for-me' setup to route everything through my vpn, I only have static default routes through my wireguard tunnel and use a stripped down dhcpcd which isn't allowed to add routes, dns servers or default gateways and instead only replaces the /32 and /128 routes to my wireguard endpoint with networking.dhcpcd.runHook.
This is certainly not for everyone, especially when it comes to weird hotel or train networks, so I appreciate the work on more viable built-in solutions like this and what @lheckemann mentioned 👍 .

ivan · 2019-08-08T12:37:13Z

As I mentioned on IRC, I cannot reach machines on my LAN with this setup enabled.

zx2c4

Fancy table stuff should be conditionalized on /0 in allowed-ips, not on fwmark being set, since there are many uses of fwmark. If you're going to go this route, just copy the exact logic wg-quick(8) uses for these options.

Do not merge this as-is.

Ma27

The change itself seems fine (apart from the comments already made). It's good that the solution is entirely opt-in as there are a lot of approaches to achieve something similar (with different up/downsides) and it still provides a simple solution for folks with less networking experience. 👍 👍

Additionally there are two minor, non-blocking things I'd like to bring up:

You explained how to use it pretty good in the commit message, however I think that it would be even better if this is mentioned in e.g. the NixOS manual to ensure that this information is easily accessible some years later :)
I didn't try it yet, but do you think it would make sense to write a simple testcase for this (in addition to the two wireguard tests we already have)?

flokli · 2019-08-11T19:35:29Z

nixos/modules/services/networking/wireguard.nix

+          ${optionalString (values.fwmark != null) ''
+            wg set ${name} fwmark "${toString values.fwmark}"
+            ip rule add not fwmark "${toString values.fwmark}" table "${toString values.table}"
+            ip rule add table "${values.table}" suppress_prefixlength 0


Suggested change

ip rule add table "${values.table}" suppress_prefixlength 0

ip rule add table main suppress_prefixlength 0

That line should be main - we want to suppress the default route in the main routing table.

This seems to also be missing IPv6 support.

I agree with @zx2c4 this shouldn't be conditionalized on fwmark being set, but we probably shouldn't conditionalize on "allowed-ips contain a default route" either.

Users might set allowedIPsAsRoutes to false, configure peers with ::/0, 0.0.0.0/0, and add routes manually, or have a routing daemon doing it - in that case, we don't really have control about whether a default route is inserted.

Still, we should provide an option to enable the "fancy table stuff" and configure the routing table name and fwmark number being used.

zx2c4 · 2019-08-11T19:59:14Z

wg-quick has dealt with lots of ugly corner cases over a while of development. Instead of repeating that process, why not just try to copy the basic logic and flow of it as much as possible, and then we can review it from there?

Ma27 · 2019-10-18T21:47:39Z

@grahamc although I'm not 100% convinced whether this is the right approach (well, I really don't consider myself a network expert tbh), may I ask if there's anything missing to get this merged? In the end I think that a lot more users would start trying/using WireGuard if stuff like whole-internet-support tasks can be automated in the WireGuard module :)

grahamc · 2019-10-20T13:10:17Z

I don't know :) I would rather another NixOS networking-friendly contributor merge it. Maybe @fpletz or @andir or @flokli or @aszlig.

fpletz

This PR and the functionality it implements is not ready and should not be merged in the current state.

IPv6 is not implemented (needs the same rules with ip -6 rule)
As @zx2c4 and @flokli mentioned, policy routing with ip rule should not be automatically used if only fwmark is set.
If policy routing is used either with another new option or by matching default routes in allowedIPs (see previous point), please make it possible to configure the pref of the rules so that other policy routing rules can still be added and all can work together without issues.
Both added policy routing rules IMHO do not make any sense because they unconditionally (either with or without fwmark) route packets via the routing table for traffic inside the VPN. AFAIK fwmark is only applied to outgoing packets that are not inside the VPN (Wireguards UDP packets), so the correct rule should be something like ip rule add fwmark ${values.fwmark} pref 42 table main for traffic to the wireguard vpn endpoint and all other traffic should go to the wg routing table, like ip rule add pref 43 table ${values.table}, shouldn't it? Those rules can, depending on preference, also be negated but the target routing tables need to be exchanged.

fpletz · 2019-10-21T13:07:57Z

There are multiple ways to achieve 0.0.0.0/0 VPNs besides fwmark, e.g. setting a separate route to your endpoint or using network namespaces, why did you decide on this one?

The problem is that separate routes only reliably work for static default gateway configurations. If you use DHCP, the DHCP client has to set the explicit route to your endpoint via the acquired default gateway which is not an option found in most DHCP clients. While dhcpcd can do this if you add a shell script hook that adds and removes the routes, it's not an ideal solution in general. (See @WilliButz's comment)

Using network namespaces adds another layer of complexity because patching of systemd units is needed. @orivej's solution in #52411 (comment) is nice to understand what's needed but it is not a general solution because IMHO for this feature the default network namespace should be the one that only routes via the VPN. Also note that our networking module is not aware of multiple network namespaces so the abstractions for network interface configurations cannot be used for interfaces on other netns than the default. This would need a lot of work to get right.

grahamc · 2019-10-21T14:23:49Z

Just for reference, my goal for this PR was to demonstrate a PoC solution of mechanically what could be done to get it working. I never actually expected it to merge. (I think/hope I made that clear in all my comments where I'm not ever assertive about anything.)

andir · 2019-10-21T14:53:03Z

I also do not feel very good regarding merging this.

There are the obvious reasons stated above. Additionally I am not feeling well with putting more scripted networking into NixOS. We are already at a point where trying to keep things going while moving the system forward becomes painful (think e.g. the default route issue with scripted networking vs. systemd-networkd, networking in initramfs, …).

We need a coordinated long-term effort to fix things up first.

I also wouldn't merge this as is because we have no way to verify it without running it on a machine and testing it manually. We are trying to sell some kind of "route everything through VPN by just flipping this switch" feature but aren't setup to verify that it will continue working. Ontop of that the wireguard maintainer clearly stated that we shouldn't be merging this without having a closer look at how and what wg-quick implements.

Following up with that @grahamc wrote just above we should probably downgrade this to a draft.

tpanum · 2020-03-10T17:25:48Z

Is there anybody actively working on implementing this? Would be extremely useful to have. The current alternatives are fairly complex.

flokli · 2020-03-10T18:13:36Z

I don't think so - we'd need some more serious networking configuration system refactor first. There was some initial work on that during the NixOS network meetup end of last year, but it stalled due to the lack of time.

nixos-discourse · 2020-07-24T18:58:35Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/route-all-traffic-through-wireguard-interface/1480/17

bqv · 2021-01-27T13:26:29Z

:(

grahamc requested review from WilliButz, fpletz and Ma27 August 7, 2019 22:24

grahamc commented Aug 7, 2019

View reviewed changes

ofborg bot added 6.topic: nixos 8.has: module (update) labels Aug 7, 2019

ajs124 reviewed Aug 7, 2019

View reviewed changes

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Aug 7, 2019

grahamc force-pushed the wg-0.0.0.0/0 branch from e1c1dea to 9b5ef8a Compare August 7, 2019 22:43

grahamc added 2 commits August 7, 2019 18:49

wireguard: if the table is created wrong, this teardown fails and the…

9c70692

…n startup fails again.

grahamc force-pushed the wg-0.0.0.0/0 branch from 9b5ef8a to 9c70692 Compare August 7, 2019 22:50

mweinelt reviewed Aug 8, 2019

View reviewed changes

WilliButz approved these changes Aug 8, 2019

View reviewed changes

zx2c4 suggested changes Aug 9, 2019

View reviewed changes

Ma27 approved these changes Aug 11, 2019

View reviewed changes

flokli requested changes Aug 11, 2019

View reviewed changes

fpletz requested changes Oct 21, 2019

View reviewed changes

grahamc changed the title ~~wireguard: allow whole-internet VPN configuration~~ 🐉🔥💀 (draft) (WIP) wireguard: allow whole-internet VPN configuration (do not merge) 💀🔥🐉 Oct 21, 2019

flokli marked this pull request as draft June 6, 2020 19:06

grahamc closed this Jul 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐉🔥💀 (draft) (WIP) wireguard: allow whole-internet VPN configuration (do not merge) 💀🔥🐉 #66300

🐉🔥💀 (draft) (WIP) wireguard: allow whole-internet VPN configuration (do not merge) 💀🔥🐉 #66300

grahamc commented Aug 7, 2019

grahamc Aug 7, 2019

zx2c4 Aug 8, 2019 •

edited

flokli Aug 11, 2019

andir commented Aug 7, 2019

ajs124 left a comment

ajs124 Aug 7, 2019

grahamc Aug 7, 2019

mweinelt Aug 8, 2019

flokli Aug 11, 2019

grahamc commented Aug 7, 2019

grahamc commented Aug 7, 2019 •

edited

mweinelt Aug 8, 2019

grahamc Aug 11, 2019

lheckemann commented Aug 8, 2019

WilliButz left a comment

ivan commented Aug 8, 2019 •

edited

zx2c4 left a comment •

edited

Ma27 left a comment

flokli Aug 11, 2019

zx2c4 commented Aug 11, 2019

Ma27 commented Oct 18, 2019

grahamc commented Oct 20, 2019

fpletz left a comment •

edited

fpletz commented Oct 21, 2019

grahamc commented Oct 21, 2019

andir commented Oct 21, 2019 •

edited

tpanum commented Mar 10, 2020

flokli commented Mar 10, 2020 via email

nixos-discourse commented Jul 24, 2020

bqv commented Jan 27, 2021

	ip rule add table "${values.table}" suppress_prefixlength 0
	ip rule add table main suppress_prefixlength 0

🐉🔥💀 (draft) (WIP) wireguard: allow whole-internet VPN configuration (do not merge) 💀🔥🐉 #66300

🐉🔥💀 (draft) (WIP) wireguard: allow whole-internet VPN configuration (do not merge) 💀🔥🐉 #66300

Conversation

grahamc commented Aug 7, 2019

Motivation for this change

Things done

Notify maintainers

Choose a reason for hiding this comment

zx2c4 Aug 8, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andir commented Aug 7, 2019

ajs124 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamc commented Aug 7, 2019

grahamc commented Aug 7, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lheckemann commented Aug 8, 2019

WilliButz left a comment

Choose a reason for hiding this comment

ivan commented Aug 8, 2019 • edited

zx2c4 left a comment • edited

Choose a reason for hiding this comment

Ma27 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zx2c4 commented Aug 11, 2019

Ma27 commented Oct 18, 2019

grahamc commented Oct 20, 2019

fpletz left a comment • edited

Choose a reason for hiding this comment

fpletz commented Oct 21, 2019

grahamc commented Oct 21, 2019

andir commented Oct 21, 2019 • edited

tpanum commented Mar 10, 2020

flokli commented Mar 10, 2020 via email

nixos-discourse commented Jul 24, 2020

bqv commented Jan 27, 2021

zx2c4 Aug 8, 2019 •

edited

grahamc commented Aug 7, 2019 •

edited

ivan commented Aug 8, 2019 •

edited

zx2c4 left a comment •

edited

fpletz left a comment •

edited

andir commented Oct 21, 2019 •

edited