nixos/mptcp: multipath TCP module #59342

teto · 2019-04-12T07:54:32Z

Multipath TCP is an IETF extension of TCP that makes use of several TCP connections to speed up transfer/increase reliability/confidentiality. Applications don't have to be changed (yet they can use the new MPTCP API for even better performance), the kernel will use MPTCP when applicable and fall back to TCP otherwise.

I had this in my repo for a while but #59301 convinced me to propose it. It only works with networkmanager for now (you need to update routing tables depending on added/removed interfaces).

Motivation for this change

Things done

memberbetty

See http://mywiki.wooledge.org/BashFAQ/082

memberbetty · 2019-04-16T21:07:08Z

nixos/modules/services/networking/mptcp/mptcp_up_raw

+fi
+
+
+if [ `grep "$DEVICE_IFACE" "$RT_TABLE" | wc -l` -eq 0 ]; then


memberbetty · 2019-04-16T21:08:42Z

nixos/modules/services/networking/mptcp/mptcp_up_raw

+fi
+
+if [ "$DHCP4_IP_ADDRESS" ]; then
+	SUBNET=`echo $IP4_ADDRESS_0 | cut -d \   -f 1 | cut -d / -f 2`


Extract this code to a function with a meaningful name, because it's duplicated below.

memberbetty · 2019-04-16T21:10:10Z

nixos/modules/services/networking/mptcp/mptcp_up_raw

+	echo "$NUM  $DEVICE_IFACE" >> "$RT_TABLE"
+fi
+
+if [ "$DHCP4_IP_ADDRESS" ]; then


Did you forget an operator here?

memberbetty · 2019-04-16T21:13:10Z

nixos/modules/services/networking/mptcp/default.nix

+    }
+
+    (mkIf (!config.networking.networkmanager.enable) {
+      warnings = [ "You have `networkmanager` disabled. Expect things to break." ];


Spelling (NetworkManager). Why is there only support for NetworkManager?

because that's the only thing I use.

I don't use NetworkManager, but I might switch because of this change.

Is there a reason to not enable this by default for everyone?

Well, given the dependency on the dispatcher script this should be a failure rather a warning.

memberbetty · 2019-04-16T21:13:36Z

nixos/modules/services/networking/mptcp/mptcp_up_raw

+
+RT_TABLE=/etc/iproute2/rt_tables
+
+if [ "$DEVICE_IFACE" = "lo" ]; then


Redundant quotes around lo.

Thanks for the comments, I can definitely improve the hook (copied from the mptcp wiki) and will do but it should already allow you to get started with mptcp (which is why I hurried with the PR :p ).

memberbetty

Interesting PR. I made some minor comments.

Mic92 · 2019-04-17T07:43:47Z

nixos/modules/services/networking/mptcp/mptcp_up_raw

+
+# TODO remove
+echo $PATH
+env > /tmp/if_up_env


Insecure use of temporary files. /tmp/if_up_env could be an attacker controlled symlink.

He said in the comment that he is going to remove that.

I still added the comment so it won't be forgotten.

Mic92 · 2019-04-17T07:45:14Z

nixos/modules/services/networking/mptcp/mptcp_up_raw

+if [ `grep "$DEVICE_IFACE" "$RT_TABLE" | wc -l` -eq 0 ]; then
+	logger -p user.info -t mptcp "Adding new routing table $DEVICE_IFACE"
+	NUM=$(wc -l < "$RT_TABLE")
+	echo "$NUM  $DEVICE_IFACE" >> "$RT_TABLE"


Why do we need this to be created at runtime rather then controlling the whole etc/iproute2/rt_tables file?

One big motivation for these multipath solutions (multipath QUIC/MPTCP/SCTP) is to deal with mobility scenarios so you need to handle dynamic usecases.

By default /etc/iproute2/rt_tables does not exists, which would make:

NUM=$(wc -l < /etc/iproute2/rt_tables) zsh: no such file or directory: /etc/iproute2/rt_tables

fail. If it is an empty file it would also collide with the local table (number 0)

Is it needed at all to maintain rt_table for this to work?

might be fixed by networking.iproute2.enable = true; so.

Mic92 · 2019-04-17T07:45:50Z

nixos/modules/services/networking/mptcp/mptcp_up_raw

+	ip rule add from "$DHCP4_IP_ADDRESS" table "$DEVICE_IFACE"
+else
+	# PPP-interface
+	IPADDR=`echo $IP4_ADDRESS_0 | cut -d \   -f 1 | cut -d / -f 1`


What about ipv6?

what's IPv6 :D ? I haven't tested with IPv6 (haven't done much IPv6 in general). I can try but if someone wanna step up before or after the PR, I would be really happy.

Mic92 · 2019-04-17T07:48:10Z

nixos/modules/services/networking/mptcp/mptcp_up_raw

+fi
+
+if [ -z "$DEVICE_IFACE" ]; then
+	logger -p user.warn -t mptcp "invalid $DEVICE_IFACE"


Is logging to stdout/stderr not better so it would end up in the journald log of network manager?

logger also ends up in journald. I used it because it was in the base example and seemed good enough: like you can set debug levels and a keyword to grep to ("mptcp" here). Now that I am looking to upstream, stdout/stderr may be better.

Upon test failure, I sometimes want to access the VM to check artifacts/rerun tests. This allows to get path to the test driver via: nixosTests.mptcp.driver.outPath

... when qemu.networkingOptions is overriden and thus there is no default SLIRP interface.

Sane defaults for configuration of MPTCP kernel.

Starts a multihomed VM, configure the redundant scheduler with the msh path manager so that all paths are used. We then launch an iperf session. nix-build -A nixosTests.mptcp .

teto · 2019-08-06T09:35:22Z

I've spent a fair deal of time looking into the nixos testing infrastructure (also to reuse it in https://github.com/Mic92/nixos-shell) and finally cleaned up the PR.
Looking forward to getting this merged: Multipath TCP is not easy to configure properly and this is really an area where nixos shines.

teto · 2019-08-13T07:32:21Z

@grahamc I would very much like to reproduce the error locally. Looking at ofborg's repo, I ran this script

#!/bin/sh
# --argstr system thesystem 
HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate ./nixos/release.nix -A manual --option restrict-eval true --option build-timeout 1800 --show-trace

to no avail (it succeeds).

EDIT: got a solution on IRC

stale · 2020-06-01T12:03:19Z

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the
related code and @ mention them in a comment.
Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
irc.freenode.net.

teto · 2020-06-01T12:17:00Z

let me alone bot, I am still on it :)

teto · 2021-04-26T08:55:45Z

I am moving this module and other MPTCP-related software to https://github.com/teto/mptcp-flake.

teto mentioned this pull request Apr 12, 2019

Multipath TCP #59301

Closed

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Apr 12, 2019

memberbetty suggested changes Apr 16, 2019

View reviewed changes

memberbetty reviewed Apr 16, 2019

View reviewed changes

Mic92 reviewed Apr 17, 2019

View reviewed changes

teto mentioned this pull request Jun 25, 2019

Mptcp kernel updates #63713

Merged

10 tasks

teto added 5 commits August 6, 2019 17:57

tests: add a way to retrieve test driver

127c4a4

Upon test failure, I sometimes want to access the VM to check artifacts/rerun tests. This allows to get path to the test driver via: nixosTests.mptcp.driver.outPath

tests: make VM mac address deterministic

0886ea7

tests: fix interface numbering

eda318a

... when qemu.networkingOptions is overriden and thus there is no default SLIRP interface.

nixos/mptcp: multipath TCP module

6c1e6d4

Sane defaults for configuration of MPTCP kernel.

tests/mptcp: test for the multipath TCP module

999c337

Starts a multihomed VM, configure the redundant scheduler with the msh path manager so that all paths are used. We then launch an iperf session. nix-build -A nixosTests.mptcp .

teto force-pushed the mptcp_module branch from 3a64160 to 999c337 Compare August 6, 2019 09:26

teto marked this pull request as ready for review August 6, 2019 09:32

teto requested a review from infinisil as a code owner August 6, 2019 09:32

teto mentioned this pull request Sep 4, 2019

nixos/mptcp: multipath TCP module #68092

Closed

10 tasks

teto mentioned this pull request Oct 15, 2019

nixos testing for NetworkManager #71151

Closed

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 1, 2020

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 1, 2020

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

teto closed this Apr 26, 2021

teto deleted the mptcp_module branch December 31, 2023 16:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/mptcp: multipath TCP module #59342

nixos/mptcp: multipath TCP module #59342

teto commented Apr 12, 2019 •

edited

memberbetty left a comment

memberbetty Apr 16, 2019

memberbetty Apr 16, 2019

memberbetty Apr 16, 2019

memberbetty Apr 16, 2019

teto Apr 17, 2019

memberbetty Apr 17, 2019

Mic92 Apr 17, 2019

memberbetty Apr 16, 2019

teto Apr 17, 2019

memberbetty left a comment

Mic92 Apr 17, 2019

memberbetty Apr 17, 2019

Mic92 Apr 17, 2019

Mic92 Apr 17, 2019

teto Apr 17, 2019

Mic92 Apr 17, 2019 •

edited

Mic92 Apr 17, 2019

Mic92 Apr 17, 2019

Mic92 Apr 17, 2019 •

edited

teto Apr 17, 2019

Mic92 Apr 17, 2019

teto Apr 17, 2019

teto commented Aug 6, 2019

teto commented Aug 13, 2019 •

edited

stale bot commented Jun 1, 2020

teto commented Jun 1, 2020

teto commented Apr 26, 2021

		fi


		if [ `grep "$DEVICE_IFACE" "$RT_TABLE" \| wc -l` -eq 0 ]; then


		RT_TABLE=/etc/iproute2/rt_tables

		if [ "$DEVICE_IFACE" = "lo" ]; then

nixos/mptcp: multipath TCP module #59342

nixos/mptcp: multipath TCP module #59342

Conversation

teto commented Apr 12, 2019 • edited

Motivation for this change

Things done

memberbetty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

memberbetty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 Apr 17, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 Apr 17, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teto commented Aug 6, 2019

teto commented Aug 13, 2019 • edited

stale bot commented Jun 1, 2020

teto commented Jun 1, 2020

teto commented Apr 26, 2021

teto commented Apr 12, 2019 •

edited

Mic92 Apr 17, 2019 •

edited

Mic92 Apr 17, 2019 •

edited

teto commented Aug 13, 2019 •

edited