nixos/openvswitch : upgrade 2.5.4 -> 2.9.2 and vlans #35127

netixx · 2018-02-18T17:12:27Z

Originnaly, the need was to fix #34336 and add features in #34851.

With remarks from @volth, openvswitch was upgraded to the latest
stable version (currenty 2.8.1).

Systemd dependencies were refactored according to analysis in #34586.

Vlans and internal interfaces are implemented following the
networking.interfaces template format i.e. each interface is
described using an attributeSet (vlan and type at the moment).
If vlan is present, then interface is added to the vswitch with
given tag (access mode). Type internal enables vswitchd to create
interfaces (see openvswitch docs).

I am not sure about this implementation and this PR may need some tuning. Also, this change breaks compatibility with previous vswitch.interfaces syntax.

EDIT: More details on implementation:
I only included type and vlan options but there are a lot more (see set command in ovs-vsctl) and I hesitated to put a generic key=value options that would be translated in set port/interface <key>=<value>, but I think for usability for the user the minimal options are better. Maybe we could have a list of commands as an interface parameter, and we would replace a token by the name of the current interface in the actual commands (but I don't know how to do that).

About type argument, I originally thought (reading the docs) that it was required to apply this tag to virtual interfaces attached to the bridge, but actually, it leads to an error if the interface already exists (say created by networking.interface with option virtual=true). In the end, the 'type=internalenablesvswitchd` to create the given interface. I think we could use this in a container scenario for example (by changing the network namespace of this interface), which is why I left it.

Motivation for this change

Previous implementation would fail to start correctly.
Version 2.5.2 would create a flap during startup which needed an ugly sleep 1 to work
Vlan feature is now implemented. It enables interfaces to be added in tagged/access mode

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

netixx · 2018-02-19T07:13:48Z

I saw while rebasing that the upgrade to 2.5.4 was performed by @jokogr indeed!

I couldn't test for the setup bug with this version, but 2.8.1 brings interesting stuff (in the IPSec department for example).

jokogr · 2018-02-19T21:12:47Z

pkgs/os-specific/linux/openvswitch/default.nix

-    substituteInPlace $out/share/openvswitch/scripts/ovs-monitor-ipsec \
-      --replace "self.cert_dir" "root_prefix + self.cert_dir"
-  '';
+  # ovs-monitor-ipsec removed in ovs: 2b02d770c4cb381ec32cd4b7b1e991c42b448884


No need to leave this as a comment since the command was removed

Ok. I wanted to make sure I didn't forget some other script in this step. Should I leave the comment about removal of the script in upstream ?

IMHO no, since it is now removed and users should read the OVS documentation before using a new version.

jokogr · 2018-02-19T21:24:16Z

To be honest, I was using extraOvsctlCmds in order to tag ports so far, as well as a minor modification in network-interfaces-scripted.nix to ensure the tagged interfaces were starting after the actual ones.

This one seems much cleaner, let me try in for a couple of days (I am heavily modifying VLANs etc.) before totally approving it.

netixx · 2018-02-20T07:18:25Z

I thought about extraOvsctlCmds too but I didn't want to repeat interface names in multiple places...

Thank you for looking at this ! I have to play with the slaves variables a bit to make it work (e.g. for dhcpcd, if the attached interface is a virtual or internal then we should be able to do dhcp on it).

netixx · 2018-02-24T18:52:27Z

I did some more testing in a more complex scenario and found a few snags in the current implementation, I originally wanted to attach tap (using virtual=true and virtualType=tap) devices to the bridges (in vlans) and assign IP to them, but this does not work see here. Instead, we should let openvswitch create the interfaces (which is what type=internal does) and assign IP addresses on those interfaces.

I worked on a new implementation:

I needed to add before dependency to network-addresses-${i}.service for all internal interfaces because otherwise, ip address assignment failed (Cannot find device in the start script for addresses).
I removed the IPsec related config options (which referenced the script that was removed in 2.8)
I added an schema upgrade script in ovsdb (I got warnings when testing after upgrade to 2.8)
the binding to the subsystem devices was killing the service upon startup (during the pre-start), so I removed it for now. I don't think systemd supports binding to the subsystem when your are deleting/creating said device. Perhaps we should split the service (with creation of the bridge in a separate unit)

I have a couple of questions still:

Openvswitch 2.9 is out, should I push the upgrade as well ?
I saw db = pkgs.stdenv.mkDerivation { in openvswitch.nix, but I don't think it's used for the database (I am not sure though)
What about subsystemDevice binding ?

I will push the upgrade soon (after some more testing on my side).

netixx · 2018-02-25T18:21:37Z

I pushed the changes, I also renames vswitchd to ovs-vswitchd for clarity (we now have ovs-db and ovs-vswitchd as supporting services for ovs).

Also, I updated to 2.8.2 because I had crashes with 2.8.1 (and 2.8.2 is out with some fixes, which solved my crashes).

netixx · 2018-02-25T18:29:27Z

Since we are on the subject of openvswitch, do you know of limitations regarding usage of systemd-networkd with networking.vswitches (I know it is forbidden by nixos, but I don't know why) ?

(I am planning on supporting both networking.useNetworkd and networking.vswitches at the same time because I need it for my setup)

netixx · 2018-02-25T21:05:57Z

Thank you for this info. I was just thinking of configuring the bridge with a nixos generated unit, and then let systemd-networkd do the address and other parts. The way I see it, I just have to insert this config unit before the systemd-network process (and maybe prevent the physical interfaces from matching networkd).

netixx · 2018-02-27T08:22:37Z

I pushed an implementation in the network-interfaces-systemd.nix which means that networking.vswitches can now be used with networking.useNetworkd=true :)

I found that using systemd is much, much, much more stable than scripted interfaces. Internal interfaces (created by openvswitch) can be referred to and configured in the normal networking.interfaces. The only thing that could need fixing (improved, really), is to add a warning if the user tries to configure a IP address and a physical (i.e. not internal) interface : this actually works, but the address will be unusable.

Also, I fixed an mistake in the stop process of the switch (that was not cleaning the switch correctly)

jokogr · 2018-02-27T08:47:13Z

@netixx this is impressive, thank you for your work! I will try the module with systemd-networkd this week.

So far setting VLANs with your module (and not extraOvsctlCmds) is working as expected for my workloads.

Version 2.8.2 seems also stable to me, but I have a small request to make: Is it possible to keep the LTS version as well, and allow users to select it via a package attribute (unsure where to put it, though)?

netixx · 2018-02-27T12:51:10Z

Including both version seems the most flexible choice to me too !

I am not sure how we should implement (I am not very familiar with package building...), do we include a lts flag on the package ? What should the default value be (false ?) ?

Also, should there be a nixos flag as well ?

EDIT:
I propose the following (pushed):

add openvswitch-lts in all-packages.nix, calling openvswitch/lts.nix, this way, we don't break LTS build if we change the latest build (default.nix) because they are totally independent.
use the already existing option config.virtualisation.vswitch.package to choose the version to use.

If this seems good to you, should I bring back the IPsec options in Nixos (if package == openvswitch-lts) given they did not work in the first place ?

netixx · 2018-03-13T17:56:43Z

Did you have time to look at the new implementation ?

fpletz · 2018-03-13T18:39:27Z

nixos/modules/tasks/network-interfaces.nix

@@ -487,9 +513,8 @@ in

          interfaces = mkOption {
            example = [ "eth0" "eth1" ];


Are you sure the example still compatible with the new type?

Good catch! I forgot to update the example. All done now !

jokogr

Many thanks for including the LTS version!

netixx · 2018-03-20T18:51:52Z

I resolved the conflict (by re-adding the IPSec behaviour only for lts version).

I also added the option to select the supported openFlow version for the switch.

Openvswitch was upgraded to the latest stable version (currenty 2.12.0). This remove ovs-monitor-ipsec commands. LTS version is still available using `config.virtualisation.vswitch.package = pkgs.openvswitch-lts` it has been upgraded to 2.5.6. This commit is a split from the original PR NixOS#35127.

Openvswitch was upgraded to the latest stable version (currenty 2.12.0). This remove ovs-monitor-ipsec commands. LTS version is still available using `config.virtualisation.vswitch.package = pkgs.openvswitch-lts` it has been upgraded to 2.5.6. This commit is a split from the original PR NixOS#35127. (cherry picked from commit e8e980e)

Systemd dependencies for scripted mode were refactored according to analysis in #34586. networking.vswitches can now be used with systemd-networkd, although they are not supported by the daemon, a nixos receipe creates the switch and attached required interfaces (just like the scripted version). Vlans and internal interfaces are implemented following the template format i.e. each interface is described using an attributeSet (vlan and type at the moment). If vlan is present, then interface is added to the vswitch with given tag (access mode). Type internal enabled vswitch to create interfaces (see openvswitch docs). Added configuration for configuring supported openFlow version on the vswitch This commit is a split from the original PR #35127.

Systemd dependencies for scripted mode were refactored according to analysis in NixOS#34586. networking.vswitches can now be used with systemd-networkd, although they are not supported by the daemon, a nixos receipe creates the switch and attached required interfaces (just like the scripted version). Vlans and internal interfaces are implemented following the template format i.e. each interface is described using an attributeSet (vlan and type at the moment). If vlan is present, then interface is added to the vswitch with given tag (access mode). Type internal enabled vswitch to create interfaces (see openvswitch docs). Added configuration for configuring supported openFlow version on the vswitch This commit is a split from the original PR NixOS#35127.

Systemd dependencies for scripted mode were refactored according to analysis in #34586. networking.vswitches can now be used with systemd-networkd, although they are not supported by the daemon, a nixos receipe creates the switch and attached required interfaces (just like the scripted version). Vlans and internal interfaces are implemented following the template format i.e. each interface is described using an attributeSet (vlan and type at the moment). If vlan is present, then interface is added to the vswitch with given tag (access mode). Type internal enabled vswitch to create interfaces (see openvswitch docs). Added configuration for configuring supported openFlow version on the vswitch This commit is a split from the original PR #35127.

netixx mentioned this pull request Feb 18, 2018

nixos/vswitches : Fix vswitch-netdev.service going down just after startup #34586

Closed

8 tasks

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 101-500 labels Feb 18, 2018

jokogr reviewed Feb 19, 2018

View reviewed changes

netixx force-pushed the better-openvswitch branch from afbb447 to c16f748 Compare February 25, 2018 18:20

netixx force-pushed the better-openvswitch branch from c16f748 to 1087c24 Compare February 27, 2018 08:16

netixx changed the title ~~nixos/openvswitch : upgrade 2.5.4 -> 2.8.1 and vlans~~ nixos/openvswitch : upgrade 2.5.4 -> 2.8.2 and vlans Feb 27, 2018

netixx force-pushed the better-openvswitch branch 2 times, most recently from 6970d4e to 872236e Compare February 27, 2018 09:20

netixx force-pushed the better-openvswitch branch from 872236e to 06a2d4d Compare February 27, 2018 13:53

GrahamcOfBorg added the 8.has: package (new) label Feb 27, 2018

fpletz reviewed Mar 13, 2018

View reviewed changes

netixx force-pushed the better-openvswitch branch from 06a2d4d to 5ed6a48 Compare March 14, 2018 07:20

jokogr approved these changes Mar 15, 2018

View reviewed changes

netixx force-pushed the better-openvswitch branch from 5ed6a48 to e124038 Compare March 20, 2018 18:50

netixx force-pushed the better-openvswitch branch from e124038 to 194295e Compare March 20, 2018 19:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/openvswitch : upgrade 2.5.4 -> 2.9.2 and vlans #35127

nixos/openvswitch : upgrade 2.5.4 -> 2.9.2 and vlans #35127

netixx commented Feb 18, 2018 •

edited

netixx commented Feb 19, 2018

jokogr Feb 19, 2018

netixx Feb 20, 2018

jokogr Feb 27, 2018

netixx Feb 27, 2018

jokogr commented Feb 19, 2018

netixx commented Feb 20, 2018

netixx commented Feb 24, 2018

netixx commented Feb 25, 2018

netixx commented Feb 25, 2018

netixx commented Feb 25, 2018

netixx commented Feb 27, 2018

jokogr commented Feb 27, 2018

netixx commented Feb 27, 2018 •

edited

netixx commented Mar 13, 2018

fpletz Mar 13, 2018

netixx Mar 14, 2018 •

edited

jokogr left a comment

netixx commented Mar 20, 2018

		@@ -487,9 +513,8 @@ in

		interfaces = mkOption {
		example = [ "eth0" "eth1" ];

nixos/openvswitch : upgrade 2.5.4 -> 2.9.2 and vlans #35127

nixos/openvswitch : upgrade 2.5.4 -> 2.9.2 and vlans #35127

Conversation

netixx commented Feb 18, 2018 • edited

Motivation for this change

Things done

netixx commented Feb 19, 2018

jokogr Feb 19, 2018

Choose a reason for hiding this comment

netixx Feb 20, 2018

Choose a reason for hiding this comment

jokogr Feb 27, 2018

Choose a reason for hiding this comment

netixx Feb 27, 2018

Choose a reason for hiding this comment

jokogr commented Feb 19, 2018

netixx commented Feb 20, 2018

netixx commented Feb 24, 2018

netixx commented Feb 25, 2018

netixx commented Feb 25, 2018

netixx commented Feb 25, 2018

netixx commented Feb 27, 2018

jokogr commented Feb 27, 2018

netixx commented Feb 27, 2018 • edited

netixx commented Mar 13, 2018

fpletz Mar 13, 2018

Choose a reason for hiding this comment

netixx Mar 14, 2018 • edited

Choose a reason for hiding this comment

jokogr left a comment

Choose a reason for hiding this comment

netixx commented Mar 20, 2018

netixx commented Feb 18, 2018 •

edited

netixx commented Feb 27, 2018 •

edited

netixx Mar 14, 2018 •

edited