libvirt: Add different options for VM IP detection #586

Nadrieril · 2017-01-18T21:51:28Z

This PR adds different methods for detecting the VM's IP address, in case DHCP is not used for example.
It also removes the hacky way to get static MACs since we can get the devices' MAC from libvirt. I understand this could cause libvirt's DHCP to more often assign different IPs to a given machine across deployments. I however believe that people needing a static IP should not have to rely on a DHCP anyways (I'm working on enabling that too).

domenkozar · 2017-01-19T12:54:12Z

nix/libvirtd.nix

+    deployment.libvirtd.privateIPv4 = mkOption {
+      default = "dhcp";
+      type = types.str;
+      description = "IP to use to ssh into the machine. Put 'dhcp' to let nixops get the IP from libvirt's dhcp (works with default libvirt network); put 'arp' to let nixops detect IP in the host's ARP table (works with most setups)";


Split into multiple lines using

description = '' '';

Nadrieril · 2017-02-01T12:04:41Z

@domenkozar I have fixed what you suggested.
Are you waiting for some one else's approval or is this mergeable ?

rlupton20 · 2017-02-17T22:52:21Z

Suggest this might help with issue #500 (or at least similar issues).

For me, I'm not using libvirtd for dhcp, so queries with virsh fail, but the arp possibilities here would work.

Of course, from a completely self-interested point of view, I'd like to see this merged 😄

rlupton20 · 2017-02-18T01:28:22Z

Using libvirt, openvswitch and dhcp (dnsmasq) on the router, with the arp setting, trying to create a new vm still gets stuck looking for an arp. However, I can clear it by pinging it's IP from the host.

I'm guessing that's because openvswitch is dealing with all the networking to the VM, so the host never gets the chance to add the IP to it's ARP tables.

Ugh, there should be a nicer way to deal with all of this...

Nadrieril · 2017-02-18T09:51:25Z

Yeah, for arp to work, the VM and the host need to have exchanged at least one packet, which does not always happen. That's why I tend to use only static IPs

rlupton20 · 2017-02-18T10:53:23Z

I'm just looking into inverse arps and similar schemes (but it usually needs special permissions, presumably because we're grabbing layer 2 / raw sockets).

Ping sweeps would be another possibility:

fping seems quite good but needs root permissions.
nmap -sP subnet also would do, and doesn't need root permissions. Old 'n' gold. Running as root with -n even displays the MAC address for us.

libvirtd and various vswitches must be a fairly common use case. ping sweeping a subnet seems like a sledgehammer approach to discovering an IP but at the moment I haven't found a better approach to doing this.

My idea at the moment would be, on merge of the arp functionality you've built, to add another option to libvirtd to aid IP discovery, by ~~periodically ping sweeping a specified subnet. This should make using arps usable in other cases, but requires specifying a subnet.~~ by dumping the xml of the network, grabbing the bridge, and extracting the subnet being used, then periodically ping sweeping it.

All this seems a bit ugly though. Better suggestions?

Nadrieril · 2017-02-18T11:08:42Z

I kind of like this idea ^^ Wouldn't it be enough do ping sweep only just before trying to get the IP from the arp table ?
I'm not sure we can do much better with libvirt unfortunately. See e.g. https://libvirt.org/formatnwfilter.html#nwfelemsRulesAdvIPAddrDetection , where libvirt has to listen to passing packets to determine the IP assigned to the VM's interface.

rlupton20 · 2017-02-18T11:13:01Z

Yeah, that would be more or less how I'd implement it. The only issue I can see is if we don't get an IP in time for the ping sweep to find it. I guess if its an option, we can do it on every test of the arp table, because you'd likely only enable this if you know your virtual network isn't going to be witnessed directly by the host.

rlupton20 · 2017-02-18T11:13:31Z

Perhaps that's what you meant, actually.

Nadrieril · 2017-02-18T11:14:18Z

Pretty much, yes :)

rlupton20 · 2017-02-18T13:45:04Z

I guess we wait for the merge. @domenkozar ?

NotBrianZach · 2017-02-23T22:32:45Z

nixops/backends/libvirtd.py

+                        continue
+                    else:
+                        ip_with_subnet = lines[i + 2]
+                        ip = ip_with_subnet.split('/')[0]


not really a review;
but I wanted to be able to quickly start and restart vms; and since the vms were using dhcp lookups to query services running on the other vms, I ended up adding code like this here in my fork (to avoid having to wait for leases to expire (I think this was (at least part of) the problem (also occurred with your arp method, and was fixed by similar code))

self._logged_exec( ["virsh", "-c", "qemu:///system", "net-update", net, "add", "ip-dhcp-host", "<host mac='{0}' name='{1}' ip='{2}' />".format( mac, self.name, ip), "--live", "--config" ])

along with something like this in nixops destroy (which is stop in libvirtd.py)

map(lambda (net, mac): self._logged_exec( ["virsh", "-c", "qemu:///system", "net-update", net, "delete", "ip-dhcp-host", "<host mac='{0}' name='{1}' ip='{2}' />".format( mac, self.name, self.private_ipv4), "--live", "--config" ]), nets )

all this to say, is a similar issue why you are interested in static ips/ would this change make sense or is there a hidden drawback I don't see?

accidentally opened pull request with referenced changes here https://github.com/NixOS/nixops/pull/610/files :P

My PR adds different methods for nixops to get the VM IP, but does nothing to give the VM a particular IP.
Your change would totally make sense in another PR that would aim at giving the VM a given static IP. It is rather complementary to what I'm doing here, and would be a logical next step.

Wait no, I had not looked at the context of your code. What you are doing is rather weird since it tells libvirt to give to the VM the IP it already has. Moreover, it has unreasonable assumptions like the fact that the given network has a dhcp server, which is exactly what I'm trying to get rid of.

I don't disagree it's weird (leads to pollution of network xml if you don't shut them down properly), problem I had that this fixes is quickly restarting setups like this :
infrastructure-libvirtd.nix

let defaultConfig = { deployment.targetEnv = "libvirtd"; deployment.libvirtd.privateIPv4 = "dhcp"; }; in { apiServer = defaultConfig; postgresqlServer = defaultConfig; }

network.nix

apiServer = { config, pkgs, ... }: systemd.services.postgrest = { description = "Wraps postgresql database with REST api"; # Start the service after the network is available after = [ "network.target" ]; wantedBy = [ "multi-user.target" ]; serviceConfig = { # The actual command to run ExecStart = "${postgrest}/bin/postgrest postgres://root@postgresqlServer:5432/postgres \\ --port 3030 \\ --schema public \\ --anonymous root \\ --pool 200"; }; postgresqlServer = { config, pkgs ? import <nixpkgs> {}, ... }: { ...etc. etc. }

which I suppose you would fix by replacing the hostname (postgresqlServer in this case) by a static ip address (which idk how to do at this point)

...okay I see what you're saying, thanks, I apologize for newbness.

Don't, your idea was reasonable. Sorry if I sounded rough. I would definitely be interested in a PR implementing that once this PR is merged.

hmm looked into this some more and it doesn't seem like there's an easy way to do this through libvirt api without upstream change to said api.
https://www.redhat.com/archives/libvir-list/2016-June/msg01564.html
setting static ip address for guests only works on type=ethernet interfaces. (and I'm not sure if even that is true with qemu)

also just getting rid of --config (and keeping --live flag) removes the need to to the update/delete in nixops destroy (stop). (I think, may require restarting network :P)

rlupton20 · 2017-02-25T15:19:21Z

I'd be keen to see this merged before doing this (if it's actually a good idea), but might we pull out IP detection as a separate abstract class, which has instances for each (broad) method of obtaining an IP? I can see these nested ifs getting a little out of hand the more configuration options are added here.

Is there an argument for doing this more broadly? I'm guessing other hosting systems provide a more reliable way to obtain your IP.

As mentioned, would be keen to see this merged first, so I can do a little work on it, and use it without having to pull from your fork!

Nadrieril · 2017-07-16T10:38:18Z

I'm closing this because I'm not using nixops anymore. If anyone still wants this, you can fetch the branch from my fork and make a new PR yourself.

libvirt: Get mac address from libvirt

8f1929d

Nadrieril mentioned this pull request Jan 18, 2017

libvirt: Allow more complex network setups #585

Closed

domenkozar approved these changes Jan 19, 2017

View reviewed changes

Nadrieril force-pushed the libvirt-ip-detection branch from 09ad555 to eb849b3 Compare January 19, 2017 18:01

libvirt: Add different options for VM IP detection

031ca1b

Nadrieril force-pushed the libvirt-ip-detection branch from eb849b3 to 031ca1b Compare January 19, 2017 23:24

domenkozar added the backends/libvirtd label Jan 20, 2017

NotBrianZach reviewed Feb 23, 2017

View reviewed changes

NotBrianZach mentioned this pull request Feb 27, 2017

Libvirt dhcp lease fix #613

Closed

Nadrieril closed this Jul 16, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libvirt: Add different options for VM IP detection #586

libvirt: Add different options for VM IP detection #586

Nadrieril commented Jan 18, 2017

domenkozar Jan 19, 2017

Nadrieril commented Feb 1, 2017

rlupton20 commented Feb 17, 2017

rlupton20 commented Feb 18, 2017

Nadrieril commented Feb 18, 2017

rlupton20 commented Feb 18, 2017 •

edited

Nadrieril commented Feb 18, 2017

rlupton20 commented Feb 18, 2017

rlupton20 commented Feb 18, 2017

Nadrieril commented Feb 18, 2017

rlupton20 commented Feb 18, 2017

NotBrianZach Feb 23, 2017

NotBrianZach Feb 23, 2017

Nadrieril Feb 23, 2017

Nadrieril Feb 23, 2017 •

edited

NotBrianZach Feb 23, 2017 •

edited

NotBrianZach Feb 23, 2017

Nadrieril Feb 24, 2017

NotBrianZach Feb 26, 2017 •

edited

NotBrianZach Feb 26, 2017 •

edited

NotBrianZach Feb 27, 2017

rlupton20 commented Feb 25, 2017

Nadrieril commented Jul 16, 2017

libvirt: Add different options for VM IP detection #586

libvirt: Add different options for VM IP detection #586

Conversation

Nadrieril commented Jan 18, 2017

domenkozar Jan 19, 2017

Choose a reason for hiding this comment

Nadrieril commented Feb 1, 2017

rlupton20 commented Feb 17, 2017

rlupton20 commented Feb 18, 2017

Nadrieril commented Feb 18, 2017

rlupton20 commented Feb 18, 2017 • edited

Nadrieril commented Feb 18, 2017

rlupton20 commented Feb 18, 2017

rlupton20 commented Feb 18, 2017

Nadrieril commented Feb 18, 2017

rlupton20 commented Feb 18, 2017

NotBrianZach Feb 23, 2017

Choose a reason for hiding this comment

NotBrianZach Feb 23, 2017

Choose a reason for hiding this comment

Nadrieril Feb 23, 2017

Choose a reason for hiding this comment

Nadrieril Feb 23, 2017 • edited

Choose a reason for hiding this comment

NotBrianZach Feb 23, 2017 • edited

Choose a reason for hiding this comment

NotBrianZach Feb 23, 2017

Choose a reason for hiding this comment

Nadrieril Feb 24, 2017

Choose a reason for hiding this comment

NotBrianZach Feb 26, 2017 • edited

Choose a reason for hiding this comment

NotBrianZach Feb 26, 2017 • edited

Choose a reason for hiding this comment

NotBrianZach Feb 27, 2017

Choose a reason for hiding this comment

rlupton20 commented Feb 25, 2017

Nadrieril commented Jul 16, 2017

rlupton20 commented Feb 18, 2017 •

edited

Nadrieril Feb 23, 2017 •

edited

NotBrianZach Feb 23, 2017 •

edited

NotBrianZach Feb 26, 2017 •

edited

NotBrianZach Feb 26, 2017 •

edited