libvirt: 5.4.0 -> 6.1.0 #81161

wedens · 2020-02-27T05:57:35Z

Motivation for this change

Things done

I'm unable to test on Darwin, but I tested with buildFromTarball set to true and false.

wedens · 2020-02-27T06:01:56Z

Hopefully it can make it into 20.03 to avoid qemu co-existing with old libvirt (potential issues), but I guess there is just not enough time to extensively test this update.

Izorkin · 2020-02-27T06:20:07Z

@wedens fixed this bug? - #66584 (comment)

wedens · 2020-02-27T06:27:00Z

@Izorkin Unfortunately, no idea. I don't use (and can't easily setup) bridge networking to test it. If you (or someone else who uses bridge) can test, that'd be neat.

Works fine with NAT networking fwiw.

Izorkin · 2020-02-27T06:51:37Z

@wedens error build pythonPackages.libvirt

Executing setuptoolsBuildPhase
libvirt-python requires Python 3.x to build

Izorkin · 2020-02-27T06:57:34Z

If rebuild systev used python2Packages.libvirt, need change to python3Packages.libvirt

jonringer · 2020-02-27T07:00:04Z

please freeze the 5.4.0 version for python 2

[41 built (1 failed), 271 copied (5085.6 MiB), 916.4 MiB DL]
error: build of '/nix/store/adv1j8r1342895ysgviibjbbnl5y8d4b-env.drv' failed
https://github.com/NixOS/nixpkgs/pull/81161
4 package failed to build:
nixops nixopsUnstable nixops_1_6_1 python27Packages.libvirt

33 package built:
collectd collectd-data docker-machine-kvm docker-machine-kvm2 gnome3.gnome-boxes haskellPackages.libvirt-hs libguestfs libguestfs-with-appliance libvirt libvirt-glib libvmi minikube minishift perl528Packages.SysVirt perl530Packages.SysVirt python27Packages.guestfs python37Packages.guestfs python37Packages.libvirt python38Packages.guestfs python38Packages.libvirt rubyPackages.ruby-libvirt rubyPackages_2_5.ruby-libvirt rubyPackages_2_7.ruby-libvirt terraform-full terraform-providers.libvirt terraform_0_11-full terragrunt vagrant virt-manager virt-manager-qt virt-top virt-viewer virtlyst

Izorkin · 2020-02-27T07:02:34Z

With 5.9.0 correct builded

wedens · 2020-02-27T08:11:46Z

Yeah, python2 support was dropped in 6.0.0. I'll try to figure out how to use different versions for python2/3. Hopefully it won't make it confusing.

wedens · 2020-02-27T08:25:01Z

Alright, I've added 5.9.0 for python2. Not sure if it's the correct thing to do though (how possible that there will be some incompatibilities between different libvirt and libvirt-python versions?).

@jonringer maybe I misunderstood and you suggest pinning both libvirt and libvirt-python to the latest version that supports python2?

Izorkin · 2020-02-27T12:39:08Z

@wedens error rebuild
error: assertion failed at ../pkgs/development/python-modules/libvirt/5.9.0.nix:8:9

Izorkin · 2020-02-27T12:40:35Z

@GrahamcOfBorg build nixops

Izorkin · 2020-02-27T14:03:16Z

@wedens error build. Used libvirt 6.0.0, needed 5.9.0

Executing setuptoolsBuildPhase
running bdist_wheel
running build
/nix/store/w6pfcj87889hgz0f77blq59z85mdrsh8-pkg-config-0.29.2/bin/pkg-config --print-errors --atleast-version=0.9.11 libvirt
/nix/store/bh447min7f6vqdw3x1iywr1vxyx4ix8y-python-2.7.17/bin/python2.7 generator.py libvirt /nix/store/4aqz37c2fdg3klqhxhji707ypzdcra96-libvirt-6.0.0/share/libvirt/api/libvirt-ap$
Found 478 functions in /nix/store/4aqz37c2fdg3klqhxhji707ypzdcra96-libvirt-6.0.0/share/libvirt/api/libvirt-api.xml
Found 0 functions in libvirt-override-api.xml
Generated 389 wrapper functions
/nix/store/bh447min7f6vqdw3x1iywr1vxyx4ix8y-python-2.7.17/bin/python2.7 generator.py libvirt-qemu /nix/store/4aqz37c2fdg3klqhxhji707ypzdcra96-libvirt-6.0.0/share/libvirt/api/libvi$
Found 5 functions in /nix/store/4aqz37c2fdg3klqhxhji707ypzdcra96-libvirt-6.0.0/share/libvirt/api/libvirt-qemu-api.xml
Found 0 functions in libvirt-qemu-override-api.xml
Generated 3 wrapper functions
Traceback (most recent call last):
  File "generator.py", line 2143, in <module>
    qemuBuildWrappers(sys.argv[1])
  File "generator.py", line 2008, in qemuBuildWrappers
    items.sort(key=lambda i: (int(i[1]), i[0]))
  File "generator.py", line 2008, in <lambda>
    items.sort(key=lambda i: (int(i[1]), i[0]))
ValueError: invalid literal for int() with base 10: 'VIR_DOMAIN_AGENT_RESPONSE_TIMEOUT_BLOCK'
error: command '/nix/store/bh447min7f6vqdw3x1iywr1vxyx4ix8y-python-2.7.17/bin/python2.7' failed with exit status 1

wedens · 2020-02-27T14:21:19Z

I'm not sure what to do about it, tbh.

I'd rather not use a band aid like updating to 5.9.0 instead of 6.0.0 and put off the python2 problem for later.

jonringer · 2020-02-27T21:16:25Z

I'd rather not use a band aid like updating to 5.9.0 instead of 6.0.0 and put off the python2 problem for later.

If it was just the python-module, i would be fine with deprecating. But I'm not okay with breaking nixops packages

wedens · 2020-02-28T03:38:40Z

What are chances that nixops will update to python 3 in the nearby future?

I guess it'll be a reasonable compromise to update to 5.9.0 for now and to 6.* closer to 20.09 release. If there is a high chance that nixops will move to python 3 somewhere before 20.09.

Anyway, I'll wait for more opinions on the subject.

jonringer · 2020-02-28T05:52:34Z

Since the python2 was not the goal of this PR, you could also just do a freeze at 5.4.0, instead of bumping to 5.9.0

wedens · 2020-02-28T06:15:09Z

@jonringer Are you suggesting having two libvirt/libvirt-python versions (5.4.0 and 6.0.0) in nixpkgs?

I'm not sure how this compatibility works, but as far as I understand, if libvirt daemon (presumably used via libvirt nixos module) is 6.0.0, nixops (that'll use libvirt-python with libvirt 5.4.0), won't be able to manage this libvirt daemon (which is 6.0.0).

jonringer · 2020-02-28T07:05:12Z

I suggesting that, instead of freezing the python2 version at 5.9.0, you keep it at 5.4.0. python3 will be at latest

wedens · 2020-02-28T08:30:23Z

Ok. I've added libvirt_5_4_0 attr which is used as a dependency for python-libvirt 5.4.0 with python2.

wedens · 2020-03-06T10:26:03Z

With 6.0.0 frezing start server.

Any details?

Izorkin · 2020-03-06T10:28:10Z

With 6.0.0 frezing start server.

Is it the same issue we had trying to bump 5.4.0->5.6.0 (#66584) ?
You offered a fix then, does it help with 6.0.0 ?

Yes, this error.
My fix not full worked. After nixos-rebuild switch need manual runnig systemctl reload br0-netdev

Izorkin · 2020-03-06T10:42:25Z

@volth new version libvirt used systemd socket activation - https://bugzilla.redhat.com/show_bug.cgi?id=1750340

Izorkin · 2020-03-09T12:22:17Z

With libvirt 5.9.0 and 6.1.0 correct work.

nixos-discourse · 2020-03-09T13:25:49Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-20-03-beta/5935/22

mmilata · 2020-03-11T00:21:26Z

Thank you for the effort to bring recent libvirt into 20.03! I've briefly tested this branch and things seem to work fine - installing QEMU/KVM virtual machine and using it with the default NATted network as well as NixOS-manged bridge.

jonringer · 2020-03-14T22:14:42Z

Personally, i would like to see the merge commits be removed, should be able to clean up with git rebase -i.

otherwise LGTM

[20 built, 0.0 MiB DL]
https://github.com/NixOS/nixpkgs/pull/81161
38 package built:
collectd collectd-data docker-machine-kvm docker-machine-kvm2 gnome3.gnome-boxes haskellPackages.libvirt-hs libguestfs libguestfs-with-appliance libvirt libvirt-glib libvirt_5_4_0 libvmi minikube minishift nixops nixopsUnstable nixops_1_6_1 perl528Packages.SysVirt perl530Packages.SysVirt python27Packages.guestfs python27Packages.libvirt python37Packages.guestfs python37Packages.libvirt python38Packages.guestfs python38Packages.libvirt rubyPackages.ruby-libvirt rubyPackages_2_5.ruby-libvirt rubyPackages_2_7.ruby-libvirt terraform-full terraform-providers.libvirt terraform_0_11-full terragrunt vagrant virt-manager virt-manager-qt virt-top virt-viewer virtlyst

… activation

wedens · 2020-03-15T04:31:43Z

Personally, i would like to see the merge commits be removed, should be able to clean up with git rebase -i.

Done.

wedens · 2020-03-15T04:33:30Z

It makes virsh unreliable, sometimes it get stuck :(

@volth Is it some known issue with libvirt itself?

Izorkin · 2020-03-15T06:39:06Z

Really upgrade 5.4 to 5.9 and 6.1?
I use this patch.

From 908c3140f04fa1e9a6a18d2e19831d51f7e800f7 Mon Sep 17 00:00:00 2001
From: Izorkin <izorkin@elven.pw>
Date: Fri, 6 Mar 2020 12:16:33 +0300
Subject: [PATCH 1/6] libvirt: 5.4.0 -> 5.9.0

---
 .../libvirt/{5.4.0.nix => 5.9.0.nix}          | 34 +++----------------
 .../libvirt/{5.4.0.nix => 5.9.0.nix}          |  4 +--
 pkgs/top-level/all-packages.nix               |  2 +-
 pkgs/top-level/python-packages.nix            |  4 +--
 4 files changed, 10 insertions(+), 34 deletions(-)
 rename pkgs/development/libraries/libvirt/{5.4.0.nix => 5.9.0.nix} (77%)
 rename pkgs/development/python-modules/libvirt/{5.4.0.nix => 5.9.0.nix} (85%)

diff --git a/pkgs/development/libraries/libvirt/5.4.0.nix b/pkgs/development/libraries/libvirt/5.9.0.nix
similarity index 77%
rename from pkgs/development/libraries/libvirt/5.4.0.nix
rename to pkgs/development/libraries/libvirt/5.9.0.nix
index dd0a88974d0..3a57a91a22a 100644
--- a/pkgs/development/libraries/libvirt/5.4.0.nix
+++ b/pkgs/development/libraries/libvirt/5.9.0.nix
@@ -4,7 +4,7 @@
 , iproute, iptables, readline, lvm2, utillinux, systemd, libpciaccess, gettext
 , libtasn1, ebtables, libgcrypt, yajl, pmutils, libcap_ng, libapparmor
 , dnsmasq, libnl, libpcap, libxslt, xhtml1, numad, numactl, perlPackages
-, curl, libiconv, gmp, zfs, parted, bridge-utils, dmidecode
+, curl, libiconv, gmp, zfs, parted, bridge-utils, dmidecode, glib
 , enableXen ? false, xen ? null
 , enableIscsi ? false, openiscsi
 , enableCeph ? false, ceph
@@ -17,50 +17,26 @@ let
   buildFromTarball = stdenv.isDarwin;
 in stdenv.mkDerivation rec {
   pname = "libvirt";
-  version = "5.4.0";
+  version = "5.9.0";

   src =
     if buildFromTarball then
       fetchurl {
         url = "http://libvirt.org/sources/${pname}-${version}.tar.xz";
-        sha256 = "0ywf8m9yz2hxnic7fylzlmgy4m353r4vv5zsvp89zq5yh4h81yhw";
+        sha256 = "0fc9jxw3v6x5hc10bkd7bbcayn24hbld5adj2gh5s648v7hx55il";
       }
     else
       fetchgit {
         url = git://libvirt.org/libvirt.git;
         rev = "v${version}";
-        sha256 = "1dja1mf295w0sl83zag62c4j55cfbzzfbhdxpkyv2zm3zv0mwdyc";
+        sha256 = "0smm77ag8bg24xkbhl4akqikjrsq2pd3wk31nj0hk1avqnl00gmk";
         fetchSubmodules = true;
       };

-  patches = optionals (!stdenv.isDarwin) [
-    (fetchpatch {
-      name = "5.4.0-CVE-2019-10161.patch";
-      url = "https://libvirt.org/git/?p=libvirt.git;a=patch;h=aed6a032cead4386472afb24b16196579e239580";
-      sha256 = "19k9z9xx68nf03igbgy1imxnlp5ppj7cgdbq9kri3s834hkjcygs";
-    })
-  ] ++ [
-    (fetchpatch {
-      name = "5.4.0-CVE-2019-10166.patch";
-      url = "https://libvirt.org/git/?p=libvirt.git;a=patch;h=db0b78457f183e4c7ac45bc94de86044a1e2056a";
-      sha256 = "17pd1rab2mxj4q0vg30vi2gh78mf52ik1p5l12wrghb0wjf7swml";
-    })
-    (fetchpatch {
-      name = "5.4.0-CVE-2019-10167.patch";
-      url = "https://libvirt.org/git/?p=libvirt.git;a=patch;h=8afa68bac0cf99d1f8aaa6566685c43c22622f26";
-      sha256 = "0hgbwk0y2n6ihzjk8vqabhw914axjqgzcb7c5xx893r86c54c0ml";
-    })
-    (fetchpatch {
-      name = "5.4.0-CVE-2019-10168.patch";
-      url = "https://libvirt.org/git/?p=libvirt.git;a=patch;h=bf6c2830b6c338b1f5699b095df36f374777b291";
-      sha256 = "0s4hc3hsjncx1852ndjas1nng9v23pxf4mi1jxcajsqvhw89la0g";
-    })
-  ];
-
   nativeBuildInputs = [ makeWrapper pkgconfig ];
   buildInputs = [
     libxml2 gnutls perl python2 readline gettext libtasn1 libgcrypt yajl
-    libxslt xhtml1 perlPackages.XMLXPath curl libpcap
+    libxslt xhtml1 perlPackages.XMLXPath curl libpcap glib
   ] ++ optionals (!buildFromTarball) [
     libtool autoconf automake
   ] ++ optionals stdenv.isLinux [
diff --git a/pkgs/development/python-modules/libvirt/5.4.0.nix b/pkgs/development/python-modules/libvirt/5.9.0.nix
similarity index 85%
rename from pkgs/development/python-modules/libvirt/5.4.0.nix
rename to pkgs/development/python-modules/libvirt/5.9.0.nix
index 46d8a990f53..ef5da737e08 100644
--- a/pkgs/development/python-modules/libvirt/5.4.0.nix
+++ b/pkgs/development/python-modules/libvirt/5.9.0.nix
@@ -2,12 +2,12 @@

 buildPythonPackage rec {
   pname = "libvirt";
-  version = "5.4.0";
+  version = "5.9.0";

   src = fetchgit {
     url = git://libvirt.org/libvirt-python.git;
     rev = "v${version}";
-    sha256 = "0ja35z90i3m7vsjfpzfm7awkmja3h0150376i5pzmf2q8vp61fi5";
+    sha256 = "0qvr0s7yasswy1s5cvkm91iifk33pb8s7nbb38zznc46706b358r";
   };

   nativeBuildInputs = [ pkgconfig ];
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 93a756f5a07..70f5ab1d5a1 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -13237,7 +13237,7 @@ in
   libversion = callPackage ../development/libraries/libversion { };

   libvirt = callPackage ../development/libraries/libvirt { };
-  libvirt_5_4_0 = callPackage ../development/libraries/libvirt/5.4.0.nix { };
+  libvirt_5_9_0 = callPackage ../development/libraries/libvirt/5.9.0.nix { };

   libvirt-glib = callPackage ../development/libraries/libvirt-glib { };

diff --git a/pkgs/top-level/python-packages.nix b/pkgs/top-level/python-packages.nix
index b48c0e31954..86ab9144dc7 100644
--- a/pkgs/top-level/python-packages.nix
+++ b/pkgs/top-level/python-packages.nix
@@ -6408,9 +6408,9 @@ in {

   libvirt = if isPy3k then (callPackage ../development/python-modules/libvirt {
     inherit (pkgs) libvirt pkgconfig;
-  }) else (callPackage ../development/python-modules/libvirt/5.4.0.nix {
+  }) else (callPackage ../development/python-modules/libvirt/5.9.0.nix {
     inherit (pkgs) pkgconfig;
-    libvirt = pkgs.libvirt_5_4_0;
+    libvirt = pkgs.libvirt_5_9_0;
   });

   rpdb = callPackage ../development/python-modules/rpdb { };
--
2.25.1

wedens · 2020-03-15T08:15:06Z

Alright, let's do it while we're at it. @Izorkin I've applied your patch

wedens · 2020-03-24T03:27:58Z

How can we move this PR forward?

nh2 · 2020-03-28T03:40:55Z

What's remaining here?

Would be very nice to have working libvirt in 20.03!

globin · 2020-03-28T10:18:32Z

I'll work on this first thing on monday

worldofpeace · 2020-03-29T16:46:25Z

@globin Perfect, I'd like to see this in 20.03 so we can have this working. Thanks.

Co-Authored-By: conferno <conferno@camfex.cz>

FRidh

Python bit looks good. I haven't checked the rest.

wedens requested review from FRidh and jonringer as code owners February 27, 2020 05:57

ofborg bot added the 6.topic: python label Feb 27, 2020

ofborg bot requested review from fpletz and globin February 27, 2020 06:03

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Feb 27, 2020

wedens force-pushed the libvirt-6.0.0 branch from a59199d to f97c95e Compare February 27, 2020 08:23

ofborg bot added the 8.has: clean-up label Feb 27, 2020

wedens force-pushed the libvirt-6.0.0 branch from f97c95e to d11295c Compare February 27, 2020 13:44

wedens force-pushed the libvirt-6.0.0 branch from d11295c to d239632 Compare February 28, 2020 08:29

wedens changed the title ~~libvirt: 5.4.0 -> 6.0.0~~ libvirt: 5.4.0 -> 6.1.0 Mar 9, 2020

ofborg bot added 6.topic: nixos 8.has: module (update) labels Mar 9, 2020

mmilata mentioned this pull request Mar 10, 2020

collectd: 5.8.1 -> 5.10.0 #82253

Merged

10 tasks

wedens and others added 4 commits March 15, 2020 11:28

libvirt: 5.4.0 -> 6.0.0

24c8d75

libvirt: 6.0.0 -> 6.1.0, fix module

d8664c7

nixos/scripted-networking: fix bridge setup when libvirtd uses socket…

687aa06

… activation

perlPackages.SysVirt: use upstream tag

ddbac2e

wedens force-pushed the libvirt-6.0.0 branch from feb3eb5 to ddbac2e Compare March 15, 2020 04:31

libvirt: 5.4.0 -> 5.9.0

9336b15

globin self-assigned this Mar 28, 2020

worldofpeace added this to the 20.03 milestone Mar 29, 2020

libvirt: fix escapeShellArg usage

b80edca

Co-Authored-By: conferno <conferno@camfex.cz>

FRidh approved these changes Mar 30, 2020

View reviewed changes

globin merged commit d6fa642 into NixOS:master Mar 30, 2020

libvirt: 5.4.0 -> 6.1.0 #81161

libvirt: 5.4.0 -> 6.1.0 #81161

Conversation

wedens commented Feb 27, 2020

Motivation for this change

Things done

wedens commented Feb 27, 2020 • edited

Izorkin commented Feb 27, 2020

wedens commented Feb 27, 2020 • edited

Izorkin commented Feb 27, 2020

Izorkin commented Feb 27, 2020

jonringer commented Feb 27, 2020

Izorkin commented Feb 27, 2020 • edited

wedens commented Feb 27, 2020

wedens commented Feb 27, 2020 • edited

Izorkin commented Feb 27, 2020

Izorkin commented Feb 27, 2020

Izorkin commented Feb 27, 2020

wedens commented Feb 27, 2020 • edited

jonringer commented Feb 27, 2020

wedens commented Feb 28, 2020 • edited

jonringer commented Feb 28, 2020

wedens commented Feb 28, 2020

jonringer commented Feb 28, 2020

wedens commented Feb 28, 2020 • edited

wedens commented Mar 6, 2020

Izorkin commented Mar 6, 2020 • edited

Izorkin commented Mar 6, 2020

Izorkin commented Mar 9, 2020

nixos-discourse commented Mar 9, 2020

mmilata commented Mar 11, 2020

jonringer commented Mar 14, 2020

wedens commented Mar 15, 2020

wedens commented Mar 15, 2020

Izorkin commented Mar 15, 2020 • edited

wedens commented Mar 15, 2020

wedens commented Mar 24, 2020

nh2 commented Mar 28, 2020

globin commented Mar 28, 2020

worldofpeace commented Mar 29, 2020

FRidh left a comment

Choose a reason for hiding this comment

wedens commented Feb 27, 2020 •

edited

wedens commented Feb 27, 2020 •

edited

Izorkin commented Feb 27, 2020 •

edited

wedens commented Feb 27, 2020 •

edited

wedens commented Feb 27, 2020 •

edited

wedens commented Feb 28, 2020 •

edited

wedens commented Feb 28, 2020 •

edited

Izorkin commented Mar 6, 2020 •

edited

Izorkin commented Mar 15, 2020 •

edited