Another batch of Python 2 removals #75668

jtojnar · 2019-12-14T22:20:22Z

I was digging through gnome3.gnome-shell’s build-time closure using nix-store -q --graph $(nix-instantiate '<nixpkgs>' -A gnome3.gnome-shell) | dijkstra -da $(nix-instantiate '<nixpkgs>' -A python2) | gvpr -c 'N[dist>1000.0]{delete(NULL, $)}' | xdot - and eliminating Python 2 from derivations referencing it by replacing it with Python 3 whenever possible.

Testing

Some packages were not handled:

asciidoc (asciidoctor in Ruby is the new upstream, switch to https://github.com/asciidoc/asciidoc-py3/releases)
llvm_7, llvm_9 (not sure about bootstrapping)
ninja (Darwin bootstrapping issue https://gist.github.com/GrahamcOfBorg/130aa188dad522a3e0df7ce27006565e)
parted (two tests are failing)
spidermonkey_60 (takes too long to build)
qtbase (takes too long to build)
libplist (does not build with Python 3)
telepathy-glib (telepathy-logger depends on Python 2)

It was never used in the recent history.

pkgs/development/python-modules/anytree/default.nix

* format with nixpkgs-fmt * drop unneeded dependencies * gnome-doc-utils → yelp-tools since 3.6.0 * intltool → gettext since 3.16.2 * move build-time deps to nativeBuildInputs * add metadata

pkgs/servers/x11/xorg/default.nix

worldofpeace · 2019-12-15T00:05:37Z

pkgs/top-level/all-packages.nix

-  libselinux = callPackage ../os-specific/linux/libselinux { };
+  libselinux = callPackage ../os-specific/linux/libselinux {
+    python = python3;
+  };


could we do this, but in the expression?
people could override python3 = python?

The reason I did not do that is because it is overridden in python-packages.nix and it felt weird having python3 = python there.

worldofpeace · 2019-12-15T00:05:58Z

pkgs/top-level/all-packages.nix

-  libsemanage = callPackage ../os-specific/linux/libsemanage { };
+  libsemanage = callPackage ../os-specific/linux/libsemanage {
+    python = python3;
+  };


can we do this in the expression?
python references replaced to python3.

for most usages of python outside of python-modules, you generally want to parameterize python like this incase you need to override package versions, or if you want to test with a different version of python

worldofpeace · 2019-12-15T00:10:17Z

I was digging through gnome3.gnome-shell’s build-time closure using nix-store -q --graph $(nix-instantiate '' -A gnome3.gnome-shell) | dijkstra -da $(nix-instantiate '' -A python2) | gvpr -c 'N[dist>1000.0]{delete(NULL, $)}' | xdot - and eliminating Python 2 from derivations referencing it by replacing it with Python 3 whenever possible.

The graph is truly crazy to look at 😄

jtojnar · 2019-12-15T00:24:09Z

Good thing you can just python3 into stuff that does not build with it and it will disappear:

diff --git a/pkgs/desktops/gnome-3/core/gnome-shell/default.nix b/pkgs/desktops/gnome-3/core/gnome-shell/default.nix
index e8d5b7dbffd..64ffeadf93e 100644
--- a/pkgs/desktops/gnome-3/core/gnome-shell/default.nix
+++ b/pkgs/desktops/gnome-3/core/gnome-shell/default.nix
@@ -6,7 +6,7 @@
 , accountsservice, gdk-pixbuf, gdm, upower, ibus, networkmanagerapplet, libgnomekbd, gnome-desktop
 , gsettings-desktop-schemas, gnome-keyring, glib, gjs, mutter, evolution-data-server, gtk3
 , sassc, systemd, gst_all_1, adwaita-icon-theme, gnome-bluetooth, gnome-clocks, gnome-settings-daemon
-, gnome-autoar, asciidoc-full }:
+, gnome-autoar, asciidoc }:
 
 # http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/gnome-base/gnome-shell/gnome-shell-3.10.2.1.ebuild?revision=1.3&view=markup
 
@@ -26,7 +26,7 @@ in stdenv.mkDerivation rec {
 
   nativeBuildInputs = [
     meson ninja pkgconfig gettext docbook_xsl docbook_xsl_ns docbook_xml_dtd_42 perl wrapGAppsHook glibcLocales
-    sassc desktop-file-utils libxslt.bin python3 asciidoc-full
+    sassc desktop-file-utils libxslt.bin python3 asciidoc
   ];
   buildInputs = [
     systemd
diff --git a/pkgs/development/compilers/llvm/9/default.nix b/pkgs/development/compilers/llvm/9/default.nix
index 1590d180c3f..cfe88b5d215 100644
--- a/pkgs/development/compilers/llvm/9/default.nix
+++ b/pkgs/development/compilers/llvm/9/default.nix
@@ -42,12 +42,12 @@ let
 
     llvm-manpages = lowPrio (tools.llvm.override {
       enableManpages = true;
-      python = pkgs.python;  # don't use python-boot
+      python = pkgs.python3;  # don't use python-boot
     });
 
     clang-manpages = lowPrio (tools.clang-unwrapped.override {
       enableManpages = true;
-      python = pkgs.python;  # don't use python-boot
+      python = pkgs.python3;  # don't use python-boot
     });
 
     libclang = tools.clang-unwrapped.lib;
diff --git a/pkgs/development/compilers/rust/cargo.nix b/pkgs/development/compilers/rust/cargo.nix
index 91a31be9f8d..f639d330566 100644
--- a/pkgs/development/compilers/rust/cargo.nix
+++ b/pkgs/development/compilers/rust/cargo.nix
@@ -1,4 +1,4 @@
-{ stdenv, file, curl, pkgconfig, python, openssl, cmake, zlib
+{ stdenv, file, curl, pkgconfig, python3, openssl, cmake, zlib
 , makeWrapper, libiconv, cacert, rustPlatform, rustc, libgit2
 , CoreFoundation, Security
 }:
@@ -18,7 +18,7 @@ rustPlatform.buildRustPackage {
   dontUpdateAutotoolsGnuConfigScripts = true;
 
   nativeBuildInputs = [ pkgconfig cmake makeWrapper ];
-  buildInputs = [ cacert file curl python openssl zlib libgit2 ]
+  buildInputs = [ cacert file curl python3 openssl zlib libgit2 ]
     ++ stdenv.lib.optionals stdenv.isDarwin [ CoreFoundation Security libiconv ];
 
   LIBGIT2_SYS_USE_PKG_CONFIG = 1;
diff --git a/pkgs/development/compilers/rust/rustc.nix b/pkgs/development/compilers/rust/rustc.nix
index ca9bf05b7e7..49eaeef0d9c 100644
--- a/pkgs/development/compilers/rust/rustc.nix
+++ b/pkgs/development/compilers/rust/rustc.nix
@@ -1,5 +1,5 @@
 { stdenv, removeReferencesTo, pkgsBuildBuild, pkgsBuildHost, pkgsBuildTarget
-, fetchurl, file, python2
+, fetchurl, file, python3
 , llvm_9, darwin, git, cmake, rust, rustPlatform
 , pkgconfig, openssl
 , which, libffi
@@ -121,7 +121,7 @@ in stdenv.mkDerivation rec {
   dontUseCmakeConfigure = true;
 
   nativeBuildInputs = [
-    file python2 rustPlatform.rust.rustc git cmake
+    file python3 rustPlatform.rust.rustc git cmake
     which libffi removeReferencesTo pkgconfig
   ];
 
diff --git a/pkgs/development/libraries/libplist/default.nix b/pkgs/development/libraries/libplist/default.nix
index 19b700ca5e3..948a8dfefcc 100644
--- a/pkgs/development/libraries/libplist/default.nix
+++ b/pkgs/development/libraries/libplist/default.nix
@@ -1,7 +1,7 @@
-{ stdenv,  autoreconfHook, fetchFromGitHub, pkgconfig, python2Packages, glib }:
+{ stdenv,  autoreconfHook, fetchFromGitHub, pkgconfig, python3Packages, glib }:
 
 let
-  inherit (python2Packages) python cython;
+  inherit (python3Packages) python cython;
 in
 stdenv.mkDerivation rec {
   pname = "libplist";
diff --git a/pkgs/development/tools/build-managers/wafHook/default.nix b/pkgs/development/tools/build-managers/wafHook/default.nix
index 4184bd5fe8c..d33abb88a69 100644
--- a/pkgs/development/tools/build-managers/wafHook/default.nix
+++ b/pkgs/development/tools/build-managers/wafHook/default.nix
@@ -1,7 +1,7 @@
-{ lib, stdenv, pkgs, python, makeSetupHook, waf }:
+{ lib, stdenv, pkgs, python3, makeSetupHook, waf }:
 
 makeSetupHook {
-  deps = [ python ];
+  deps = [ python3 ];
   substitutions = {
     inherit waf;
     crossFlags = lib.optionalString (stdenv.hostPlatform.system != stdenv.targetPlatform.system)
diff --git a/pkgs/tools/typesetting/asciidoc/default.nix b/pkgs/tools/typesetting/asciidoc/default.nix
index 83aa667f58f..98546fd9209 100644
--- a/pkgs/tools/typesetting/asciidoc/default.nix
+++ b/pkgs/tools/typesetting/asciidoc/default.nix
@@ -1,4 +1,4 @@
-{ fetchurl, stdenv, python2
+{ fetchurl, stdenv, python3
 
 , enableStandardFeatures ? false
 , sourceHighlight ? null
@@ -151,7 +151,7 @@ stdenv.mkDerivation rec {
     sha256 = "1w71nk527lq504njmaf0vzr93pgahkgzzxzglrq6bay8cw2rvnvq";
   };
 
-  buildInputs = [ python2 unzip ];
+  buildInputs = [ python3 unzip ];
 
   # install filters early, so their shebangs are patched too
   patchPhase = with stdenv.lib; ''
@@ -251,7 +251,7 @@ stdenv.mkDerivation rec {
         -i a2x.py
   '' + ''
     for n in $(find "$out" . -name \*.py); do
-      sed -i -e "s,^#![[:space:]]*.*/bin/env python,#!${python2}/bin/python,g" "$n"
+      sed -i -e "s,^#![[:space:]]*.*/bin/env python,#!${python3}/bin/python,g" "$n"
       chmod +x "$n"
     done
 
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 36f02e50be2..8cbf0b86533 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -8479,6 +8479,7 @@ in
 
   llvmPackages_9 = callPackage ../development/compilers/llvm/9 ({
     inherit (stdenvAdapters) overrideCC;
+    python = python3;
     buildLlvmTools = buildPackages.llvmPackages_9.tools;
     targetLlvmLibraries = targetPackages.llvmPackages_9.libraries;
   } // stdenv.lib.optionalAttrs (buildPackages.stdenv.cc.isGNU && stdenv.hostPlatform.isi686) {

We are dropping Python support from gamin so nothing is holding us any more. Instead of gamin, we will use systemd on Linux and polling on Darwin; gamin is the worst backend any way, according to upstream: fail2ban/fail2ban#1330 (comment)

They are not compatible with Python 3, upstream as dead, and nothing uses the bindings any more. This is a progress in getting rid of Python 2 from closures of most people (samba, which depends on gamin shared library is common dependency).

Python is only used for a single console test these dayls and it likely requires non-trivial introspection hacks.

Also move the build-time dependencies to nativeBuildInputs, and drop perl as it no longer seems to be necessary (CMake is not looking for it any more).

Nothing was using them. Another step in Python 2 removal.

jtojnar · 2019-12-15T13:26:30Z

Successfully built rustc and cargo with Python 3 overnight.

LnL7 · 2019-12-15T14:01:39Z

@jtojnar The darwin stdenv (llvm, ninja, etc.) depends on python to bootstrap so switching to python3 for those requires a bit of fiddling with the stdenv stages. Perhaps the next llvm 7 -> 9 upgrade which I've started to look into would be a good opportunity to also tackle python3.

worldofpeace · 2019-12-15T22:55:03Z

@GrahamcOfBorg build glib

(darwin gtk-doc sometimes is...)

jtojnar added 3 commits December 14, 2019 02:41

farstream: format with nixpkgs-fmt

1094113

farstream: remove python2 dependency

9c18e1f

It was never used in the recent history.

libevdev: switch to python3

f78ac78

jtojnar requested review from FRidh and jonringer as code owners December 14, 2019 22:20

jtojnar changed the title ~~Another batch of Python 2 removal~~ Another batch of Python 2 removals Dec 14, 2019

FRidh reviewed Dec 14, 2019

View reviewed changes

pkgs/development/python-modules/anytree/default.nix Show resolved Hide resolved

jtojnar force-pushed the py2-cleanup branch from 85086e1 to 08e4f1b Compare December 14, 2019 22:23

ofborg bot added 6.topic: GNOME GNOME desktop environment and its underlying platform 6.topic: python 8.has: documentation labels Dec 14, 2019

jtojnar added 3 commits December 14, 2019 23:34

python3.pkgs.anytree: 2.7.2 → 2.7.3

03423da

zenity: clean up

54a75e9

* format with nixpkgs-fmt * drop unneeded dependencies * gnome-doc-utils → yelp-tools since 3.6.0 * intltool → gettext since 3.16.2 * move build-time deps to nativeBuildInputs * add metadata

libdbusmenu-gtk3: do not depend on python2 (via gnome-doc-utils)

d58a42a

jtojnar force-pushed the py2-cleanup branch 2 times, most recently from 215b3f2 to c8598bb Compare December 14, 2019 22:50

ofborg bot added 8.has: package (new) 10.rebuild-darwin: 501+ 10.rebuild-darwin: 2501-5000 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Dec 14, 2019

worldofpeace reviewed Dec 14, 2019

View reviewed changes

pkgs/servers/x11/xorg/default.nix Outdated Show resolved Hide resolved

worldofpeace reviewed Dec 14, 2019

View reviewed changes

pkgs/servers/x11/xorg/default.nix Outdated Show resolved Hide resolved

worldofpeace added this to Needs review in Python 2 deprecation Dec 15, 2019

worldofpeace reviewed Dec 15, 2019

View reviewed changes

jtojnar added 2 commits December 15, 2019 01:50

xorg.xcbproto: do not depend on python2

eb1d315

cacert: switch to python3

3a8d826

jtojnar added 13 commits December 15, 2019 01:50

libgit2: switch to python3

c8490a8

serd: switch to python3

486e3ec

sord: switch to python3

d06be85

sratom: switch to python3

e942647

lv2: switch to python3

6b5e932

lilv: switch to python3

a472105

fail2ban: switch to Python 3

7adbfb1

We are dropping Python support from gamin so nothing is holding us any more. Instead of gamin, we will use systemd on Linux and polling on Darwin; gamin is the worst backend any way, according to upstream: fail2ban/fail2ban#1330 (comment)

gamin: drop Python bindings

9d9a20f

They are not compatible with Python 3, upstream as dead, and nothing uses the bindings any more. This is a progress in getting rid of Python 2 from closures of most people (samba, which depends on gamin shared library is common dependency).

gst_all_1.gst-plugins-good: switch to python 3

feb4e68

libgsf: remove Python dependency

908db05

Python is only used for a single console test these dayls and it likely requires non-trivial introspection hacks.

doxygen: switch to python3

239e4c1

Also move the build-time dependencies to nativeBuildInputs, and drop perl as it no longer seems to be necessary (CMake is not looking for it any more).

jbig2dec: switch to python3

628abfe

libproxy: remove Python 2 bidnings

f288a7c

Nothing was using them. Another step in Python 2 removal.

jtojnar force-pushed the py2-cleanup branch from c8598bb to f288a7c Compare December 15, 2019 00:51

jtojnar added 2 commits December 15, 2019 13:52

cargo: switch to python3

08c0f12

rustc: switch to python3

628ff8d

jtojnar requested review from LnL7 and Mic92 as code owners December 15, 2019 12:53

ofborg bot added the 6.topic: rust label Dec 15, 2019

FRidh added this to Done in Staging Dec 15, 2019

FRidh moved this from Done to Needs review in Staging Dec 15, 2019

jtojnar merged commit b171037 into NixOS:staging Dec 19, 2019

Staging automation moved this from Needs review to Done Dec 19, 2019

Python 2 deprecation automation moved this from Needs review to Done Dec 19, 2019

jtojnar deleted the py2-cleanup branch December 19, 2019 00:24

jtojnar mentioned this pull request Dec 25, 2019

Another batch of Python 2 clean-ups #76488

Merged

17 tasks

samueldr mentioned this pull request Jan 13, 2020

libselinux: Fix cross-compilation with Python 3 #77596

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Another batch of Python 2 removals #75668

Another batch of Python 2 removals #75668

jtojnar commented Dec 14, 2019 •

edited

worldofpeace Dec 15, 2019

jtojnar Dec 15, 2019

worldofpeace Dec 15, 2019 •

edited

jonringer Dec 16, 2019

worldofpeace commented Dec 15, 2019

jtojnar commented Dec 15, 2019

jtojnar commented Dec 15, 2019

LnL7 commented Dec 15, 2019

worldofpeace commented Dec 15, 2019

Another batch of Python 2 removals #75668

Another batch of Python 2 removals #75668

Conversation

jtojnar commented Dec 14, 2019 • edited

Testing

worldofpeace Dec 15, 2019

Choose a reason for hiding this comment

jtojnar Dec 15, 2019

Choose a reason for hiding this comment

worldofpeace Dec 15, 2019 • edited

Choose a reason for hiding this comment

jonringer Dec 16, 2019

Choose a reason for hiding this comment

worldofpeace commented Dec 15, 2019

jtojnar commented Dec 15, 2019

jtojnar commented Dec 15, 2019

LnL7 commented Dec 15, 2019

worldofpeace commented Dec 15, 2019

jtojnar commented Dec 14, 2019 •

edited

worldofpeace Dec 15, 2019 •

edited