WIP: Python packages cleanup #30919

adisbladis · 2017-10-29T08:02:41Z

Motivation for this change

pythonPackages is a mess and needs lots of cleanup and removal of unused unmaintained packages.
This is an attempt at starting the effort of cleaning things up.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

cc @domenkozar @bjornfor @lovek323 @nlewo @cillianderoiste

FRidh

Can you cc the maintainers of the packages that are involved.

FRidh · 2017-10-29T09:16:23Z

pkgs/applications/networking/instant-messengers/salut-a-toi/default.nix

@@ -1,7 +1,12 @@
-{stdenv, fetchurl, python27Packages, file }:
+{ stdenv, fetchurl, python27Packages, file, pkgs }:


Don't use pypi2nix in 5666dff703372a9aaa0ec5fe03a2edd713f9f4ee.

I did not use pypi2nix for any of this.

Right, I see. In any case, functions should preferably have exact arguments instead of package sets.

FRidh · 2017-10-29T09:18:13Z

pkgs/applications/networking/pyload/default.nix

+, pkgs }:
+
+let
+  beautifulsoup = pythonPackages.callPackage ./beautifulsoup.nix {


Forgot to include this file?

Yes I did. Will update.

bjornfor

bitstring isn't dead, it just needs URL update (and version bump).

UPDATE: Other non-dead packages (just need update): demjson, pyodbc, robotsuite.

adisbladis · 2017-10-29T09:46:32Z

@bjornfor I know a bunch of the packages are not dead but since no one is maintaining them in nixpkgs I figure that they do more harm than good.

I started this by doing a git blame | grep sha256 and starting from the oldest packages.
This is all stuff that hasn't been updated since early 2014 and older.

I see no point in keeping around packages that no one has the intention of maintaining.

bjornfor · 2017-10-29T10:27:37Z

I have updated bitstring, demjson, pyodbc, robotsuite in master (0b93dbe and parents). Please rebase.

bjornfor

I've updated bitstring, demjson, pyodbc, robotsuite in master, so you can drop them from this PR.

adisbladis · 2017-10-29T10:30:31Z

@bjornfor Please also split them out from python-packages.nix.

No longer applies.

bjornfor · 2017-10-29T10:55:43Z

@adisbladis: Done (489cdc2 and 3 parents).

FRidh · 2017-10-29T10:57:47Z

Thank you @bjornfor

adisbladis · 2017-10-29T11:08:24Z

@bjornfor Thanks :) Packages dropped from PR.

bjornfor · 2017-10-29T11:26:49Z

@adisbladis:

@bjornfor I know a bunch of the packages are not dead but since no one is maintaining them in nixpkgs I figure that they do more harm than good.

Harm to whom? Nixpkgs users or maintainers?

adisbladis · 2017-10-29T12:14:53Z

@bjornfor I have a few reasons for this.

It's harmful to users. if I'm on the unstable channel I would expect to get the latest stable release of a given package, not something that is 3-7 years out of date.
It's harmful for security. Lagging several years behind on security updates can never be a good thing.
It's harmful for maintainers. A lot of the python packages are abandoned by their maintainer. You yourself is listed as a maintainer on 25 python packages, another maintainer with a bunch of abandoned packages are listed on 65 python packages. Can you honestly say that you are maintaining all of these?
It's harmful to new contributors. I have on multiple occasions tried to improve and update packages with python dependencies and ended up giving up on the endeavour because it turns out every single dependency is horribly out of date. I'm pretty sure I'm not the only one.

FRidh · 2017-10-29T13:01:40Z

@adisbladis I mostly agree with you, but I do want to add the following. Many of us that have been using Nixpkgs for some time are listed as maintainers for a lot of packages. That's mostly because we have (or had) a need for that package, and simply want to be warned when they are touched. Ideally, each of us would maintain fewer packages, but as is I think obvious, we just do not have that many maintainers, but still we have a need of the packages.

bjornfor · 2017-10-29T13:08:45Z

@adisbladis:

It's harmful to users. if I'm on the unstable channel I would expect to get the latest stable release of a given package, not something that is 3-7 years out of date.

The alternative is not having the package at all. (I'd rather have old package than no package.)

It's harmful for security. Lagging several years behind on security updates can never be a good thing.

Good point.

It's harmful for maintainers. A lot of the python packages are abandoned by their maintainer. You yourself is listed as a maintainer on 25 python packages, another maintainer with a bunch of abandoned packages are listed on 65 python packages. Can you honestly say that you are maintaining all of these?

(That doesn't explain how it's harmful to maintainers. I think it's harmful when there are packages that require maintenance, but nobody really cares about the packages anymore. Then it is better to remove, instead of "forcing" people to spend energy on things that don't matter.)

No, I cannot claim that I maintain all my packages, if that means always having the latest version (unfortunately). I could remove myself as maintainer and/or remove all "my" packages if they lack maintainers, but I don't see that as a good solution either.

It's harmful to new contributors. I have on multiple occasions tried to improve and update packages with python dependencies and ended up giving up on the endeavour because it turns out every single dependency is horribly out of date. I'm pretty sure I'm not the only one.

But the alternative would be not having any dependency packages to begin with. Isn't it better to update a few old packages when needed than having to write all expressions from scratch?

To sum up, I think it's good that nixpkgs receives some "clean-up" from time to time (thanks!), but we must also find a balance in how eager we are when cleaning/removing stuff. Being too eager or too lazy is both bad.

adisbladis · 2017-10-29T13:11:00Z

Since bpython is well underway in #30922 I have dropped it from the PR.

bjornfor · 2017-10-29T22:00:38Z

pkgs/development/python-modules/python-fuse/default.nix

@@ -0,0 +1,25 @@
+{ lib
+, pkgs
+, pythonPackages


I think it's better to have explicit dependencies, so please replace "pkgs" and "pythonPackages" with specific packages. (Unless there is a new style going on in nixpkgs python packaging that I've missed.)

From upstream: This functionality provided by this module is now part of mechanize. I don't intend to make further standalone releases of ClientForm.

…dependent packages

…ackages into salut-a-toi derivation

… upstream

…es.beautifulsoup

…on by any packages

orivej · 2017-11-07T01:10:26Z

I only noticed this PR when it was merged: it was not ready.

This merge has broken the evaluation of master (for the purpose of nox-review and Travis, and hydra advancing the channel), fixed in pythonPackages.fuse: fix infinite recursion in expression #31340.
RedNotebook is a diary application, its users will not be reconciled with its sudden removal, and its old version does not show its age and works perfectly well. I'm restoring and updating it in Restore and update RedNotebook #31341.
I do not understand or appreciate this approach to packages (particularly applications) that build fine, and just were not updated in a few years. First, they may have satisfied users who did not notice anything lacking and do not care to update; or the old version may even work better than the new. Second, it is often easier to update existing packages than to add new ones. (For instance you've removed Task Coach 1.3.22 when the latest version is 1.4.3. I guess that it has not changed much, and that its old expression is useful to package the update.)

FRidh · 2017-11-07T07:34:32Z

@orivej Ideally we have expressions, and preferably of the latest versions, but that requires maintainers. We just don't have enough people maintaining all of the expressions.

adisbladis requested a review from FRidh as a code owner October 29, 2017 08:02

FRidh requested changes Oct 29, 2017

View reviewed changes

FRidh added the 6.topic: python label Oct 29, 2017

FRidh self-assigned this Oct 29, 2017

FRidh added the 6.topic: hygiene label Oct 29, 2017

bjornfor previously requested changes Oct 29, 2017

View reviewed changes

adisbladis force-pushed the pythonPackages-cleanup branch 2 times, most recently from c3407f4 to 0211bde Compare October 29, 2017 09:56

bjornfor reviewed Oct 29, 2017

View reviewed changes

adisbladis force-pushed the pythonPackages-cleanup branch from 0211bde to 2747260 Compare October 29, 2017 11:06

adisbladis force-pushed the pythonPackages-cleanup branch 2 times, most recently from 8885eb6 to 37f5dc0 Compare October 29, 2017 13:08

bjornfor reviewed Oct 29, 2017

View reviewed changes

adisbladis added 5 commits October 30, 2017 11:55

pythonPackages.clientform: Remove package

b831d38

From upstream: This functionality provided by this module is now part of mechanize. I don't intend to make further standalone releases of ClientForm.

pythonPackages.skype4py: Remove package, not updated in years and no …

e78864e

…dependent packages

pythonPackages.pyxattr: 0.5.1 -> 0.6.0

892a000

pythonPackages.fuse: Move to python-modules

286ca78

pythonPackages.pylibacl: 0.5.1 -> 0.5.3

552922c

adisbladis added 21 commits October 30, 2017 11:55

salut-a-toi: Move pythonPackages dependencies not required by other p…

49a1221

…ackages into salut-a-toi derivation

pythonpackages.pymacs: Remove package, does not seem to be maintained…

88683da

… upstream

pythonpackages.ropemode: Remove outdated package, far behind upstream

d1144c0

pythonpackages.ropemacs: Remove outdated package, far behind upstream

b9b8efc

pythonpackages.taskcoach: Remove outdated package, far behind upstream

8b8dd09

pythonpackages.ledger-autosync: Remove outdated package

e86b3a0

anki: Depend on pythonPackages.beautifulsoup4 instead of pythonPackag…

e87ffae

…es.beautifulsoup

pythonpackages.python_tvrage: Remove dead package

12d8140

pythonPackages.beautifulsoup: Move into pyload, no longer depended up…

bb50ae9

…on by any packages

pythonpackages.rtmidi: Remove outdated package

ab66d6e

pythonpackages.doxypy: Remove outdated package

1b55fc5

pythonpackages.requests_oauth2: Remove dead package

495b099

pythonpackages.googlecl: Remove outdated package (source url dead)

3fcf9bc

pythonpackages.deform_bootstrap: Remove seemingly dead package

2ca086d

pythonpackages.trac: Remove outdated package

9383698

pythonpackages.redNotebook: Remove outdated package

33a56ef

pythonpackages.rtslib_fb: Remove outdated package

2170a73

pythonpackages.targetcli_fb: Remove outdated package

cb7caad

pythonpackages.configshell_fb: Remove outdated package

7d6a912

pythonpackages.nose-selecttests: Remove outdated package

b2de5a9

pythonpackages.pyramid_debugtoolbar: Remove outdated package

98024e2

adisbladis force-pushed the pythonPackages-cleanup branch from 37f5dc0 to 98024e2 Compare October 30, 2017 03:58

FRidh merged commit e734ecb into NixOS:master Nov 6, 2017

This was referenced Nov 6, 2017

pythonPackages.fuse: fix infinite recursion in expression #31340

Merged

Restore and update RedNotebook #31341

Merged

ryantm mentioned this pull request Jun 14, 2018

Ledger-autosync is missing #41836

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Python packages cleanup #30919

WIP: Python packages cleanup #30919

adisbladis commented Oct 29, 2017 •

edited

FRidh left a comment

FRidh Oct 29, 2017

adisbladis Oct 29, 2017

FRidh Oct 29, 2017

FRidh Oct 29, 2017

adisbladis Oct 29, 2017

bjornfor left a comment •

edited

adisbladis commented Oct 29, 2017

bjornfor commented Oct 29, 2017

bjornfor left a comment

adisbladis commented Oct 29, 2017

bjornfor commented Oct 29, 2017

FRidh commented Oct 29, 2017

adisbladis commented Oct 29, 2017

bjornfor commented Oct 29, 2017

adisbladis commented Oct 29, 2017

FRidh commented Oct 29, 2017

bjornfor commented Oct 29, 2017

adisbladis commented Oct 29, 2017

bjornfor Oct 29, 2017

orivej commented Nov 7, 2017

FRidh commented Nov 7, 2017

		@@ -1,7 +1,12 @@
		{stdenv, fetchurl, python27Packages, file }:
		{ stdenv, fetchurl, python27Packages, file, pkgs }:

WIP: Python packages cleanup #30919

WIP: Python packages cleanup #30919

Conversation

adisbladis commented Oct 29, 2017 • edited

Motivation for this change

Things done

FRidh left a comment

Choose a reason for hiding this comment

FRidh Oct 29, 2017

Choose a reason for hiding this comment

adisbladis Oct 29, 2017

Choose a reason for hiding this comment

FRidh Oct 29, 2017

Choose a reason for hiding this comment

FRidh Oct 29, 2017

Choose a reason for hiding this comment

adisbladis Oct 29, 2017

Choose a reason for hiding this comment

bjornfor left a comment • edited

Choose a reason for hiding this comment

adisbladis commented Oct 29, 2017

bjornfor commented Oct 29, 2017

bjornfor left a comment

Choose a reason for hiding this comment

adisbladis commented Oct 29, 2017

bjornfor commented Oct 29, 2017

FRidh commented Oct 29, 2017

adisbladis commented Oct 29, 2017

bjornfor commented Oct 29, 2017

adisbladis commented Oct 29, 2017

FRidh commented Oct 29, 2017

bjornfor commented Oct 29, 2017

adisbladis commented Oct 29, 2017

bjornfor Oct 29, 2017

Choose a reason for hiding this comment

orivej commented Nov 7, 2017

FRidh commented Nov 7, 2017

adisbladis commented Oct 29, 2017 •

edited

bjornfor left a comment •

edited