Add python packages: pyensembl and its dependencies, gffutils #89213

moritzschaefer · 2020-05-30T12:22:28Z

Motivation for this change

Replaces PR #87551

Both pyenseml and gffutils are commonly used python packages (> 100 stars on GitHub), so I added them to nix packages.

Things done

Built on platform(s)
- NixOS
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
Tested execution of all binary files (usually in ./result/bin/) (only pyensembl has a supplemental binary)
Ensured that relevant documentation is up to date
Determined the impact on package closure size (by running nix path-info -S before and after)
[-] Mostly Fits CONTRIBUTING.md. I designed the derivations 1:1 to existing ones.

jonringer

please add yourself as a maintainer for all packages that you're adding.

please attempt to sort the package entries in python-packages

pkgs/development/python-modules/datacache/default.nix

jonringer · 2020-05-30T20:17:48Z

pkgs/development/python-modules/gffutils/default.nix

+
+  checkPhase = ''
+    # sh gffutils/test/data/download-large-annotation-files.sh
+    # nosetests


why no test suite?

Suggested change

# nosetests

# nosetests

The tests fail inside the nix environment

general consensus is to just ignore test cases that fail due to sandbox environment. But some test failures may be indications that the package is in an actual broken state.

tests are also very useful for verifying that dependency bumps don't cause regressions.

pkgs/development/python-modules/gtfparse/default.nix

pkgs/development/python-modules/scikit-plot/default.nix

pkgs/development/python-modules/serializable/default.nix

jonringer · 2020-05-30T20:20:15Z

pkgs/development/python-modules/tinytimer/default.nix

+  checkInputs = [ ];
+  propagatedBuildInputs = [ ];
+
+  checkPhase = ''
+    ${python.interpreter} -c 'import tinytimer'
+  '';
+
+  # Tests require extra dependencies
+  doCheck = true;
+


Suggested change

checkInputs = [ ];

propagatedBuildInputs = [ ];

checkPhase = ''

${python.interpreter} -c 'import tinytimer'

'';

# Tests require extra dependencies

doCheck = true;

pythonImportsCheck = [ "tinytimer" ];

jonringer · 2020-05-30T20:20:31Z

pkgs/top-level/python-packages.nix

+  typechecks = callPackage ../development/python-modules/typechecks { };
+
+  datacache = callPackage ../development/python-modules/datacache { };
+
+  memoized-property = callPackage ../development/python-modules/memoized-property { };
+
+  gtfparse = callPackage ../development/python-modules/gtfparse { };
+
+  serializable = callPackage ../development/python-modules/serializable { };
+
+  tinytimer = callPackage ../development/python-modules/tinytimer { };
+
+  pyensembl = callPackage ../development/python-modules/pyensembl { };
+
+  gffutils = callPackage ../development/python-modules/gffutils { };
+
+  scikit-plot = callPackage ../development/python-modules/scikit-plot { };
+
+  pybedtools = callPackage ../development/python-modules/pybedtools { };


please attempt to sort these

I don't get the sorting schema of this file. It appears to be somewhat alphabetically, but not as a whole but rather for a set regions within the file.

Wouldn't it make more sense to sort the file "entirely" from time to time?

it has been evolving over time... some people grouped similar packages, some people just dropped them anywhere. At the very least, random listings should be avoided

moritzschaefer · 2020-06-03T18:14:20Z

Thank you for your comments @jonringer! I have simplified all the files and added myself as maintainer. Please let me know if something still requires improvements.

moritzschaefer · 2020-06-03T22:47:48Z

@jonringer After having applied all the requested changes, I wanted to da a final rebase on master so I did

git checkout python-packages
git pull --rebase official master # official points to github.com/NIXOS/nixpgs.git

This resulted in a merge commit and now the PR is blown up with 370 commits. Is this an issue and if yes, would you mind telling me what went wrong?

jonringer · 2020-06-03T23:18:22Z

with --rebase, there shouldn't be a merge commit.

And it looks like you pulled in master. Assuming the merge is the main culprit, you could do git reset HEAD^ --hard but, I'm not sure exactly what you've done.

If all else failes, you could cherry pick the commits into a new branch and just push that

moritzschaefer · 2020-06-04T08:16:39Z

Indeed trashing the merge commit seemed to have fixed it. So no cherry-picking-in-new-branch required.

I noticed that two packages failed to build because of failing tests, even though checkPhase was not set (it seems that it is set to some default python test runner). I explicitly disabled tests for them using doCheck = false;

I'm anyways surprised that this wasn't detected by the @GrahamcOfBorg bot, which showed green for all tests.

jonringer · 2020-11-19T04:53:04Z

if you still want to get this into nixpkgs, you can break this up into smaller PRs, just adding a package at a time. It will be a lot less time intensive than having to cover all of it.

moritzschaefer · 2020-11-19T13:42:55Z

Dear @jonringer,

thanks for the response. I'll transition to adding these packages into my overlays and close this PR as there are definitely more important PRs (from me and others) for the moment.

Still, I am kind of missing some kind of strategy for pypi packages. In a conversation on the nixos-discourse, I got the notion that it is not desired to add the majority of pypi packages to nixpkgs. However it is really cumbersome to build a derivation for each single python package, when you're working in a non-mainstream field.

Is there some kind of vision for this issue?

jonringer · 2020-11-20T02:35:04Z

In general the preference is to have python packages available which enable an application, and less so to "just have the package".

However it is really cumbersome to build a derivation for each single python package, when you're working in a non-mainstream field.

You can use nix with venv to bring in missing python packages. I got a video about it https://www.youtube.com/watch?v=jXd-hkP4xnU , and some docs here: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems

If you still want a pure nix way to do it, I would look at mach-nixhttps://github.com/DavHau/mach-nix

moritzschaefer requested review from FRidh and jonringer as code owners May 30, 2020 12:22

moritzschaefer mentioned this pull request May 30, 2020

Add python packages: pyensembl and its dependencies, gffutils, pyBigWig #87551

Closed

6 tasks

ofborg bot added 6.topic: python 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels May 30, 2020

jonringer reviewed May 30, 2020

View reviewed changes

ofborg bot added the 11.by: package-maintainer label Jun 3, 2020

moritzschaefer added 10 commits June 4, 2020 00:38

Add pyensembl library (https://github.com/openvax/pyensembl)

6a4e019

Add gffutils python package

01211cc

scikit-plot: 0.3.7

259d879

pybedtools: 0.8.1

953a1cc

Add checks for python modules

33d7903

pybedtools: Add missing dependency

b199037

Add myself as maintainer to added packages; Declutter new nix files

34b5abe

sort new packages alphabetically

e35bf00

move scikit-plot to other scikit packages in python-packages.nix

8080373

Fix typos

32da240

moritzschaefer requested review from cdepillabout, edwtjo, Ericson2314, infinisil, tfc and thoughtpolice as code owners June 3, 2020 22:39

moritzschaefer force-pushed the python-packages branch from 7d8f1ba to 32da240 Compare June 4, 2020 07:54

moritzschaefer added 2 commits June 4, 2020 11:58

Disable failing tests

f8291a2

disable failing input test

65d3811

ryantm added 2.status: merge conflict and removed 2.status: merge conflict labels Oct 3, 2020

Merge branch 'master' into python-packages

b44370b

ofborg bot removed the 2.status: merge conflict label Nov 19, 2020

moritzschaefer closed this Nov 19, 2020

moritzschaefer mentioned this pull request Nov 28, 2020

pythonPackages.swifter: 0.304 #90046

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add python packages: pyensembl and its dependencies, gffutils #89213

Add python packages: pyensembl and its dependencies, gffutils #89213

moritzschaefer commented May 30, 2020 •

edited

jonringer left a comment

jonringer May 30, 2020

moritzschaefer Jun 3, 2020

jonringer Jun 3, 2020

jonringer May 30, 2020

jonringer May 30, 2020

moritzschaefer Jun 3, 2020

jonringer Jun 3, 2020

moritzschaefer commented Jun 3, 2020

moritzschaefer commented Jun 3, 2020

jonringer commented Jun 3, 2020

moritzschaefer commented Jun 4, 2020 •

edited

jonringer commented Nov 19, 2020

moritzschaefer commented Nov 19, 2020

jonringer commented Nov 20, 2020

Add python packages: pyensembl and its dependencies, gffutils #89213

Add python packages: pyensembl and its dependencies, gffutils #89213

Conversation

moritzschaefer commented May 30, 2020 • edited

Motivation for this change

Things done

jonringer left a comment

Choose a reason for hiding this comment

jonringer May 30, 2020

Choose a reason for hiding this comment

moritzschaefer Jun 3, 2020

Choose a reason for hiding this comment

jonringer Jun 3, 2020

Choose a reason for hiding this comment

jonringer May 30, 2020

Choose a reason for hiding this comment

jonringer May 30, 2020

Choose a reason for hiding this comment

moritzschaefer Jun 3, 2020

Choose a reason for hiding this comment

jonringer Jun 3, 2020

Choose a reason for hiding this comment

moritzschaefer commented Jun 3, 2020

moritzschaefer commented Jun 3, 2020

jonringer commented Jun 3, 2020

moritzschaefer commented Jun 4, 2020 • edited

jonringer commented Nov 19, 2020

moritzschaefer commented Nov 19, 2020

jonringer commented Nov 20, 2020

moritzschaefer commented May 30, 2020 •

edited

moritzschaefer commented Jun 4, 2020 •

edited