Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor another ~100 package from python-packages.nix -> python-modules #48633

Merged
merged 111 commits into from Oct 18, 2018

Conversation

costrouc
Copy link
Member

@costrouc costrouc commented Oct 17, 2018

Things done

All have been tested to build and is ready for merge

Strictly moving package to python-modules. Did not attempt to upgrade package unless it added python 3 support or was broken. Also converted fetch... to fetchPypi to make future upgrades easier. All packages have been tested in sandbox mode to build properly.

As of the last edit there are ~332 remaining packages to move.

grep "buildPythonPackage" python-packages.nix | wc
    332    1706   13716
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@lheckemann
Copy link
Member

Sorry, I broke this by splitting slixmpp in #48545

@FRidh
Copy link
Member

FRidh commented Oct 17, 2018

e08ab90 should be squashed

@costrouc costrouc force-pushed the costrouc/refactor-to-python-modules branch from 0b053e7 to d627cbc Compare October 17, 2018 16:39
@costrouc
Copy link
Member Author

@lheckemann no worries I will rebase it. I appreciate you letting me know. @FRidh my git fu is not quite good enough and I was being lazy when I made those commits. I will learn how to properly split that commit and merge into the appropriate commit.

@costrouc costrouc force-pushed the costrouc/refactor-to-python-modules branch 3 times, most recently from b463f57 to 219bd0f Compare October 17, 2018 17:24
@Mic92
Copy link
Member

Mic92 commented Oct 17, 2018

Good job.

@costrouc
Copy link
Member Author

costrouc commented Oct 17, 2018

25 more packages -> python-modules. All tested.

@ryantm
Copy link
Member

ryantm commented Oct 24, 2018

@costrouc Thank you for this work! @FRidh is encouraging me to start updating Python packages semi-automatically with r-ryantm/nixpkgs-update and it cannot update the packages in python-modules.nix (because it won't update things in a file with multiple fetchers), so this work you are doing should help keep Python packages more up to date!

@costrouc
Copy link
Member Author

You are welcome! I was aware of your bot and someone told me that it could only work with fetchPypi and being in python-modules. To me it seems like automation is an important step in the growth of nix. Otherwise small package updates will consume too much of our time. I have a bit more free time for the next month and my goal is to really cleanup the python packages in nix.

@costrouc
Copy link
Member Author

The good news is that soon (next few years) python projects will start declaring their dependencies in a static file pyproject.toml. This will make life much easier for updating packages.

@Mic92
Copy link
Member

Mic92 commented Oct 24, 2018

pyproject.toml will probably still need some time to establish. At the moment I don't see how c-extension can be build using this.

@ryantm
Copy link
Member

ryantm commented Oct 24, 2018

As far as I know, nixpkgs-update doesn't need the derivation to use fetchPypi. In fact, here is one using fetchFromGitHub: https://github.com/NixOS/nixpkgs/pull/49037/files

@costrouc
Copy link
Member Author

@ryantm what fetch... does it work with? In some cases packages released through pypi do not contain tests which make upgrades more dangerous (not knowing if something breaks). Makes me want to favor fetchFromGitHub, fetchFromGitLab, etc. What are your thoughts?

@ryantm
Copy link
Member

ryantm commented Oct 24, 2018

@costrouc As long as the fetcher is written in some way that it gets modified by changing the version string somewhere in the file, and the hash for it is somewhere in the file, and nix-prefetch-url works, it should work. So, all the main fetchers work, but sometimes expressions are formatted in a way that doesn't works, for example, the rev might not be based on the version, or fetchgit is using the commit sha.

As for what to do for python things, I want to defer to @FRidh. I'm not familiar with the Python ecosystem.

@FRidh
Copy link
Member

FRidh commented Oct 24, 2018

pyproject.toml will probably still need some time to establish. At the moment I don't see how c-extension can be build using this.

pyproject.toml will specify that setuptools will be used as build system, and from there on everything is the same as it is now. It's a step in the right direction, giving access to dependencies in a easier way. However, we still have to do the hardest part: dependency resolution across the whole package set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants