Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updating and adding all dependencies of future openstack sdk modules pull request #46531

Closed
wants to merge 37 commits into from
Closed

Updating and adding all dependencies of future openstack sdk modules pull request #46531

wants to merge 37 commits into from

Conversation

costrouc
Copy link
Member

@costrouc costrouc commented Sep 11, 2018 •
edited

Motivation for this change

I am trying to package the openstack python modules. oslo.config, oslo.db, and openstacksdk have ALOT of dependencies that are not currently packaged in nix. Along the way I have move all dependencies that were in python-packages.nix to python-modules/..... There are a lot of package in this commit (sorry and it there is any what that I can make this easier please let me know). This set of commits prepares all the dependencies for openstack modules.

I followed best practices. Testing was always used unless it created a circular dependency or was not packaged in the pypi repository. Otherwise tests were added to the package if it did not exist.

To help with code review here all all the package versions that need to be built with GrahmcOfBorg.

python27Packages.pathlib2
python27Packages.ddt
python27Packages.testrepository
python27Packages.enum34
python27Packages.eventlet
python27Packages.ipaddress
python27Packages.jmespath
python27Packages.mock
python27Packages.monotonic
python27Packages.munch
python27Packages.netaddr
python27Packages.netifaces
python27Packages.nose
python27Packages.oauthlib
python27Packages.warlock
python27Packages.fasteners
python27Packages.mox3
python27Packages.traceback2
python27Packages.linecache2
python27Packages.pep8
python27Packages.testscenarios
python27Packages.paramiko
python27Packages.prettytable
python27Packages.pyinotify
python27Packages.subunit
python27Packages.xattr
python27Packages.unittest2
python27Packages.unicodecsv
python27Packages.python-json-logger
python27Packages.daiquiri
python27Packages.python-coveralls
python27Packages.flake8-polyfill
python27Packages.flake8-docstrings
python27Packages.runtest
python27Packages.hacking
python27Packages.reno
python27Packages.cliff
python27Packages.debtcollector
python27Packages.stestr

python36Packages.pathlib2
python36Packages.ddt
python36Packages.testrepository
python36Packages.enum34
python36Packages.eventlet
python36Packages.ipaddress
python36Packages.jmespath
python36Packages.mock
python36Packages.monotonic
python36Packages.munch
python36Packages.netaddr
python36Packages.netifaces
python36Packages.nose
python36Packages.oauthlib
python36Packages.warlock
python36Packages.fasteners
python36Packages.mox3
python36Packages.traceback2
python36Packages.linecache2
python36Packages.pep8
python36Packages.testscenarios
python36Packages.paramiko
python36Packages.prettytable
python36Packages.pyinotify
python36Packages.subunit
python36Packages.xattr
python36Packages.unittest2
python36Packages.unicodecsv
python36Packages.python-json-logger
python36Packages.daiquiri
python36Packages.python-coveralls
python36Packages.flake8-polyfill
python36Packages.flake8-docstrings
python36Packages.runtest
python36Packages.hacking
python36Packages.reno
python36Packages.cliff
python36Packages.debtcollector
python36Packages.stestr
Things done

See individual commits for more detail.

pythonPackages.pathlib2: refactor add testing all pass
pythonPackages.ddt: 1.0.0 -> 1.2.0
pythonPackages.testrepository: refactor move to python-modules
pythonPackages.enum34: refactor move to python-modules
pythonPackages.eventlet: 0.20.0 -> 0.24.1
pythonPackages.ipaddress: 1.0.18 -> 1.0.22
pythonPackages.jmespath: 0.9.0 -> 0.9.3
pythonPackages.mock: refactor
pythonPackages.monotonic: 1.3 -> 1.5
pythonPackages.munch: refactor move to python-modules
pythonPackages.netaddr: refactor
pythonPackages.netifaces: 0.10.6 -> 0.10.7
pythonPackages.nose: refactor add meta move to python-modules
pythonPackages.oauthlib: 2.0.0 -> 2.1.0
pythonPackages.warlock: 1.2.0 -> 1.3.0
pythonPackages.fasteners: refactor move to python-modules
pythonPackages.mox3: 0.23.0 -> 0.26.0
pythonPackages.traceback2: refactor move to python-modules
pythonPackages.linecache2: refactor move to python-modules
pythonPackages.pep8: refactor move to python-modules
pythonPackages.testscenarios: 0.4 -> 0.5.0
pythonPackages.paramiko: refactor move to python-modules
pythonPackages.prettytable: 0.7.1 -> 0.7.2
pythonPackages.pyinotify: refactor move to python-modules
pythonPackages.subunit: 1.0.0 -> 1.3.0
pythonPackages.xattr: 0.7.8 -> 0.9.6
pythonPackages.unittest2: refactor move to python-modules
pythonPackages.unicodecsv: refactor move to python-modules
pythonPackages.python-json-logger: init at 0.1.9
pythonPackages.daiquiri: init at 1.5.0
pythonPackages.python-coveralls: init at 2.9.1
pythonPackages.flake8-polyfill: init at 1.0.2
pythonPackages.flake8-docstrings: init at 1.3.0
pythonPackages.runtest: init at 2.1.0
pythonPackages.hacking: init at 1.1.0
pythonPackages.reno: init at 2.10.0
pythonPackages.cliff: init at 2.13.0
pythonPackages.debtcollector: init at 1.20.0
pythonPackages.stestr: init at 2.1.1

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@costrouc costrouc force-pushed the costrouc/to-python-modules branch 6 times, most recently from 9077318 to b25d6aa Compare September 11, 2018 21:25
@costrouc
pythonPackages.testrepository: refactor move to python-modules
74cfb02 
additionally tests were previously not running. Now skips tests
but 3 fail/ 350 tests so much better than before.
@costrouc
pythonPackages.enum34: refactor move to python-modules
6764c48 
removed testing script becuase it is default behavior.
@costrouc
pythonPackages.eventlet: 0.20.0 -> 0.24.1
e3028cf 
 - changed to fetchPypi
 - moved to python-modules
 - only 23/300 tests fail (so skip check)
@costrouc
pythonPackages.ipaddress: 1.0.18 -> 1.0.22
3e690e0 
 - moved to python-modules
 - all tests pass
 - fetchPypi
@costrouc
pythonPackages.jmespath: 0.9.0 -> 0.9.3
960e219 
 - fetchPypi
 - moved to python-modules
@costrouc
pythonPackages.mock: refactor
53d95cb 
  - fetchPypi
  - move to python-modules
@costrouc
pythonPackages.monotonic: 1.3 -> 1.5
7e4bbab 
 - move to python-modules
 - fetchPypi
 - add meta
@costrouc
pythonPackages.munch: refactor move to python-modules
43de79c 
 - move to python-modules
 - fetchPypi
@costrouc
pythonPackages.netaddr: refactor, pythonPackages.netifaces: 0.10.6 ->
f5b4f4b 
0.10.7

 - attributes were next to eachother in hard to create seperate
   commits
 - fetchPypi
 - move to python-modules
@costrouc
pythonPackages.nose: refactor add meta move to python-modules
353e24e 
 - move to python-modules
 - added meta
@costrouc
pythonPackages.oauthlib: 2.0.0 -> 2.1.0
d648a8d 
 - fetchPypi
 - move to python-modules
@costrouc
pythonPackages.warlock: 1.2.0 -> 1.3.0
2bb77a5 
 - move to python-modules
 - fetchPypi
 - added testing (diabled) only 1 fails due to not being packaged with
   pypi release
@costrouc
pythonPackages.fasteners: refactor move to python-modules
d4c3b10 
 - fetchPypi
 - add license
@costrouc
pythonPackages.mox3: 0.23.0 -> 0.26.0
b55113f 
 - move to python-modules
 - all but 3 test pass in python3.6 skip.
 - aded meta
@costrouc
pythonPackages.traceback2: refactor, pythonPackages.linecache2:
10fd22a 
refactor

 - fetchPypi
 - licenses
 - move to python-modules
@costrouc
pythonPackages.pep8: refactor move to python-modules
516a239 
 - move to python-modules
 - fetchPypi
@costrouc
pythonPackages.testscenarios: 0.4 -> 0.5.0
5ecaf0c 
 - move to python-modules
 - fetchPypi
 - added testing all pass
@costrouc
pythonPackages.paramiko: refactor move to python-modules
21c1d07 
 - move to python-modules
@costrouc
pythonPackages.prettytable: 0.7.1 -> 0.7.2
4de2fea 
 - move to python-modules
 - fetchPypi
 - all tests pass
@costrouc
pythonPackages.unittest2: refactor, pythonPackages.unicodecsv:
4b06bcb 
refactor

 - fetchPypi
 - move to python-modules
 - added meta
@costrouc
pythonPackages.python-json-logger: init at 0.1.9
7fb9557 
 - all test pass
 - fetchPypi
 - full meta
@costrouc
pythonPackages.daiquiri: init at 1.5.0
add5b4f 
 - all test pass
 - fetchPypi
 - full meta
@costrouc
pythonPackages.python-coveralls: init at 2.9.1
d44af20 
 - fetchPypi
 - test are not run becuase require seperate repository
 - full meta
@costrouc
pythonPackages.flake8-polyfill: init at 1.0.2
31bc654 
 - full test run
 - fetchPypi
 - full meta
@costrouc
pythonPackages.flake8-docstrings: init at 1.3.0
57f682c 
 - full tests pass
 - fetchPypi
 - full meta
@costrouc
pythonPackages.runtest: init at 2.1.0
1e38a3f 
 - most of tests pass requires additional files not in repository
 - full meta
 - fetchPypi
@costrouc
pythonPackages.hacking: init at 1.1.0
4dcf630 
 - full testing
 - fetchPypi
 - full meta
 - required fixing the flake8 version otherwise tests do not pass.
   This is a tedious requirements.
@costrouc
pythonPackages.openstackdocstheme: init at 1.23.2
a0ba133 
 - fetchPypi
 - full meta
 - no need for tests on sphinx theme
@costrouc
pythonPackages.reno: init at 2.10.0
e9a742a 
 - fetchPypi
 - full meta
 - tests mostly run but requires git repository for full testing not
   pypi release
@costrouc
pythonPackages.cliff: init at 2.13.0
6add11f 
 - all tests pass
 - fetchPypi
 - full meta
@costrouc
pythonPackages.debtcollector: init at 1.20.0
79ee661 
 - all tests pass
 - fetchPypi
 - full meta
@costrouc
pythonPackages.stestr: init at 2.1.1
d378744 
 - full tests pass
 - fetchPypi
 - full meta
@xeji
Copy link
Contributor

xeji commented Sep 12, 2018

This causes a mass rebuild and needs to go to staging.

@costrouc
Copy link
Member Author

Does this require me to pull request to a different branch? Or is this done on the other side?

@xeji
Copy link
Contributor

xeji commented Sep 12, 2018

Does this require me to pull request to a different branch?

Yes, to the staging branch. But you can modify this existing PR in two steps:

  1. modify the PR's base branch to staging using the Edit button next to PR title. (after this, the PR may temporarily show lots of unwanted commits, but these disappear after step 2).
  2. git rebase your local branch on staging and force-push to github

@xeji
Copy link
Contributor

xeji commented Sep 12, 2018 •
edited

You could also split this PR into two parts: one part (all the new packages plus those changes that don't cause mass rebuilds) goes against master, the second part (changes to existing packages that cause mass rebuilds - probably mock, paramiko, unittest etc.) goes to staging. Up to you.
Things that go through staging can take 1-2 weeks to end up in master.

@costrouc
Copy link
Member Author

costrouc commented Sep 12, 2018 •
edited

I am in no rush to get this into master so 1-2 weeks is fine. Just a side project I've been working on the package openstack. Thanks so much for your help!

Question relevant to this. From what I have gathered the long term goal with python is to move all of the packages into python-modules. python-packages.nix still has many packages that have not been moved to python-modules/..... Many of these packages when moved will have to rebuild because they are using fetchurl instead of fetchPypi and are not updated to the current version. So massive rebuilds are inevitable for all of these packages right?

@xeji
Copy link
Contributor

xeji commented Sep 12, 2018

So massive rebuilds are inevitable for all of these packages right?

That obviously depends on the number of packages depending on them 😄 . But you're right, changing a larger number of python packages will often result in a mass rebuild. That's why we try to bundle updates into large PRs (example: #43972) once in a while.

@FRidh
Copy link
Member

FRidh commented Sep 13, 2018

Related discussion on Discourse https://discourse.nixos.org/t/how-to-handle-packaging-python-packages-that-pin-dependency-version/918/8

@FRidh FRidh self-assigned this Sep 13, 2018
@costrouc
Copy link
Member Author

After having that discussion with you @FRidh on discourse I understand that supporting packages that are complex and pin dependencies are complicated and make the work on maintainers much harder. So I am not sure how much I want to persue this pull request. While I did spend a lot of time on the pull request I do not think that supporting openstack modules is a great path.

I am thinking of just splitting this up into smaller pull requests and not adding packages that are needed for openstack.

@FRidh
Copy link
Member

FRidh commented Sep 13, 2018

@costrouc add those that you think are useful and good to have and that you think you can and want to maintain.

@Ekleog
Copy link
Member

Ekleog commented Oct 6, 2018

(triage) @costrouc From my reading of your last message, I think this PR could be closed until you add only the packages you need? :)

@costrouc costrouc closed this Oct 8, 2018
@Valodim
Copy link
Contributor

Valodim commented Jul 5, 2021 •
edited

For the record, and those googling "nixpkgs openstacksdk": openstacksdk can at this point be packaged with trivial packagings of os-service-types and keystoneauth1, albeit without checks.

Simple gist of the package: https://gist.github.com/Valodim/4dd4d052f6f1124afcd1729b3d28c5a5

The checks of those packages have circular dependencies (os-service-types (test) -> keystoneauth1 -> os-service-types), and also quite a bunch of missing deps. I believe this is the original reason this PR required so many added packages, so it's still true that this is overall hard to package properly. I'm not sure we can upstream this into nixpkgs easily, unless we forego all tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh

Assignees

@FRidh FRidh

Labels
6.topic: python 8.has: documentation 8.has: package (new) 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@costrouc @xeji @FRidh @Ekleog @Valodim @GrahamcOfBorg