[WIP] oslo-config: 4.13.2 -> 5.1.0 #32602

makefu · 2017-12-12T15:17:41Z

Motivation for this change

bump oslo-config to 5.1.0 and enable tests 💯

For enabling tests i had to bump:

reno: 2.3.2 -> 2.6.0
bandit: 0.16.0 -> 1.4.0
openstackdocstheme: init at 1.17.0
pyasn1-modules: 0.1.5 -> 0.2.1
ldappool: 1.0 -> 2.1.0
pythonPackages.ldap: 2.4.45 -> 3.0.0b2

Waiting for @GrahamcOfBorg :3

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[NA] Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

FRidh

Looks good to me.

FRidh · 2017-12-12T15:21:40Z

pkgs/development/python-modules/bandit/default.nix

+  propagatedBuildInputs = [ pbr six pyyaml appdirs stevedore GitPython git ];
+  checkInputs = [ beautifulsoup4 oslosphinx testtools testscenarios
+                              testrepository fixtures mock ];
+  patchPhase = ''


postPatch is preferred so that one can still pass in patches.

FRidh · 2017-12-12T15:22:49Z

oslo-config bump is likely to break things though. That needs to be checked first.

makefu · 2017-12-12T15:57:00Z

i am currently on it, i will have to bump a number of libs to get it running. changing Headline to WIP

makefu · 2017-12-12T16:27:29Z

oh oh, it seems that bumping pyasn1 was not the wisest thing to do. it was for getting ldap to the latest version ... i will see how much stuff breaks with this.

makefu · 2017-12-12T17:02:42Z

can this PR be submitted to hydra for evaluation? i am afraid my pc is not hardcore enough to evaludate the PR in time. if it is not possible then i will probably have to revert and wait for pyasn1 be bumped by someone more adventurous than me.

edit ok i will give up trying to bump ldap and pyasn1. i will keep put this in a separate PR

makefu · 2017-12-12T18:19:05Z

for python3Packages.oslo-config i get

error: invalid command 'bdist_wheel'

however the package is building for python2. @FRidh you had similar issues before? i tried to put wheel in every *Inputs list without success.

FRidh · 2017-12-12T19:55:43Z

That typically happens when Python 2 and 3 packages get mixed during a build. Both packages have the interpreter in propagatedBuildInputs, and the interpreters each a setup hook that extend to the same PYTHONPATH. That causes trouble. In this case, certain Python 2 applications are added in the build when tests are enabled. I suggest for now disabling tests for Python 3 builds.

FRidh · 2017-12-12T19:56:51Z

When disabling tests, can you refer to the previous comment.

makefu · 2017-12-12T20:12:27Z

@FRidh i think the package is reno (which comes from top-level/all-packages.nix), the correct way would be to pull reno into pythonPackages again, right?

update: yep it is the reno package

FRidh · 2017-12-12T20:24:16Z

@makefu if its just an application, and we're not interested in the modules, then no, then its not the correct way to pull it into python-packages.nix. But it is a pragmatic way to get your tests working ;-)

The correct way, but out of scope here, is to make sure Python applications don't show their site-packages to the hook.

fix cyclic dependency with fixtures by disabling checkPhase in fixtures

makefu · 2017-12-13T13:37:19Z

so many broken packages ... is it possible with nox-review pr to check for all packages which broke due to the pr?
a lot of issues due to argparse, not sure why this happens though. i would really fix the packages but moving them one-by-one to development/python-modules is quite some effort to go through

FRidh · 2017-12-13T14:06:13Z

so many broken packages ... is it possible with nox-review pr to check for all packages which broke due to the pr?

nox-review computes what needs to be rebuild and then builds it. You could use niff to just check what packages need to be rebuild. Its in Nixpkgs.
https://github.com/FRidh/niff

a lot of issues due to argparse, not sure why this happens though.

A lot of (older) packages have an install_requires on argparse, even though argparse is part of stdlib in 2.7+. We typically patch that dependency out.

i would really fix the packages but moving them one-by-one to development/python-modules is quite some effort to go through

Absolutely. Many of the packages you touch haven't been used in a while. It may therefore also be wise to evaluate whether they could instead be dropped.

I would prefer to move this whole openstack stuff out into an overlay. Its a pain to maintain, and its presence puts a burden on others that are contributing as well.

makefu · 2017-12-13T15:58:22Z

Moving openstack out may be a good idea, this looks like a never ending rabbit hole of (circular) dependencies.

What i could think of is using pypi2nix and just freeze all required applications (with tests disabled) and refreeze on each version change.

Edit: i just remembered why i started this in first place, which is sqlalchemy_migrate which depends on some of this clusterfuck (tempest-lib that is)

FRidh · 2017-12-13T16:12:20Z

What i could think of is using pypi2nix and just freeze all required applications (with tests disabled) and refreeze on each version change.

Won't help, because we need a coherent set that works for Python 2.7, 3.4, 3.5, 3.6 and preferably also PyPI. With pypi2nix you generate for one version.

But I thought that certain openstack libraries were already marked as broken, so I wouldn't imagine you having to upgrade them then.

makefu · 2017-12-13T16:21:16Z

Thing is, that the update resulted in more dependencies which resulted in new errors (and more update) and more dependencies. a never ending cycle.

Regarding the overlay, i think only a single python version would be enough to provide the functionality of openstack. This however would not solve the issue if people would actually want to use certain libraries in their project.

imho each module should come with a huge warning sign if it pulls dependencies from openstack ...

reno: 2.3.2 -> 2.6.0

86a08d6

makefu requested a review from FRidh as a code owner December 12, 2017 15:17

FRidh reviewed Dec 12, 2017

View reviewed changes

GrahamcOfBorg added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Dec 12, 2017

makefu changed the title ~~oslo-config: 4.13.2 -> 5.1.0~~ [WIP] oslo-config: 4.13.2 -> 5.1.0 Dec 12, 2017

makefu added 3 commits December 12, 2017 17:25

bandit: 0.16.0 -> 1.4.0

75e792c

openstackdocstheme: init at 1.17.0

d14a341

oslo-config: 4.13.2 -> 5.1.0

e7f31f1

GrahamcOfBorg added 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ and removed 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Dec 12, 2017

makefu force-pushed the pkgs/oslo-config/update branch from ec58565 to 7311f4c Compare December 12, 2017 16:51

ldappool: 1.0 -> 2.1.0

be85337

makefu force-pushed the pkgs/oslo-config/update branch from 7311f4c to be85337 Compare December 12, 2017 17:13

GrahamcOfBorg added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 and removed 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Dec 12, 2017

makefu force-pushed the pkgs/oslo-config/update branch from 5baadab to be85337 Compare December 12, 2017 17:52

makefu added 7 commits December 13, 2017 12:12

testtools: 1.9.0 -> 2.3.0

e997d87

fix cyclic dependency with fixtures by disabling checkPhase in fixtures

testresources: 0.2.7 -> 2.0.1

fd25642

ddt: 1.0.0 -> 1.1.1

1c3b39f

stestr: init at 1.1.0

9c93d27

keystoneauth1: 3.2.0 -> 3.3.0

a203dcb

keystoneclient: buildInputs -> checkInputs

0a36650

oslo-cache: 0.9.0 -> 1.27.0

32e0129

makefu force-pushed the pkgs/oslo-config/update branch from 57aff7a to d1de6c1 Compare December 13, 2017 11:13

makefu added 8 commits December 13, 2017 12:14

networking-hyperv: 2015.1.0 -> 5.0.0

acea5e0

neutron-lib: init at 1.11.0

6116700

cliff: 1.15.0 -> 2.9.1

81d0781

oslo-messaging: 2.7.0 -> 5.34.1

06569fc

tenacity: init at 4.7.1

2111161

futurist: 0.7.0 -> 1.5.0

08537ac

kombu: 4.0.2 -> 4.1.0

eeb465b

amqp: 2.1.4 -> 2.2.2

de05536

makefu force-pushed the pkgs/oslo-config/update branch from d1de6c1 to de05536 Compare December 13, 2017 13:10

barbicanclient: 3.3.0 -> 4.5.2

ab68fa3

makefu added 2 commits December 13, 2017 15:21

osc-lib: init at 1.7.0

251dc3c

neutronclient: 3.1.0 -> 6.6.0

d73a9bf

makefu force-pushed the pkgs/oslo-config/update branch from c146f33 to d73a9bf Compare December 13, 2017 14:23

makefu added 2 commits December 13, 2017 15:50

castellan: 0.2.1 -> 0.16.0

85e0fad

novaclient: 2.31.0 -> 9.1.1

5579cc5

makefu closed this Dec 13, 2017

makefu mentioned this pull request Dec 15, 2017

Error when running pypi2nix command nix-community/pypi2nix#164

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] oslo-config: 4.13.2 -> 5.1.0 #32602

[WIP] oslo-config: 4.13.2 -> 5.1.0 #32602

makefu commented Dec 12, 2017 •

edited

FRidh left a comment

FRidh Dec 12, 2017

FRidh commented Dec 12, 2017

makefu commented Dec 12, 2017

makefu commented Dec 12, 2017

makefu commented Dec 12, 2017 •

edited

makefu commented Dec 12, 2017

FRidh commented Dec 12, 2017

FRidh commented Dec 12, 2017

makefu commented Dec 12, 2017 •

edited

FRidh commented Dec 12, 2017

makefu commented Dec 13, 2017 •

edited

FRidh commented Dec 13, 2017

makefu commented Dec 13, 2017 •

edited

FRidh commented Dec 13, 2017

makefu commented Dec 13, 2017

[WIP] oslo-config: 4.13.2 -> 5.1.0 #32602

[WIP] oslo-config: 4.13.2 -> 5.1.0 #32602

Conversation

makefu commented Dec 12, 2017 • edited

Motivation for this change

Things done

FRidh left a comment

Choose a reason for hiding this comment

FRidh Dec 12, 2017

Choose a reason for hiding this comment

FRidh commented Dec 12, 2017

makefu commented Dec 12, 2017

makefu commented Dec 12, 2017

makefu commented Dec 12, 2017 • edited

makefu commented Dec 12, 2017

FRidh commented Dec 12, 2017

FRidh commented Dec 12, 2017

makefu commented Dec 12, 2017 • edited

FRidh commented Dec 12, 2017

makefu commented Dec 13, 2017 • edited

FRidh commented Dec 13, 2017

makefu commented Dec 13, 2017 • edited

FRidh commented Dec 13, 2017

makefu commented Dec 13, 2017

makefu commented Dec 12, 2017 •

edited

makefu commented Dec 12, 2017 •

edited

makefu commented Dec 12, 2017 •

edited

makefu commented Dec 13, 2017 •

edited

makefu commented Dec 13, 2017 •

edited