Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python package set major updates #76059

Merged
merged 421 commits into from Dec 30, 2019
Merged

Python package set major updates #76059

merged 421 commits into from Dec 30, 2019

Conversation

FRidh
Copy link
Member

@FRidh FRidh commented Dec 19, 2019

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @

@FRidh
Copy link
Member Author

FRidh commented Dec 19, 2019

@jonringer feel free to force-push.

@FRidh
Copy link
Member Author

FRidh commented Dec 19, 2019

Should also consider #65023 again

@marsam
Copy link
Contributor

marsam commented Dec 19, 2019

can you revert the gym update 834a4f7?, because it requires python-opencv which is not in nixpkgs yet

@FRidh
Copy link
Member Author

FRidh commented Dec 19, 2019

Done

@FRidh
Copy link
Member Author

FRidh commented Dec 19, 2019

Could you point me to that python-opencv. It's a mess with all those different opencv packages.

@marsam
Copy link
Contributor

marsam commented Dec 19, 2019

Thanks.
It's https://pypi.org/project/opencv-python/ https://github.com/skvark/opencv-python
IIRC @jonringer mentioned that packaging it was not a trivial task

@jonringer
Copy link
Contributor

IIRC, we can also use the opencv package to make the bindings, the https://github.com/skvark/opencv-python repo is just there to publish the wheels.

But tying to build from opencv-python is painful as it makes a lot of assumptions about network access, git, and cmake integration.

@FRidh
Copy link
Member Author

FRidh commented Dec 19, 2019

Yes, we just need to build a dist-info folder that has the opencv-python name.

@jonringer
Copy link
Contributor

the only packages i can think of that use opencv-python are imgaug, baselines, and gym

@jonringer
Copy link
Contributor

there's a PR to fix imgaug here: #67494
gym doesn't need it unless it's bumped
and baselines currently removes it from the install_requires

Although eventually, we should add it. I just don't have the will right now.

@jonringer
Copy link
Contributor

did you want help stabilizing this? because I know a lot of leaf packages probably got broken

@FRidh
Copy link
Member Author

FRidh commented Dec 20, 2019

Yes, I think we need to do such an upgrade again. I saw some of the PR's, and those are bound to cause regressions as well. With this PR/branch we can test them.

I will actually be away for a week so probably need to rerun this after.

@jonringer
Copy link
Contributor

I feel like now would be a good time to bump pytest and hypothesis

@FRidh
Copy link
Member Author

FRidh commented Dec 21, 2019

I feel like now would be a good time to bump pytest and hypothesis

Definitely!

Thank you for looking at these packages. Next Sunday I'll be back and can have a look as well.

@jonringer
Copy link
Contributor

had to revert sip bump, as it broke a lot of pyqt builds

@jonringer
Copy link
Contributor

going to revert the boto/moto bumps apparently aws is a very fragile api

@jonringer
Copy link
Contributor

also reverted the major coverage bump, as that broke quite a few packages

FRidh and others added 22 commits December 30, 2019 16:46
Not a mandatory dependency so remove it.
When `makeWrapperArgs` variable is not set, `declare -p makeWrapperArgs`
will return with 1 and print an error message to stderr.

I did not handle the non-existence case in b063340
because I thought `mk-python-derivation` will always define `makeWrapperArgs`
but `wrapProgram` can be called independently. And even with `mk-python-derivation`,
`makeWrappers` will not be set unless explicitly declared in the derivation
because of NixOS/nix#1461.

I was lead to believe that because the builds were succeeding and I confirmed
that the mechanism fails when the variable is not defined and `-o nounset` is enabled.
It appears that `wrapPython` setup hook is not running under `-o nounset`, though,
invaldating the assumption.

Now we are checking that the variable exists before checking its type, which
will get rid of the warning and also prevent future error when `-o nounset`
is enabled in the setup hook.

For more information, see the discussion at
a6bb2ed
as we already set

    LANG = "${if python.stdenv.isDarwin then "en_US" else "C"}.UTF-8";

in `buildPythonPackage`.

This is related to issue #74904
@FRidh FRidh changed the base branch from staging to staging-next December 30, 2019 15:50
@FRidh FRidh merged commit acc3dd2 into staging-next Dec 30, 2019
@FRidh
Copy link
Member Author

FRidh commented Dec 30, 2019

Forgot to merge it into staging so that's why it went to staging-next...

@@ -8,11 +8,12 @@

buildPythonPackage rec {
pname = "gunicorn";
version = "19.9.0";
version = "20.0.4";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI this change breaks apache-airflow build:

ERROR: Could not find a version that satisfies the requirement gunicorn<20.0,>=19.5.0 (from apache-airflow==1.10.5) (from versions: none)
ERROR: No matching distribution found for gunicorn<20.0,>=19.5.0 (from apache-airflow==1.10.5)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

opening a ticket is more action-able :), I unfortunately don't have time to look into this :(

@JeffLabonte JeffLabonte mentioned this pull request Mar 7, 2020
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants