Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python fix 9 broken packages #65023

Closed
wants to merge 8 commits into from
Closed

Python fix 9 broken packages #65023

wants to merge 8 commits into from

Conversation

costrouc
Copy link
Member

Motivation for this change

Closes PR #65003 and adds the packages that were merged into python-unstable into master. I will merge into staging if this is too large of a change

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc: @FRidh

@costrouc costrouc requested a review from FRidh as a code owner July 18, 2019 12:42
@costrouc
Copy link
Member Author

Running nix-review right now

@costrouc
Copy link
Member Author

Results

Result of nix-review pr 65023 1

61 package failed to build:
  • aws-sam-cli (docker_compose)
  • aws_shell (docker_compose)
  • awscli (docker_compose)
  • buildbot (docker_compose)
  • csvs-to-sqlite (docker_compose)
  • elasticsearch-curator (docker_compose)
  • fedpkg (docker_compose)
  • flexget (docker_compose)
  • iceshelf (docker_compose)
  • nbstripout (docker_compose)
  • nixops_1_6_1 (docker_compose)
  • poretools (docker_compose)
  • pyCA (docker_compose)
  • pyload (docker_compose)
  • python27Packages.altair (docker_compose)
  • python27Packages.atomman (docker_compose)
  • python27Packages.awkward (docker_compose)
  • python27Packages.bkcharts (docker_compose)
  • python27Packages.blaze (docker_compose)
  • python27Packages.blivet (docker_compose)
  • python27Packages.bokeh (docker_compose)
  • python27Packages.bugwarrior (docker_compose)
  • python27Packages.celery (docker_compose)
  • python27Packages.cfn-lint (docker_compose)
  • python27Packages.csvs-to-sqlite (docker_compose)
  • python27Packages.cufflinks (docker_compose)
  • python27Packages.dask (docker_compose)
  • python27Packages.dask-glm (docker_compose)
  • python27Packages.dask-image (docker_compose)
  • python27Packages.dask-jobqueue (docker_compose)
  • python27Packages.dask-ml (docker_compose)
  • python27Packages.dask-mpi (docker_compose)
  • python27Packages.distributed (docker_compose)
  • python27Packages.djmail (docker_compose)
  • python27Packages.dm-sonnet (docker_compose)
  • python27Packages.drms (docker_compose)
  • python27Packages.fastparquet (docker_compose)
  • python27Packages.geopandas (docker_compose)
  • python27Packages.glymur (docker_compose)
  • python27Packages.google_cloud_bigquery (docker_compose)
  • python27Packages.google_cloud_monitoring (docker_compose)
  • python27Packages.graph_nets (docker_compose)
  • python27Packages.histbook (docker_compose)
  • python27Packages.holoviews (docker_compose)
  • python27Packages.hvplot (docker_compose)
  • python27Packages.image-match (docker_compose)
  • python27Packages.imbalanced-learn (docker_compose)
  • python27Packages.koji (docker_compose)
  • python27Packages.mapsplotlib (docker_compose)
  • python27Packages.moto (docker_compose)
  • python27Packages.nixpart0 (docker_compose)
  • python27Packages.odo (docker_compose)
  • python27Packages.osc (docker_compose)
  • python27Packages.pandas (docker_compose)
  • python27Packages.panel (docker_compose)
  • python27Packages.paperspace (docker_compose)
  • python27Packages.partd (docker_compose)
  • python27Packages.pims (docker_compose)
  • python27Packages.pvlib (docker_compose)
  • python27Packages.pyarrow (docker_compose)
  • python27Packages.pycurl (docker_compose)
51 package were build:
  • aws-google-auth (docker_compose)
  • awslogs (docker_compose)
  • caffeine-ng (docker_compose)
  • credstash (docker_compose)
  • devpi-server (docker_compose)
  • docker-compose (docker_compose)
  • luigi (docker_compose)
  • matrix-synapse (docker_compose)
  • nixops (docker_compose)
  • nixopsUnstable (docker_compose)
  • or-tools (docker_compose)
  • poetry (docker_compose)
  • python27Packages.ansible-kernel (docker_compose)
  • python27Packages.argon2_cffi (docker_compose)
  • python27Packages.astunparse (docker_compose)
  • python27Packages.aws-adfs (docker_compose)
  • python27Packages.aws-sam-translator (docker_compose)
  • python27Packages.baselines (docker_compose)
  • python27Packages.boto3 (docker_compose)
  • python27Packages.cartopy (docker_compose)
  • python27Packages.chalice (docker_compose)
  • python27Packages.colorcet (docker_compose)
  • python27Packages.credstash (docker_compose)
  • python27Packages.cupy (docker_compose)
  • python27Packages.edward (docker_compose)
  • python27Packages.fiona (docker_compose)
  • python27Packages.flask-restplus (docker_compose)
  • python27Packages.flowlogs_reader (docker_compose)
  • python27Packages.fs-s3fs (docker_compose)
  • python27Packages.gast (docker_compose)
  • python27Packages.gensim (docker_compose)
  • python27Packages.glob2 (docker_compose)
  • python27Packages.ipywidgets (docker_compose)
  • python27Packages.jsonschema (docker_compose)
  • python27Packages.jupyter (docker_compose)
  • python27Packages.jupytext (docker_compose)
  • python27Packages.kmapper (docker_compose)
  • python27Packages.modeled (docker_compose)
  • python27Packages.moretools (docker_compose)
  • python27Packages.nbconvert (docker_compose)
  • python27Packages.nbformat (docker_compose)
  • python27Packages.nbmerge (docker_compose)
  • python27Packages.nbsmoke (docker_compose)
  • python27Packages.nbsphinx (docker_compose)
  • python27Packages.nbval (docker_compose)
  • python27Packages.notebook (docker_compose)
  • python27Packages.ortools (docker_compose)
  • python27Packages.passlib (docker_compose)
  • python27Packages.plotly (docker_compose)
  • python27Packages.poetry (docker_compose)
  • python37Packages.ortools (docker_compose)

@costrouc
Copy link
Member Author

Does this mean that docker_compose is broken right now?

@costrouc costrouc mentioned this pull request Jul 18, 2019
Closed
10 tasks
@ofborg ofborg bot requested a review from domenkozar July 18, 2019 14:22
@jonringer
Copy link
Contributor

it looks like your jsonschema bump to the next major version is what broke most things.

The most proper (but pain-in-the-***) fix I can think of, would be to pin pythonPackages.jsonschema_2 and go to each package that still requires it and amend them.

Collecting jsonschema~=2.6 (from cfn-lint==0.19.1)
  ERROR: Could not find a version that satisfies the requirement jsonschema~=2.6 (from cfn-lint==0.19.1) (from versions: none)
ERROR: No matching distribution found for jsonschema~=2.6 (from cfn-lint==0.19.1)
builder for '/nix/store/aav481gj38kdy5qj5rzm2qrjx7gsvlsk-python3.7-cfn-lint-0.19.1.drv' failed with exit code 1
cannot build derivation '/nix/store/v1rdmnwwf1q04jggbp1r53nrb182y13h-python3.7-moto-1.3.8.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/w7vfwa40gj71apdvniavxs0gah1927ia-python3.7-buildbot-2.3.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/w7vfwa40gj71apdvniavxs0gah1927ia-python3.7-buildbot-2.3.1.drv' failed

@FRidh will probably have better opinions on how to go about this.

@FRidh
Copy link
Member

FRidh commented Jul 19, 2019

would be to pin pythonPackages.jsonschema_2 and go to each package that still requires it and amend them.

We cannot have multiple versions of the same package in python-packages.nix because that leads to collisions when using them together.

@FRidh
Copy link
Member

FRidh commented Jul 19, 2019

Here the typical approach is to be a bit patient, and when a significant majority is compatible just make the switch and break the rest.

@jonringer
Copy link
Contributor

oh right, because they would both exist on PYTHONPATH, and wouldn't be deterministic in which one would be loaded first....

@FRidh FRidh self-assigned this Jul 20, 2019
@multun
Copy link
Member

multun commented Jul 31, 2019 •
edited

@FRidh

We cannot have multiple versions of the same package in python-packages.nix because that leads to collisions when using them together.

Isn't it somehow already the case ?

(I'm not quite sure about what I say, I'm pretty much a nix newbie)

@FRidh
Copy link
Member

FRidh commented Jul 31, 2019

I know. That went in but should not have gone in.

@worldofpeace
Copy link
Contributor

@FRidh ping. Most pertinent thing here is @costrouc updated jsonschema and its gone unnoticed for months...

@FRidh
Copy link
Member

FRidh commented Oct 1, 2019

It's not gone unnoticed for months. It's just that we simply cannot update jsonschema because it would break too many other reverse dependencies. Now we could investigate again.

@FRidh FRidh mentioned this pull request Dec 19, 2019
Merged
10 tasks
@jonringer
Copy link
Contributor

most of these commits have gone very stale.

bumping @costrouc

@worldofpeace
Copy link
Contributor

I think they might have got integrated somehow?
I hope this hasn't discouraged @costrouc in any way from contributing. I simply can't remember what exactly happened, perhaps it was an issue with me not being informed enough about python-unstable?

Anyways, I will close it because it's out of date and conflicting.
I hope any similar situation shouldn't occur anymore.

@jonringer
Copy link
Contributor

it was paused for the longest time because the jsonschema major bump broke a lot of packages that didn't support >=3.0.

Then in october, python 3.8 was released and much of the python landscape changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh Awaiting requested review from FRidh

@domenkozar domenkozar Awaiting requested review from domenkozar

Assignees

@FRidh FRidh

Labels
6.topic: python 8.has: package (new) 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@costrouc @jonringer @FRidh @multun @worldofpeace