python3Packages.pyspark: Fix out of date postPatch #104723
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation for this change
Pyspark switched to pinning py4j==0.10.9 with v3.0.0 - see this commit:
apache/spark@fc4e56a
This meant that since the bump to pyspark v3.0.0 - in this commit:
5181547 -
the patch was no longer matching on the 'py4j==0.10.7' string that was
working previously.
The failing patch went unnoticed previously because the version of py4j
pinned by pyspark>=3.0.0 was the same as the py4j provided by nixpkgs.
However, a recent PR (#101636) bumped the version of py4j to 0.10.9.1 in
this commit:
43a9128
which caused the version pinned by pyspark to no longer match the
version provided by nixpkgs. FWIW, @jonringer flagged this issue on
another PR that tried to bump py4j: #100623.
My solution here was to upgrade the patch's target string to match the
version found in pyspark's current setup.py.
Things done
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)nix path-info -S
before and after)