Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pythonPackages.pyspark: call wrapped python script correctly #64740

Closed
wants to merge 1 commit into from
Closed

pythonPackages.pyspark: call wrapped python script correctly #64740

wants to merge 1 commit into from

Conversation

mattmelling
Copy link
Contributor

Motivation for this change

See #64358. pyspark tries to figure out SPARK_HOME by calling a Python script, but it has already been wrapped. This change invokes the wrapped script correctly, as well as sets a correct search path for the pyspark package.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@mattmelling mattmelling requested a review from FRidh as a code owner July 14, 2019 20:34
@ofborg ofborg bot requested a review from shlevy July 14, 2019 20:48
jonringer
jonringer reviewed Jul 15, 2019
View reviewed changes
Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option may be to just patch the (eventually) wrapped python script to hand back the exact home of spark, instead of worry about the env var logic. One of the benefits of nix :).

example commit, if im talking gibberish 06960ac

FRidh
FRidh reviewed Jul 15, 2019
View reviewed changes
pkgs/development/python-modules/pyspark/default.nix Outdated
# includes SPARK_HOME
substituteInPlace $out/bin/find-spark-home \
--replace 'export SPARK_HOME=$($PYSPARK_DRIVER_PYTHON "$FIND_SPARK_HOME_PYTHON_SCRIPT")' \
'export SPARK_HOME=$("$FIND_SPARK_HOME_PYTHON_SCRIPT")''\n export PYTHONPATH="$(dirname $SPARK_HOME):$PYTHONPATH"'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please avoid exporting PYTHONPATH.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FRidh Is it OK to modify an existing PYTHONPATH export in $out/bin/pyspark (latest commit) rather than adding a new one in find-spark-home? I can't think of a better way to change the search path.

@stale
Copy link

stale bot commented Jun 1, 2020

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

  1. Search for maintainers and people that previously touched the
    related code and @ mention them in a comment.
  2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
    irc.freenode.net.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 1, 2020
@cript0nauta cript0nauta mentioned this pull request Aug 19, 2020
Closed
@SuperSandro2000 SuperSandro2000 marked this pull request as draft November 28, 2020 04:29
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 28, 2020
@stale
Copy link

stale bot commented Jun 3, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 3, 2021
@mattmelling mattmelling closed this Nov 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FRidh FRidh FRidh left review comments

@jonringer jonringer jonringer left review comments

@shlevy shlevy Awaiting requested review from shlevy

Assignees
No one assigned
Labels
2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: python 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@mattmelling @FRidh @jonringer