Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pythonPackages.jupyter_core: fix search path #83619

Closed
wants to merge 1 commit into from
Closed

pythonPackages.jupyter_core: fix search path #83619

wants to merge 1 commit into from

Conversation

eadwu
Copy link
Member

@eadwu eadwu commented Mar 28, 2020

Motivation for this change

Not sure if this is expected but if one does python.withPackages (ps: [ ps.jupyter ]), the jupyter-* commands aren't found in environments where PATH isn't propagated from the shell environment.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@jonringer
Copy link
Contributor

should this be upstreamed?

@FRidh
Copy link
Member

FRidh commented Mar 29, 2020

in environments where PATH isn't propagated from the shell environment.

What type of environment is that? Please show an example how it does not function.

@eadwu
Copy link
Member Author

eadwu commented Mar 29, 2020 •
edited

I'm not sure if it should be upstreamed since I'm not sure if this just because of a unique use case I had.

I had this when attempting to launch a Jupyter server through the vscode python extension (direnv and lorri mix to get the python linked into the folder)

Starting Jupyter from /nix/store/bsyb9kxanycs1zp6jrna07dmag7c8k6h-python3-3.7.6-env/bin/python3.7 with command line --no-browser --notebook-dir=/tmp/74698225-c878-48ca-a804-6c27cad9c517 --config=/tmp/74698225-c878-48ca-a804-6c27cad9c517/jupyter_notebook_config.py --NotebookApp.iopub_data_rate_limit=10000000000.0
Traceback (most recent call last):
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nix/store/bsyb9kxanycs1zp6jrna07dmag7c8k6h-python3-3.7.6-env/lib/python3.7/site-packages/jupyter.py", line 4, in <module>
    main()
  File "/nix/store/bsyb9kxanycs1zp6jrna07dmag7c8k6h-python3-3.7.6-env/lib/python3.7/site-packages/jupyter_core/command.py", line 247, in main
    command = _jupyter_abspath(subcommand)
  File "/nix/store/bsyb9kxanycs1zp6jrna07dmag7c8k6h-python3-3.7.6-env/lib/python3.7/site-packages/jupyter_core/command.py", line 134, in _jupyter_abspath
    'Jupyter command `{}` not found.'.format(jupyter_subcommand)
Exception: Jupyter command `jupyter-notebook` not found.

and sys.argv[0] is pointing to the raw python interpreter instead of the wrapper.

Another solution would be patching the sysconfig of new python environments, but I'm not sure if it's really needed.

gnprice
gnprice reviewed Apr 2, 2020
View reviewed changes
pkgs/development/python-modules/jupyter_core/fix-search-path.patch
Comment on lines +9 to +10
- scripts = [sys.argv[0]]
+ scripts = [sys.argv[0],sys.executable]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused here; perhaps you can fill in something I'm missing.

In a comment, you say that when you run into the issue this is meant to fix:

sys.argv[0] is pointing to the raw python interpreter instead of the wrapper.

But sys.executable is itself meant to be the path to the Python interpreter. (That's its documented meaning upstream, at least.)

What path is sys.executable pointing to in the situation where you're seeing the issue, and how does it relate to the path pointed to by sys.argv[0]?

Copy link
Member Author

@eadwu eadwu Apr 2, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too sure on how the those are actually evaluated, but sys.argv[0] doesn't point to sys.executable since I had to make this in the first place.

The actual python script in the python-env is a wrapper that calls the python interpreter, so that's probably why sys.argv[0] evaluates to the interpreter while sys.executable refers to the script that started it.

I'll be able to get some output to illustrate this tomorrow.

@FRidh
Copy link
Member

FRidh commented Apr 2, 2020

I have a Python environment installed system-wide on NixOS:

$ cat $(readlink -f $(which jupyter))
#! /nix/store/z4ajipns0l1s8b2lrgpy6nng4cys7h99-bash-4.4-p23/bin/bash -e
export NIX_PYTHONEXECUTABLE='/nix/store/bba1as20vdsh8clzjiyybk4n5ymph5z6-python3-3.7.6-env/bin/python3.7'
export NIX_PYTHONPATH='/nix/store/bba1as20vdsh8clzjiyybk4n5ymph5z6-python3-3.7.6-env/lib/python3.7/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/96f3b85v9bd31n3x4pv9xi8j9accajj2-python3.7-jupyter_core-4.6.1/bin/jupyter"  "$@"

PATH is indeed not exported.

It functions in the following cases:

  • installed in a profile, profiles bin is added to $PATH
  • nix run, because it adds bin to $PATH
    It however does not function when using nix-shell because it opens a shell to develop the python.buildEnv, which is based on buildEnv, which does not add each of its inputs bin folder to $PATH.

In a composed environment one should expect all executables can find each other, so we should add $PATH to the python.buildEnv wrapper.

@eadwu eadwu force-pushed the pythonPackages.jupyter_core/fix-search-path branch from 4d0f9ec to a1fb399 Compare April 7, 2020 18:16
@eadwu eadwu force-pushed the pythonPackages.jupyter_core/fix-search-path branch from a1fb399 to 6ae6ec5 Compare May 18, 2020 20:28
@stale
Copy link

stale bot commented Nov 14, 2020

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 14, 2020
@SuperSandro2000
Copy link
Member

@eadwu ping

@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 23, 2020
@eadwu eadwu closed this Nov 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gnprice gnprice gnprice left review comments

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

Assignees
No one assigned
Labels
6.topic: python 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@eadwu @jonringer @FRidh @SuperSandro2000 @gnprice