Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update jupyter packages to python 3 only packages with 27 support #49721

Closed
wants to merge 9 commits into from

Conversation

costrouc
Copy link
Member

@costrouc costrouc commented Nov 4, 2018

Motivation for this change & Things done

Jupyter has moved their releases to only supporting python 3+. This has blocked upgrading of several python packages within nixpkgs. These include ipython, ipykernel, and jupyter. This PR upgrades to newest releases while keeping python 2.7 support for these packages as well.

I would like input on if this is the appropriate way to support multiple versions. The important details are in the changes to python-packages.nix

This expression allows support for the most recent versions of ipython for each python version.

This update required making prompt_toolkit 2.x.x the default version. I have ensure that all applications that depended on prompt_toolkit 1.x.x still work. This issue was addressed recently with #49853.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@costrouc costrouc requested a review from FRidh as a code owner November 4, 2018 02:32
@costrouc costrouc changed the title Update jupyter packages to python 3 only packages Update jupyter packages to python 3 only packages with 27 support Nov 4, 2018
@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python27Packages.ipython, python36Packages.jupyter_console

Partial log (click to expand)

/build/jupyter_console-6.0.0
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/9n7nhr0wvryxqg064xwj74fzvgijy0n5-python3.6-jupyter_console-6.0.0
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/9n7nhr0wvryxqg064xwj74fzvgijy0n5-python3.6-jupyter_console-6.0.0/lib  /nix/store/9n7nhr0wvryxqg064xwj74fzvgijy0n5-python3.6-jupyter_console-6.0.0/bin
patching script interpreter paths in /nix/store/9n7nhr0wvryxqg064xwj74fzvgijy0n5-python3.6-jupyter_console-6.0.0
checking for references to /build in /nix/store/9n7nhr0wvryxqg064xwj74fzvgijy0n5-python3.6-jupyter_console-6.0.0...
wrapping `/nix/store/9n7nhr0wvryxqg064xwj74fzvgijy0n5-python3.6-jupyter_console-6.0.0/bin/jupyter-console'...
/nix/store/3s4ggi31iw4w9l52wzk37cylkm1qv5i7-python2.7-ipython-5.8.0
/nix/store/9n7nhr0wvryxqg064xwj74fzvgijy0n5-python3.6-jupyter_console-6.0.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python27Packages.ipython, python36Packages.jupyter_console

Partial log (click to expand)

/build/jupyter_console-6.0.0
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/hrkn6dxvvnbibmrrvw006zihfmsqvdvy-python3.6-jupyter_console-6.0.0
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/hrkn6dxvvnbibmrrvw006zihfmsqvdvy-python3.6-jupyter_console-6.0.0/lib  /nix/store/hrkn6dxvvnbibmrrvw006zihfmsqvdvy-python3.6-jupyter_console-6.0.0/bin
patching script interpreter paths in /nix/store/hrkn6dxvvnbibmrrvw006zihfmsqvdvy-python3.6-jupyter_console-6.0.0
checking for references to /build in /nix/store/hrkn6dxvvnbibmrrvw006zihfmsqvdvy-python3.6-jupyter_console-6.0.0...
wrapping `/nix/store/hrkn6dxvvnbibmrrvw006zihfmsqvdvy-python3.6-jupyter_console-6.0.0/bin/jupyter-console'...
/nix/store/l085qn94y2q27j47cyjk3rcfr3lsibh1-python2.7-ipython-5.8.0
/nix/store/hrkn6dxvvnbibmrrvw006zihfmsqvdvy-python3.6-jupyter_console-6.0.0

pkgs/top-level/python-packages.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/ipykernel/4.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/ipykernel/4.nix Outdated Show resolved Hide resolved
pkgs/top-level/python-packages.nix Show resolved Hide resolved
@costrouc
Copy link
Member Author

costrouc commented Nov 4, 2018

Fixed with all the recommended changes.

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python27Packages.ipython, python36Packages.jupyter_console

Partial log (click to expand)

/build/jupyter_console-6.0.0
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/m6xyy5jzhwzdc49g3wwg5bisrhdvaz0d-python3.6-jupyter_console-6.0.0
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/m6xyy5jzhwzdc49g3wwg5bisrhdvaz0d-python3.6-jupyter_console-6.0.0/lib  /nix/store/m6xyy5jzhwzdc49g3wwg5bisrhdvaz0d-python3.6-jupyter_console-6.0.0/bin
patching script interpreter paths in /nix/store/m6xyy5jzhwzdc49g3wwg5bisrhdvaz0d-python3.6-jupyter_console-6.0.0
checking for references to /build in /nix/store/m6xyy5jzhwzdc49g3wwg5bisrhdvaz0d-python3.6-jupyter_console-6.0.0...
wrapping `/nix/store/m6xyy5jzhwzdc49g3wwg5bisrhdvaz0d-python3.6-jupyter_console-6.0.0/bin/jupyter-console'...
/nix/store/3s4ggi31iw4w9l52wzk37cylkm1qv5i7-python2.7-ipython-5.8.0
/nix/store/m6xyy5jzhwzdc49g3wwg5bisrhdvaz0d-python3.6-jupyter_console-6.0.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python27Packages.ipython, python36Packages.jupyter_console

Partial log (click to expand)

/build/jupyter_console-6.0.0
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/zlq94qqnijh4kixx813phqf6f94p9nz3-python3.6-jupyter_console-6.0.0
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/zlq94qqnijh4kixx813phqf6f94p9nz3-python3.6-jupyter_console-6.0.0/lib  /nix/store/zlq94qqnijh4kixx813phqf6f94p9nz3-python3.6-jupyter_console-6.0.0/bin
patching script interpreter paths in /nix/store/zlq94qqnijh4kixx813phqf6f94p9nz3-python3.6-jupyter_console-6.0.0
checking for references to /build in /nix/store/zlq94qqnijh4kixx813phqf6f94p9nz3-python3.6-jupyter_console-6.0.0...
wrapping `/nix/store/zlq94qqnijh4kixx813phqf6f94p9nz3-python3.6-jupyter_console-6.0.0/bin/jupyter-console'...
/nix/store/l085qn94y2q27j47cyjk3rcfr3lsibh1-python2.7-ipython-5.8.0
/nix/store/zlq94qqnijh4kixx813phqf6f94p9nz3-python3.6-jupyter_console-6.0.0

@costrouc
Copy link
Member Author

costrouc commented Nov 7, 2018

Bump. I should also add this this PR will enable "auto-install" of extensions which should make #49807 much easier. This package should be ready for merge. No python 27, 3+ support was lost. Only separated to allow upgrade of package and easy removal of 27 support in the future.

…7.0 -> 7.1.1

 - removed patch for sage as it is applied in most recent release.
 - simplified `ipython` attribute no need for explicit version `5`
   vs. `6`.
 - upgrade to most recent version of ipython
Created a version to prompt_toolkit_1 for legacy ipython
@costrouc
Copy link
Member Author

@GrahamcOfBorg build python36Packages.ipython python36Packages.prompt_toolkit python36Packages.notebook python36Packages.prompt_toolkit python36Packages.ipykernel python36Packages.jupyterlab_server python36Packages.jupyterlab python36Packages.jupyterhub

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python27Packages.ipython, python36Packages.jupyter_console

Partial log (click to expand)

/build/jupyter_console-6.0.0
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0/lib  /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0/bin
patching script interpreter paths in /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0
checking for references to /build in /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0...
wrapping `/nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0/bin/jupyter-console'...
/nix/store/3s4ggi31iw4w9l52wzk37cylkm1qv5i7-python2.7-ipython-5.8.0
/nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python27Packages.ipython, python36Packages.jupyter_console

Partial log (click to expand)

/build/jupyter_console-6.0.0
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0/lib  /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0/bin
patching script interpreter paths in /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0
checking for references to /build in /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0...
wrapping `/nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0/bin/jupyter-console'...
/nix/store/l085qn94y2q27j47cyjk3rcfr3lsibh1-python2.7-ipython-5.8.0
/nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python36Packages.ipython, python36Packages.prompt_toolkit, python36Packages.notebook, python36Packages.prompt_toolkit, python36Packages.ipykernel, python36Packages.jupyterlab_server, python36Packages.jupyterlab, python36Packages.jupyterhub

Partial log (click to expand)

wrapping `/nix/store/x0g0i3y075mvcsj3n43s36nd7hjmgc2y-python3.6-jupyterlab-0.35.4/bin/jupyter-lab'...
wrapping `/nix/store/x0g0i3y075mvcsj3n43s36nd7hjmgc2y-python3.6-jupyterlab-0.35.4/bin/jlpm'...
/nix/store/7l5p0w4821cnyf0g3qpri4sq1mmclza5-python3.6-ipython-7.1.1
/nix/store/hl13kcjqdj2p60qyrrmkzlklkl3nh07j-python3.6-prompt_toolkit-2.0.7
/nix/store/2658f1j7yrcbkfw126wmxflpva0ccirl-python3.6-notebook-5.7.0
/nix/store/hl13kcjqdj2p60qyrrmkzlklkl3nh07j-python3.6-prompt_toolkit-2.0.7
/nix/store/kshdyxn4cda7bdh8iq3603y9bw4wsk1l-python3.6-ipykernel-5.1.0
/nix/store/kfhhhi11fzhls9pf79n9nqysycpgys8c-python3.6-jupyterlab_server-0.2.0
/nix/store/x0g0i3y075mvcsj3n43s36nd7hjmgc2y-python3.6-jupyterlab-0.35.4
/nix/store/f9c1sjc2vk30h1sqb76jf6q03slm647s-python3.6-jupyterhub-0.9.4

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python36Packages.ipython, python36Packages.prompt_toolkit, python36Packages.notebook, python36Packages.prompt_toolkit, python36Packages.ipykernel, python36Packages.jupyterlab_server, python36Packages.jupyterlab, python36Packages.jupyterhub

Partial log (click to expand)

wrapping `/nix/store/wgddy7vaqv26gzn5i8b5f1b58lnzci7g-python3.6-jupyterlab-0.35.4/bin/jupyter-labhub'...
wrapping `/nix/store/wgddy7vaqv26gzn5i8b5f1b58lnzci7g-python3.6-jupyterlab-0.35.4/bin/jlpm'...
/nix/store/q72mmmmyphbrkmm2z1zpa1xxrrjnq1hk-python3.6-ipython-7.1.1
/nix/store/hishzg08ilk33b54p21vd8dy5kfnihvr-python3.6-prompt_toolkit-2.0.7
/nix/store/rwff97q6hxxl47by39qwnsyq2jlxfq4l-python3.6-notebook-5.7.0
/nix/store/hishzg08ilk33b54p21vd8dy5kfnihvr-python3.6-prompt_toolkit-2.0.7
/nix/store/q4q5yhwqvqmm03aklzydv5nvgg3p3zph-python3.6-ipykernel-5.1.0
/nix/store/b92jrs0aiy6k7r1x5jqff8i0ffcp489f-python3.6-jupyterlab_server-0.2.0
/nix/store/wgddy7vaqv26gzn5i8b5f1b58lnzci7g-python3.6-jupyterlab-0.35.4
/nix/store/49xcck1af14bcls1j7qkbiw54yprz9xs-python3.6-jupyterhub-0.9.4

@Mic92
Copy link
Member

Mic92 commented Nov 15, 2018

@GrahamcOfBorg build python27Packages.jupyter python27Packages.jupyter_console

@GrahamcOfBorg
Copy link

Failure on x86_64-linux (full log)

Attempted: python27Packages.jupyter, python27Packages.jupyter_console

Partial log (click to expand)

checking for references to /build in /nix/store/2rvm35cnywsawfr49szvb71x473cvvsv-python2.7-ipywidgets-7.4.2...
running install tests
........................................
----------------------------------------------------------------------
Ran 40 tests in 0.112s

OK
pytestcachePhase
cannot build derivation '/nix/store/cpdhrch8r19l4cnan1jp44zsw9blqbss-python2.7-jupyter-1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/azp24d0jlj267k24mvwr4sy407016rxk-python2.7-jupyter_console-5.2.0.drv', '/nix/store/cpdhrch8r19l4cnan1jp44zsw9blqbss-python2.7-jupyter-1.0.0.drv' failed

@Mic92
Copy link
Member

Mic92 commented Nov 15, 2018

I get the same build errors.

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: python27Packages.jupyter, python27Packages.jupyter_console

Partial log (click to expand)

cannot build derivation '/nix/store/m8gk440hdgm9imkl5gr6xb3ray0d2rgf-python2.7-PyQt-5.10.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/rrg84r1643ikylqhryv5dxb86ynrdifx-python2.7-qtconsole-4.4.2.drv': 1 dependencies couldn't be built
........................................
----------------------------------------------------------------------
Ran 40 tests in 0.114s

OK
pytestcachePhase
cannot build derivation '/nix/store/lqqgzzgnqyqrkng4ibi27sfbv6d4sp13-python2.7-jupyter-1.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/98sqchc2rbhvwxgiqncc1y2dskwwh92s-python2.7-jupyter_console-5.2.0.drv', '/nix/store/lqqgzzgnqyqrkng4ibi27sfbv6d4sp13-python2.7-jupyter-1.0.0.drv' failed

@GrahamcOfBorg
Copy link

Failure on aarch64-linux (full log)

Attempted: python27Packages.jupyter, python27Packages.jupyter_console

Partial log (click to expand)

make[1]: *** [CMakeFiles/Makefile2:2325: Source/WebKit2/CMakeFiles/WebKit2.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 83%] Linking CXX static library ../../lib/libWebCoreTestSupport.a
[ 83%] Built target WebCoreTestSupport
make: *** [Makefile:163: all] Error 2
builder for '/nix/store/m00h6571r6ng5vg7hb0s0r2h361wqx6b-qtwebkit-5.212-alpha-01-26-2018.drv' failed with exit code 2
cannot build derivation '/nix/store/0qf7z12dqhmizn2qn6vfdzqxm76w8wr9-python2.7-PyQt-5.10.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/cjqip8zb46vq44lkvif2vy5aw0dgh1lb-python2.7-qtconsole-4.4.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5m82sa7ix3wp009qclw0gjfrcdpq5n3h-python2.7-jupyter-1.0.0.drv': 2 dependencies couldn't be built
error: build of '/nix/store/5m82sa7ix3wp009qclw0gjfrcdpq5n3h-python2.7-jupyter-1.0.0.drv', '/nix/store/sh1qxp5i17inm1r3chdszmahh8axvpzv-python2.7-jupyter_console-5.2.0.drv' failed

@costrouc
Copy link
Member Author

costrouc commented Nov 15, 2018

@Mic92 thanks for checking and catching that. I am fixing the error right now. It had to do with me rebasing with a recent commit that changed prompt_toolkit #49853.

I often resort to nix-build <path-to-nixpkgs> -A python36Packages.jupyter --option sandbox true for each package which can miss things. Is their an easier way to test out all my changes in a PR? E.g. test all python<version>Packages.<package> at once. I would really appreciate learning a better approach

@Mic92
Copy link
Member

Mic92 commented Nov 15, 2018

@costrouc
Copy link
Member Author

costrouc commented Nov 15, 2018

THANK YOU and written by none other than yourself. I guess I had seen that tool but never thought it was applicable to my use case (should have RTFM). I will do this to confirm my PRs in the future (and now)!

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python27Packages.ipython, python36Packages.jupyter_console

Partial log (click to expand)

/build/jupyter_console-6.0.0
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0/lib  /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0/bin
patching script interpreter paths in /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0
checking for references to /build in /nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0...
wrapping `/nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0/bin/jupyter-console'...
/nix/store/3s4ggi31iw4w9l52wzk37cylkm1qv5i7-python2.7-ipython-5.8.0
/nix/store/nxwng99pw7rpzg99jy5d71f48n2i428i-python3.6-jupyter_console-6.0.0

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python27Packages.ipython, python36Packages.jupyter_console

Partial log (click to expand)

/build/jupyter_console-6.0.0
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0
strip is /nix/store/p9akxn2sfy4wkhqdqa3li97pc6jaz3r1-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0/lib  /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0/bin
patching script interpreter paths in /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0
checking for references to /build in /nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0...
wrapping `/nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0/bin/jupyter-console'...
/nix/store/l085qn94y2q27j47cyjk3rcfr3lsibh1-python2.7-ipython-5.8.0
/nix/store/g6sclsvvgnjbkbm1xd7cfpy1a58pi22n-python3.6-jupyter_console-6.0.0

@costrouc
Copy link
Member Author

I will also be closing this along with two other PRs to combine with this one. #49807, #49721, and #50858.

This will lead to a PR with better python 3 support for jupyter (newest releases are python 3.5+ only). Addition of 3 kernels (c, go, ansible) and 20 jupyterlab extensions. Also will have an editor that has a clean way to add kernels and extensions. This will be a big PR but I am finding that they all depend on each other.

@costrouc costrouc closed this Nov 21, 2018
@FRidh
Copy link
Member

FRidh commented Nov 22, 2018

I seem to have forgotten to mention it, but these changes have landed in master.

@FRidh
Copy link
Member

FRidh commented Nov 22, 2018

An important change I made is that with Python 2 one always gets prompt_toolkit version 1, and with Python 3 always version 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants