Integration of kernels and extensions with jupyter #49807

costrouc · 2018-11-05T22:54:40Z

Motivation for this change

this main comment has been edited to reflect progress

Anyone who has used jupyterlab and notebooks has run into the issue of adding kernels and extensions to the UI. This is tricky because often times the kernels run in a different language e.g. julia, haskell, etc. Nix can handle this extremely well and would make jupyter notebook installations much easier. This PR provides intructions on a way to "auto-installkernels and extensions into the nix environment. Any input would be much appreciated. Here is what my research has shown me so far (I dont understand extensions yet wherestatic` directory is put).

Jupyter Kernels

The environment variable JUPYTER_PATH is a list of : separated paths that have a kernels directory. Every directory within contains kernels kernels/<kernel-name>/kernel.json. Here is an example kernel.json.

{
    "argv":["/nix/store/bwfygfcdvis9wd1c1v51xwnwhw1hx0a0-python3-3.6.6/bin/python3.6m","-m","jupyter_c_kernel","-f","{connection_file}"],
    "display_name":"C",
    "language":"c"
}

I would like to automatically add a set of kernels via the JUPYTER_PATH variable to the jupyterlab and notebook packages. Not sure exactly how to do this here is an example of me setting the variable after the fact in a nix-shell. It has been show below how to automatically do this but the issue is that kernels exist for many languages not just python. What I have settled on is that kernel.json and the images should be included at share/jupyter/kernels/<kernel-name>/.

JupyterLab Extensions

Jupyterlab is the future so I have put the most effort into installation instructions. Additionally notebook's installation paths are not as well documented and often times work without any additional steps. Jupyterlab has a command line option --app-dir and environment variable JUPYTERLAB_DIR that control where the extensions are installed usually share/jupyter/lab. These extensions are npm packages that after being combined through a package.json must be "compiled" with webpack to produce compressed static files. The npm package dependencies can easily be in the thousands... thanks Javascript leading to about 100MB - 1GB of dependencies that must be compiled for each additional extension added. This step cannot be pure unless we override the jupyterlab package management and handle all of the npm dependencies. Currently it seems that npm packages are not a good fit for nixpkgs. So I have come up with an unpure but working way to package jupyterlab extensions. Each package that comes with a jupyterlab extension uses passthru = { jupyterlabExtensions = [ .... ]; };. Then we include the following lines

# make temp directory for extensions
TEMPDIR=$(mktemp -d -p /tmp)
mkdir -p $TEMPDIR
# copy base jupyter lab setup
cp -r ${pkgs.python36Packages.jupyterlab}/share/jupyter/lab/* $TEMPDIR
chmod -R 755 $TEMPDIR

# gather unique jupyterlab extensions
# for each extension
#      jupyter labextension install --no-build --app-dir=$TEMPDIR <extension>
${pkgs.stdenv.lib.concatMapStrings
     (s: "jupyter labextension install --no-build --app-dir=$TEMPDIR ${s}; ")
     (pkgs.lib.unique
       ((pkgs.lib.concatMap
           (d: pkgs.lib.attrByPath ["passthru" "jupyterlabExtensions"] [] d)
           buildInputs) ++ additionalExtensions))  }
# finally build jupyterlab npm extensions with webpack
jupyter lab build --app-dir=$TEMPDIR

# start jupyterlab service
jupyter lab --app-dir=$TEMPDIR

Working Example with 20 jupyerlab extension + 4 kernels (c, python, go, ansible)

All that needs to be edited by user is kernels, additionalExtensions, and buildInputs. The rest is automatic and will launch a jupyterlab instance for you.

shell.nix

{ pkgs ? import <nixpkgs> {}, pythonPackages ? pkgs.python36Packages }:

let kernels = [
      pkgs.python36Packages.ansible-kernel
      pythonPackages.jupyter-c-kernel
      pkgs.gophernotes
    ];

    additionalExtensions = [
      "@jupyterlab/toc"
      "@jupyterlab/fasta-extension"
      "@jupyterlab/geojson-extension"
      "@jupyterlab/katex-extension"
      "@jupyterlab/mathjax3-extension"
      "@jupyterlab/plotly-extension"
      "@jupyterlab/vega2-extension"
      "@jupyterlab/vega3-extension"
      "@jupyterlab/xkcd-extension"
      "jupyterlab-drawio"
      "@jupyterlab/hub-extension"
      "jupyterlab_bokeh"
    ];
in
pkgs.mkShell rec {
  buildInputs = [
    ### Base Packages
    pythonPackages.jupyterlab pkgs.nodejs

    ### Extensions
    pythonPackages.ipywidgets
    pythonPackages.ipydatawidgets
    pythonPackages.ipywebrtc
    pythonPackages.pythreejs
    pythonPackages.ipyvolume
    pythonPackages.jupyterlab-git
    pythonPackages.jupyterlab-latex
    pythonPackages.ipyleaflet
    pythonPackages.ipympl
  ] ++ kernels;

  shellHook = ''
    TEMPDIR=$(mktemp -d -p /tmp)
    mkdir -p $TEMPDIR
    cp -r ${pkgs.python36Packages.jupyterlab}/share/jupyter/lab/* $TEMPDIR
    chmod -R 755 $TEMPDIR
    echo "$TEMPDIR is the app directory"

    # kernels
    export JUPYTER_PATH="${pkgs.lib.concatMapStringsSep ":" (p: "${p}/share/jupyter/") kernels}"

# labextensions
${pkgs.stdenv.lib.concatMapStrings
     (s: "jupyter labextension install --no-build --app-dir=$TEMPDIR ${s}; ")
     (pkgs.lib.unique
       ((pkgs.lib.concatMap
           (d: pkgs.lib.attrByPath ["passthru" "jupyterlabExtensions"] [] d)
           buildInputs) ++ additionalExtensions))  }
jupyter lab build --app-dir=$TEMPDIR

# start jupyterlab
jupyter lab --app-dir=$TEMPDIR
  '';

}

Things done

Not all of these kernels work currently (gnuplot, zeus-cling)

Adding four kernels:

Want to add extension support for python/jupyterlab extension:

ipywebrtc (webrtc in jupyterlab)
pythreejs (three.js in jupyterlab)
ipyvolume (3d volumetric rendering in jupyterlab(
jupyterlab-latex (latex document editing in jupyterlab)
ipywidgets (widgets alot of packages depend on this one)
jupyterlab-git (git support)
ipyleaflet (leaflet extension)
ipympl (matplotlib extension)
ipydatawidgets (A set of widgets to help facilitate reuse of large datasets)

This will require input on how to properly include extensions and kernels within jupyter derivations.

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

FRidh · 2018-11-06T16:04:56Z

You have seen the top-level jupyter and jupyter-kernel attributes that are used by the jupyter NixOS module?

costrouc · 2018-11-06T16:17:08Z

I had not seen that before and thank you for pointing that out! However, I see some issues with declaring this in nixos modules in that only nixos can use that. Ideally I would like to say "start a jupyter notebook with these 3 kernels (julia, C, and python) and these 3 extensions (ipywidget, ipyvolume, jupyterlab-drawio)" for example in any linux distribution. This currently is not possible and a pain point I have had with jupyter. Welcome to any ideas.

I have learned that the JUPYTER_PATH environment variable can point to each kenel json definition and JUPYTERLAB_DIR points to the javascript extensions static files. This will require a meta package or something to collect all of the extensions in one location and set JUPYTER_PATH correctly.

FRidh · 2018-11-06T16:35:41Z

I had not seen that before and thank you for pointing that out! However, I see some issues with declaring this in nixos modules in that only nixos can use that. Ideally I would like to say "start a jupyter notebook with these 3 kernels (julia, C, and python) and these 3 extensions (ipywidget, ipyvolume, jupyterlab-drawio)" for example in any linux distribution. This currently is not possible and a pain point I have had with jupyter. Welcome to any ideas.

The NixOS module only provides an interface and a service, but the actual logic for running a notebook with specified kernels is implemented in the two top-level attributes I referred to.

FRidh · 2018-11-06T16:38:43Z

Original PR #33673

costrouc · 2018-11-06T16:41:03Z

Thank you and I should read more carefully the expressions. next time. I will try it out with the new kernels with the derivations and also would be interested in adding extensions support to the nixpkgs/pkgs/applications/editors/jupyter/default.nix.

FRidh · 2018-11-06T16:41:28Z

pkgs/development/python-modules/gnuplot-kernel/default.nix

+  };
+
+  checkInputs = [ pytest ];
+  propagatedBuildInputs = [ notebook metakernel gnuplot ];


Should we not somewhere hardcode a path to gnuplot?

Yes we should. Currently I am trying to get the gnuplot_kernel to work. It freezes on me for an unknown reason.

veprbl · 2018-11-06T16:54:48Z

I would like to automatically add a set of kernels via the JUPYTER_PATH variable to the jupyterlab and notebook packages.

This should be easy to achieve by adding a setupHook to jupyterlab.
Examples are

cmake:

nixpkgs/pkgs/development/tools/build-managers/cmake/setup-hook.sh

Lines 1 to 3 in a6809c0

addCMakeParams() {

addToSearchPath CMAKE_PREFIX_PATH $1

}

nixpkgs/pkgs/development/tools/build-managers/cmake/setup-hook.sh

Line 94 in a6809c0

addEnvHooks "$targetOffset" addCMakeParams

python:

nixpkgs/pkgs/development/interpreters/python/setup-hook.sh

Lines 1 to 3 in a6809c0

    
           addPythonPath() { 
        
               addToSearchPathWithCustomDelimiter : PYTHONPATH $1/@sitePackages@ 
        
           }

nixpkgs/pkgs/development/interpreters/python/setup-hook.sh

Line 15 in a6809c0

addEnvHooks "$hostOffset" addPythonPath

costrouc · 2018-11-07T14:00:29Z

ansible-kernel depends on #49764 being move to python-modules for python 2.7 and 3+ support.

Here is the error I get (due to using python2.7 for 3 packages)

  File "/nix/store/bwfygfcdvis9wd1c1v51xwnwhw1hx0a0-python3-3.6.6/lib/python3.6/re.py", line 142, in <module>
    class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'

costrouc · 2018-11-12T17:42:13Z

@veprbl I am not very knowledgeable on how to properly do this. Could you give some further advice? Additionally I should state that jupyterlab kernels are not necessarily for python only. For example this PR adds support for Go and C.

costrouc · 2018-11-12T17:51:05Z

I would like some feedback of using passthru. Otherwise I do not see anything controversial about this PR.

...
passthru ={
   jupyterlabExtensions = [ "jupyterlab-datawidgets" "@jupyter-widgets/jupyterlab-manager
" ];
};
...

Is the correct way to pass that the package requires an npm dependency that cannot be included in the build requirements. Is this a good way to pass these dependencies? This expression is then used to select the necessary extensions to install (additionallExtensions is a list of strings of extensions that do not require a python package).

(pkgs.lib.unique
       ((pkgs.lib.concatMap
           (d: pkgs.lib.attrByPath ["passthru" "jupyterlabExtensions"] [] d)
           buildInputs) ++ additionalExtensions))

costrouc · 2018-11-20T16:54:20Z

I was unaware of link and how it works but if it were able to isolate each extension this that might just solve it. Looks promising. I would really appreciate help in this area but I will be looking into it as well. Anything we can do to make the extensions a derivation is perfect

edit: @MMesch after looking more carefully https://github.com/jupyterlab/jupyterlab/blob/master/jupyterlab/commands.py#L659-L684 it looks link jupyter labextension link <path to extension>. Would work. I would like to work on this but I think that it deserves another PR. As this PRs goal was to add some of the packages. I will open another PR that we should discus in.

This PR is still ready for merge.

Extensions: - ipyvolume - @jupyter-widgets/jupyterlab-manager - @jupyterlab/fasta-extension - @jupyterlab/geojson-extension - @jupyterlab/git - @jupyterlab/hub-extension - @jupyterlab/katex-extension - @jupyterlab/latex - @jupyterlab/mathjax3-extension - @jupyterlab/plotly-extension - @jupyterlab/toc - @jupyterlab/vega2-extension - @jupyterlab/vega3-extension - @jupyterlab/xkcd-extension - jupyter-leaflet - jupyter-matplotlib - jupyter-threejs - jupyter-webrtc - jupyterlab-datawidgets - jupyterlab-drawio

MMesch · 2018-11-20T21:47:32Z

agree. I'll help you with the other PR ...

GrahamcOfBorg · 2018-11-21T16:17:10Z

Success on aarch64-linux (full log)

Attempted: gophernotes

Partial log (click to expand)

}
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin
shrinking /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin/bin/gophernotes
strip is /nix/store/6dpnd5aniypn8124mmy8f88s4mq2zl07-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin/bin
patching script interpreter paths in /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin
checking for references to /build in /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin...
strip is /nix/store/6dpnd5aniypn8124mmy8f88s4mq2zl07-binutils-2.30/bin/strip
/nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin

GrahamcOfBorg · 2018-11-21T16:18:28Z

Success on x86_64-linux (full log)

Attempted: gophernotes

Partial log (click to expand)

}
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin
shrinking /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin/bin/gophernotes
strip is /nix/store/rpbg8gmqxhz8g61p1plz5d2srs84pvmv-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin/bin
patching script interpreter paths in /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin
checking for references to /build in /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin...
strip is /nix/store/rpbg8gmqxhz8g61p1plz5d2srs84pvmv-binutils-2.30/bin/strip
/nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin

costrouc · 2018-11-21T18:10:49Z

I will also be closing this along with two other PRs to combine with this one. #49807, #49721, and #50858.

This will lead to a PR with better python 3 support for jupyter (newest releases are python 3.5+ only). Addition of 3 kernels (c, go, ansible) and 20 jupyterlab extensions. Also will have an editor that has a clean way to add kernels and extensions. This will be a big PR but I am finding that they all depend on each other.

ariutta · 2019-12-02T21:49:36Z

This will lead to a PR...

Did the new PR land?

Related: it appears similar functionality can be found in the JupyterWith repo.

costrouc requested a review from FRidh as a code owner November 5, 2018 22:54

GrahamcOfBorg added 6.topic: python 8.has: package (new) 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Nov 5, 2018

FRidh reviewed Nov 6, 2018

View reviewed changes

costrouc force-pushed the costrouc/python-jupyter-kernels branch from cbad6bc to f6177a9 Compare November 6, 2018 21:27

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 and removed 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Nov 6, 2018

costrouc force-pushed the costrouc/python-jupyter-kernels branch from 14f120f to 5a81da3 Compare November 7, 2018 02:16

costrouc mentioned this pull request Nov 7, 2018

Update jupyter packages to python 3 only packages with 27 support #49721

Closed

9 tasks

costrouc force-pushed the costrouc/python-jupyter-kernels branch 3 times, most recently from fc9bc5f to 6ac8cf3 Compare November 12, 2018 17:15

costrouc changed the title ~~[WIP] Integration of kernels and extensions with jupyter~~ Integration of kernels and extensions with jupyter Nov 12, 2018

costrouc mentioned this pull request Nov 20, 2018

[WIP] jupyter editor including extensions #50858

Closed

11 tasks

costrouc added 20 commits November 21, 2018 10:59

pythonPackages.ansible-kernel: init at 0.8.0

9398f2b

pythonPackages.ansible-runner: init at 1.1.2

adf72b7

pythonPackages.jupyter-c-kernel: init at 1.2.2

219a7a6

gophernotes: init at 1.0.0-RC1

8acfc9e

pythonPackages.nbval: init at 0.9.1

da894d1

pythonPackages.nbdime: init at 1.0.3

01aee3b

pythonPackages.ipywebrtc: init at 0.4.1

99bc744

pythonPackages.pythreejs: init at 2.0.2

b2d12cb

pythonPAckages.traittypes: init at 0.2.1

ab8b97b

pythonPackages.pytest-tornado5: init at 1.0.0

a0e9d8a

pythonPackages.ipydatawidgets: init at 4.0.0

692408d

pythonPackages.ipyvolume: init at 0.5.1

c6e777c

pythonPackages.jupytext: init at 0.8.4

80bfba8

pythonPackages.asteval: init at 0.9.13

5367622

pythonPackages.lmfit: init at 0.9.11

12fee2d

pythonPackages.jupyterlab-latex: init at 0.4.1

8285c06

pythonPackages.ipywidgets: refactor add jupyterlab extension

8fbbd7c

pythonPackages.jupyterlab-git: init at 0.3.0

dba4ef6

pythonPackages.ipyleaflet: init at 0.9.1

6519626

pythonPackages.ipympl: init at 0.2.1

436bba0

costrouc force-pushed the costrouc/python-jupyter-kernels branch from a6fb7ce to 436bba0 Compare November 21, 2018 16:00

costrouc closed this Nov 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration of kernels and extensions with jupyter #49807

Integration of kernels and extensions with jupyter #49807

costrouc commented Nov 5, 2018 •

edited

FRidh commented Nov 6, 2018

costrouc commented Nov 6, 2018 •

edited

FRidh commented Nov 6, 2018 •

edited

FRidh commented Nov 6, 2018

costrouc commented Nov 6, 2018 •

edited

FRidh Nov 6, 2018

costrouc Nov 6, 2018

veprbl commented Nov 6, 2018

costrouc commented Nov 7, 2018

costrouc commented Nov 12, 2018

costrouc commented Nov 12, 2018 •

edited

costrouc commented Nov 20, 2018 •

edited

MMesch commented Nov 20, 2018 •

edited

GrahamcOfBorg commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

costrouc commented Nov 21, 2018

ariutta commented Dec 2, 2019

Integration of kernels and extensions with jupyter #49807

Integration of kernels and extensions with jupyter #49807

Conversation

costrouc commented Nov 5, 2018 • edited

Motivation for this change

Jupyter Kernels

JupyterLab Extensions

Working Example with 20 jupyerlab extension + 4 kernels (c, python, go, ansible)

Things done

FRidh commented Nov 6, 2018

costrouc commented Nov 6, 2018 • edited

FRidh commented Nov 6, 2018 • edited

FRidh commented Nov 6, 2018

costrouc commented Nov 6, 2018 • edited

FRidh Nov 6, 2018

Choose a reason for hiding this comment

costrouc Nov 6, 2018

Choose a reason for hiding this comment

veprbl commented Nov 6, 2018

costrouc commented Nov 7, 2018

costrouc commented Nov 12, 2018

costrouc commented Nov 12, 2018 • edited

costrouc commented Nov 20, 2018 • edited

MMesch commented Nov 20, 2018 • edited

GrahamcOfBorg commented Nov 21, 2018

GrahamcOfBorg commented Nov 21, 2018

costrouc commented Nov 21, 2018

ariutta commented Dec 2, 2019

costrouc commented Nov 5, 2018 •

edited

costrouc commented Nov 6, 2018 •

edited

FRidh commented Nov 6, 2018 •

edited

costrouc commented Nov 6, 2018 •

edited

costrouc commented Nov 12, 2018 •

edited

costrouc commented Nov 20, 2018 •

edited

MMesch commented Nov 20, 2018 •

edited