nixos/jupyter: init service #33673

aborsu · 2018-01-09T21:32:29Z

Motivation for this change

I believe that having a configurable service for jupyter with a template to write new kernels is a wanted feature.
cfr
https://www.reddit.com/r/NixOS/comments/7oujo4/getting_all_the_jupyter_kernels/
#10713

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

This pull requests brings a jupyter service to nixos.
It contains a default bare python3 kernel and a literal example on how to define multiple kernels.

aborsu · 2018-01-09T22:59:04Z

@FRidh @ixxie Any feedback?

FRidh · 2018-01-10T13:14:56Z

Related issue: #10713

FRidh · 2018-01-10T13:15:54Z

nixos/modules/services/development/jupyter.nix

+      example = "users";
+    };
+
+    password = mkOption {


I think it is better to keep this out of the store. Maybe you can point to a path and read that during runtime?

I made a compromise on that, that I think should be agreeable to all.

FRidh · 2018-01-10T13:16:45Z

nixos/modules/services/development/jupyter.nix

+            type = types.package;
+            default = (pkgs.python36.withPackages (
+              pythonPackages: with pythonPackages; [
+                notebook


ipykernel should be sufficient as the notebook should already be offered by this service

Indeed, good catch

FRidh · 2018-01-10T13:17:10Z

nixos/modules/services/development/jupyter.nix

+          "python3" = {
+            name = "Python 3 for Machine Learning";
+            env = (pkgs.python36.withPackages (pythonPackages: with pythonPackages; [
+              notebook


FRidh · 2018-01-11T09:03:58Z

I think the logic of creating a jupyter notebook with a set of kernels can be factored out of the module and into a function in Nixpkgs.

aborsu · 2018-01-11T23:18:16Z

@FRidh I agree, but it was a bit more work.
I think it's pretty good for now.
We could provide a few additional kernels but I think this should be done in other PR.

aborsu · 2018-01-18T20:21:34Z

@FRidh @globin
I've made a seperate package with the kernel logic, any feedback?
Let me know if you want me to change anything.
I've tested it on my server but so far only with Python kernels.
It would be great to have a Julia and R kernel so we could have the Ju Pyt R but i'm not very good in those two languages.

FRidh

I think it is good to have a separate derivation for the kernel config, and having a new top level item for Jupyter. However, instead of making it a derivation that needs to be overriden, I suggest making jupyter a function:

kernelDefinitions: { the, other, args }: derivation

FRidh · 2018-01-20T14:00:09Z

pkgs/applications/editors/jupyter/default.nix

+, lib
+, pkgs
+, python3
+, python3Packages 


python3 or python3Packages

FRidh · 2018-01-20T14:00:19Z

pkgs/applications/editors/jupyter/default.nix

@@ -0,0 +1,21 @@
+{ stdenv
+, lib
+, pkgs


FRidh · 2018-01-20T14:02:32Z

pkgs/applications/editors/jupyter/default.nix

+python3Packages.buildPythonApplication rec {
+  inherit (python3Packages.notebook) pname version name src LC_ALL buildInputs propagatedBuildInputs checkPhase doCheck meta;
+
+  makeWrapperArgs = ["--set JUPYTER_PATH ${jupyterPath}"];


let ... in with python3.pkgs; toPythonModule (notebook.override { makeWrapperArgs = ["--set JUPYTER_PATH ${jupyterPath}"]; });

toPythonModule (buildPythonPackage {...}) is basically the same as buildPythonApplication. In this case I would go for the former, because the current approach requires keeping up with required attributes.

FRidh · 2018-01-20T14:05:41Z

pkgs/development/misc/jupyterKernels/default.nix

@@ -0,0 +1,66 @@
+{pkgs, python3, lib, kernelDefinitions ? null }:


FRidh · 2018-01-20T14:06:18Z

pkgs/development/misc/jupyterKernels/default.nix

@@ -0,0 +1,66 @@
+{pkgs, python3, lib, kernelDefinitions ? null }:
+


let defaultKernelDefinitions = { ... }; in {python3, lib, kernelDefinitions ? defaultKernelDefinitions }:

FRidh · 2018-01-20T14:06:32Z

pkgs/development/misc/jupyterKernels/default.nix

+let
+  kernels = if kernelDefinitions != null
+    then kernelDefinitions
+    else {


this is then defaultKernelDefinitions

FRidh · 2018-01-20T14:06:45Z

pkgs/development/misc/jupyterKernels/default.nix

+          "{connection_file}"
+        ];
+        language = "python";
+        logo32 = "${env}/lib/python3.6/site-packages/ipykernel/resources/logo-32x32.png";


env.sitePackages

aborsu · 2018-01-21T19:20:36Z

@FRidh
I didn't change everything exactly like you asked.

I'm not convinced making the top package jupyter a function is such a great idea.
The error message that I got when running nix-env -i jupyter or trying to add jupyter in my environment.systemPackages without giving it a kernelDefinition where unhelpful and I think will lead to a lot of head scratching by first time users.

error: The option value `environment.systemPackages.[definition 1-entry 1]' in `...configuration.nix' is not of type `package'.

Even if I give a default argument, the function still needs to be called.

It seems to me that override is a standard way of providing customizability to packages and will avoid a lot of pain while imo not reducing understandability or ease of use.

As always I'm ready to discuss this further. And thank you for your helpfull corrections.

aborsu · 2018-01-31T19:27:08Z

@FRidh ping?

FRidh · 2018-01-23T16:48:52Z

pkgs/applications/editors/jupyter/default.nix

@@ -0,0 +1,16 @@
+{ pkgs


pkgs is not used, adding jupyterKernels is better

FRidh · 2018-01-23T16:49:54Z

pkgs/development/misc/jupyterKernels/default.nix

+
+in
+
+kernelDefinitions: (


{ kernelDefinitions ? defaultKernelDefinitions }:

FRidh · 2018-01-23T16:50:34Z

pkgs/development/misc/jupyterKernels/default.nix

+    in {
+      displayName = "Python 3";
+      argv = [
+        "${env}/bin/python"


${env.interpreter}

FRidh · 2018-01-23T16:51:19Z

pkgs/development/misc/jupyterKernels/default.nix

@@ -0,0 +1,88 @@
+{pkgs, lib}:


I don't think pkgs is used

aborsu · 2018-01-31T20:52:36Z

pkgs/development/misc/jupyterKernels/default.nix

+in
+{
+  defaultKernelDefinition = defaultKernelDefinition;
+  create = { kernelDefinitions ?  defaultKernelDefinition }:


@FRidh I'm not certain this is better.
Now I'm exporting a set instead of exporting a function so I can re-use the default value in all the packages that might use it.

aborsu · 2018-01-31T20:52:46Z

pkgs/applications/editors/jupyter/default.nix

@@ -0,0 +1,16 @@
+{ python3
+, jupyterKernels
+, kernelDefinitions ? jupyterKernels.defaultKernelDefinition


This is neat.

aborsu · 2018-01-31T20:55:03Z

nixos/modules/services/development/jupyter/default.nix

+
+
+  kernels = (pkgs.jupyterKernels.create  {
+    kernelDefinitions = if cfg.kernels != null


I've basically had to move the null check here now.
I though of passing the pkgs.jupyterKernels.defaultKernelConfig as the default value for jupyter.kernels, but the it complains that it contains a reference to a derivation. (I can understand that having to build a derivation to create the doc is not the greatest).

aborsu · 2018-02-03T13:59:12Z

@FRidh opinion?

schneefux

compiles and runs basic examples

dustinlacewell-wk · 2018-02-22T22:34:31Z

Mind providing an example of how to get Jupyter installed with these changes?

schneefux · 2018-03-03T15:16:50Z

Here is my configuration (the password is "test", change it):

  services.jupyter = {
    enable = true;
    port = 8123;
    notebookDir = "/var/www/jupyter";
    password = "'sha1:1b961dc713fb:88483270a63e57d18d43cf337e629539de1436ba'";
  };

schneefux · 2018-03-03T15:20:42Z

pkgs/development/misc/jupyterKernels/default.nix

+      env = (pkgs.python3.withPackages (pythonPackages: with pythonPackages; [
+              ipykernel
+              pandas
+              scikitlearn


Are these Jupyter dependencies or just nice to have packages?

ipykernel is needed, the others are just basic packages used for data science.
I was thinking of getting something inline with the jupyter dockerimages.
Base python, base data-science, data-science + spark

Keep the default as minimal as possible.

ixxie · 2018-04-19T12:54:23Z

I plan to draft a sort of RFC issue for Nixpkgs where we could coordinate data science infrastructure work. I would like to provide an overview or stuff related to the Jupyter ecosystem, Python data science ecosystem and Haskell data science ecosystem; perhaps other stuff too, but scoping this is a little tricky, so I think that is an okay start. I plan to make a script to query nix search (which can return JSONs now) for a package list. datahaskell.org provides a nice overview of interesting Haskell stuff but I would still need to find a decent Python list.

aborsu · 2018-04-19T18:15:43Z

@ixxie for python, using the packages used by the jupyter docker images would be a good start https://github.com/jupyter/docker-stacks/tree/master/datascience-notebook

ixxie · 2018-04-25T17:30:26Z

@aborsu thanks for the tip

Sorry everybody about that offtopic-ish post; I didn't realize I could reply to an email in my mail client and it would post to GitHub!

phile314-fh · 2018-05-14T14:57:40Z

I have been using this for the last few weeks and it works nicely 👍

ixxie · 2018-05-22T17:23:01Z

I got this working on my machine, except setting notebookDir to anything but "~/" seems to break the service for some reason.

phile314-fh · 2018-05-23T05:58:03Z

I have set notebookDir to /var/www/jupyter and it works fine for me. I don't remember though if I created the notebookDir by hand and set the permissions accordingly or if the jupyter service did it automatically.

ixxie · 2018-05-24T17:48:20Z

@phile314-fh - looks like you did it manually. With @cleverca22's help I managed to resolve this by adding:

    serviceConfig.PermissionsStartOnly = true

and

    preStart = ''
      if [ ! -d ${cfg.notebookDir} ]; then
        mkdir -p ${cfg.notebookDir}
      fi
      chown ${cfg.user}:${cfg.group} ${cfg.notebookDir}
    '';

to systemd.services.jupyter.

@aborsu - its probably easiest if you add it yourself to the PR?

aborsu · 2018-05-27T17:13:52Z

Hello, sorry, I've been a bit busy.
@FRidh any comments before I start hacking this again?

ixxie · 2018-05-29T16:45:11Z

@aborsu I think I will soon use this as a template to create a Jupyterlab service and it occurred to me that if that would go in nixos/modules/services/development/jupyterlab/default.nix then it would make sense for kernel-options.nix to live somewhere else, maybe:

jupyter/default.nix
jupyterlab/default.nix
jupyterhub/default.nix
jupyterkernels/default.nix      # = kernel-options.nix

Alternatively we can have:

jupyter/
 |-default.nix
 |-kernels/default.nix          #  or just kernel-options.nix
 |-hub/default.nix
 |-lab/default.nix

or is this a bit awkward?

FRidh

I don't really have the time to review this any further, so please ask also other maintainers if they can help it forward / get it in.

FRidh · 2018-05-29T17:12:12Z

pkgs/development/misc/jupyterKernels/default.nix

+      env = (pkgs.python3.withPackages (pythonPackages: with pythonPackages; [
+              ipykernel
+              pandas
+              scikitlearn


Keep the default as minimal as possible.

FRidh · 2018-05-29T17:14:01Z

nixos/modules/services/development/jupyter/default.nix

+
+        wantedBy = [ "multi-user.target" ];
+
+        path = [ pkgs.bash ]; # needed for sh in cell magic to work


Have you considered patching notebook to hardcode bash?

FRidh · 2018-05-29T17:15:39Z

pkgs/applications/editors/jupyter/default.nix

+
+let
+
+  jupyterPath = (jupyterKernels.create  { kernelDefinitions =  kernelDefinitions; });


inherit / space

FRidh · 2018-05-29T17:16:14Z

nixos/modules/services/development/jupyter/kernel-options.nix

+    argv = mkOption {
+      type = types.listOf types.str;
+      example = [
+        "{customEnv}/bin/python"


${customEnv.interpreter}

FRidh · 2018-05-29T17:16:23Z

nixos/modules/services/development/jupyter/kernel-options.nix

+    logo32 = mkOption {
+      type = types.nullOr types.path;
+      default = null;
+      example = "{env}/lib/python3.6/site-packages/ipykernel/resources/logo-32x32.png";


${env.sitePackages}

FRidh · 2018-05-29T17:17:39Z

pkgs/applications/editors/jupyter/default.nix

+in
+
+with python3.pkgs; toPythonModule (
+  notebook.overrideAttrs(oldAttrs: {


overridePythonAttrs

FRidh · 2018-05-29T17:18:23Z

pkgs/development/misc/jupyterKernels/default.nix

@@ -0,0 +1,76 @@
+{ lib, pkgs, stdenv}:


no pkgs, but python3

FRidh · 2018-05-29T17:19:07Z

pkgs/development/misc/jupyterKernels/default.nix

+    meta = {
+      description = "Wrapper to create jupyter notebook kernel definitions";
+      homepage = http://jupyter.org/;
+      # NIXOS license as this is a nixos meta package.


FRidh · 2018-05-29T17:19:19Z

pkgs/development/misc/jupyterKernels/default.nix

+
+    src = "/dev/null";
+
+    unpackCmd ="mkdir jupyter_kernels";


FRidh · 2018-05-29T17:19:24Z

pkgs/development/misc/jupyterKernels/default.nix

+
+    unpackCmd ="mkdir jupyter_kernels";
+
+    installPhase =  ''


ixxie · 2018-05-29T18:07:15Z

@dotlambda perhaps you have time to have a look?

dotlambda · 2018-05-29T21:03:34Z

Not until Sunday or even next week, sadly.

ixxie · 2018-06-01T22:24:16Z

I was wondering if you could provide a sequence of recent PRs some context and have a Nixpkgs data science workgroup so I made a data science workgroup page on the wiki and #nixos-data on freenode.

dotlambda · 2018-06-05T07:30:32Z

@ixxie Please fix the things FRidh mentioned. I'll then take a look. However, I've never worked with Jupyter on NixOS so I might need some time to read through it.

ixxie · 2018-06-06T16:53:19Z

@dotlambda well this is @aborsu's PR really and I believe he is on it; if he doesn't have time eventually I can take over, but this is great work and I would hate to steal the credit ;)

I am working on the Jupyterlab module (no PR for that yet).

teto · 2018-08-02T15:53:53Z

Really looking forward to this. Keep up the good work @aborsu :D

FRidh · 2018-08-26T10:01:35Z

I've made some minor changes and pushed to master 4d3ce5c.

teto · 2018-08-27T04:53:04Z

I'll try to give some feedback on the feature, has anyone tried it with an haskell kernel already ?
The jupyter service might deserve a paragraph in the manual (at least a mention in changelog)

timokau · 2018-11-25T14:58:16Z

In #51030 I add jupyter support for sage. I've added it so that sage -n jupyter can start a jupyter notebook with the sagemath kernel. I'm not sure how that would integrate with this service, but I've used the kernel spec definition from here and made it available as sage.kernelspec. So if someone wants to integrate that into the jupyter service, that'd be nice.

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jan 9, 2018

FRidh requested changes Jan 10, 2018

View reviewed changes

FRidh mentioned this pull request Jan 10, 2018

Jupyter Notebook service #10713

Closed

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jan 11, 2018

FRidh requested changes Jan 20, 2018

View reviewed changes

FRidh added 6.topic: python 2.status: work-in-progress labels Jan 20, 2018

FRidh self-assigned this Jan 20, 2018

FRidh requested changes Jan 31, 2018

View reviewed changes

aborsu commented Jan 31, 2018

View reviewed changes

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) 8.has: package (new) labels Feb 15, 2018

schneefux mentioned this pull request Feb 21, 2018

jupyter: add ijavascript kernel #35290

Closed

8 tasks

schneefux approved these changes Feb 21, 2018

View reviewed changes

schneefux reviewed Mar 3, 2018

View reviewed changes

GrahamcOfBorg removed 8.has: changelog 8.has: documentation 2.status: merge conflict labels Apr 14, 2018

ttuegel removed their request for review April 14, 2018 14:52

timokau mentioned this pull request Apr 15, 2018

pythonPackages.python-openid: init at 2.2.5 #38788

Closed

8 tasks

zimbatm removed their request for review April 19, 2018 15:47

FRidh reviewed May 29, 2018

View reviewed changes

FRidh closed this Aug 26, 2018

teto mentioned this pull request Aug 28, 2018

New jupyter service #45697

Closed

timokau mentioned this pull request Oct 16, 2018

sage jupyter notebook not working #48544

Closed

FRidh mentioned this pull request Nov 6, 2018

Integration of kernels and extensions with jupyter #49807

Closed

12 tasks

Centaurioun mentioned this pull request Mar 2, 2023

[Snyk] Fix for 15 vulnerabilities Centaurioun/nixpkgs#137

Merged

		@@ -0,0 +1,66 @@
		{pkgs, python3, lib, kernelDefinitions ? null }:



		kernels = (pkgs.jupyterKernels.create {
		kernelDefinitions = if cfg.kernels != null


		wantedBy = [ "multi-user.target" ];

		path = [ pkgs.bash ]; # needed for sh in cell magic to work


		let

		jupyterPath = (jupyterKernels.create { kernelDefinitions = kernelDefinitions; });

nixos/jupyter: init service #33673

nixos/jupyter: init service #33673

Conversation

aborsu commented Jan 9, 2018 • edited

Motivation for this change

Things done

aborsu commented Jan 9, 2018 • edited

FRidh commented Jan 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FRidh commented Jan 11, 2018

aborsu commented Jan 11, 2018

aborsu commented Jan 18, 2018

FRidh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FRidh Jan 20, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aborsu commented Jan 21, 2018 • edited

aborsu commented Jan 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aborsu commented Feb 3, 2018

schneefux left a comment

Choose a reason for hiding this comment

dustinlacewell-wk commented Feb 22, 2018

schneefux commented Mar 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ixxie commented Apr 19, 2018 via email • edited

aborsu commented Apr 19, 2018

ixxie commented Apr 25, 2018

phile314-fh commented May 14, 2018

ixxie commented May 22, 2018 • edited

phile314-fh commented May 23, 2018

ixxie commented May 24, 2018 • edited

aborsu commented May 27, 2018

ixxie commented May 29, 2018 • edited

FRidh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ixxie commented May 29, 2018

dotlambda commented May 29, 2018

ixxie commented Jun 1, 2018

dotlambda commented Jun 5, 2018

ixxie commented Jun 6, 2018 • edited

teto commented Aug 2, 2018

FRidh commented Aug 26, 2018

teto commented Aug 27, 2018

timokau commented Nov 25, 2018

aborsu commented Jan 9, 2018 •

edited

aborsu commented Jan 9, 2018 •

edited

FRidh Jan 20, 2018 •

edited

aborsu commented Jan 21, 2018 •

edited

ixxie commented Apr 19, 2018 via email •

edited

ixxie commented May 22, 2018 •

edited

ixxie commented May 24, 2018 •

edited

ixxie commented May 29, 2018 •

edited

ixxie commented Jun 6, 2018 •

edited