New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nixos/jupyterhub: init service #94783
Conversation
I think where possible the Please also ping other interested parties such as past contributors to the |
@FRidh completely agree. Also some context about what jupyterhub is since I had some misconceptions about it as well. Jupyterhub has the unfortunate name of "jupyter" in it because it is not restricted to the python/jupyter ecosystem. It is really a standard way to Also I have no plan to work on getting jupyterlab extensions working here but I do want to be able to support kernels in the many different languages. cc: @kristoff3r, @MMesch, @guaraqe (hope I didn't leave anyone else out) |
afee50f
to
d060a31
Compare
d060a31
to
db82ca1
Compare
The current implementation is enough to test that things work. Still need to add the kernel support done in the jupyter module.
let pkgs = import <nixpkgs> { };
in {
machine = {
deployment.targetEnv = "libvirtd";
deployment.libvirtd.imageDir = "/var/lib/libvirt/images";
services.jupyterhub = {
enable = true;
port = 80;
kernels = import ./kernels.nix { inherit pkgs; };
};
networking.firewall.allowedTCPPorts = [ 80 ];
security.pam.services.login.setLoginUid = false;
users.users.costrouc = {
isNormalUser = true;
home = "/home/costrouc";
description = "Chris Ostrouchov";
extraGroups = [ "wheel"];
hashedPassword = "$6$og2KAu0wj$ce4zFxN5bd0fz1q/jTDO6o.wcxtm6lhXOZ.Dno34W5Rih1IJVjyiRJiIIqVxNF309XK0cP.F0eNu7ttW5qF8N.";
};
};
}
{ pkgs }:
{
python27Kernel = (
let env = (pkgs.python2.withPackages(p: with p; [
ipykernel numpy scipy
])).override (args: { ignoreCollisions = true; });
in {
displayName = "Python 2.7";
argv = ["${env.interpreter}" "-m" "ipykernel_launcher" "-f" "{connection_file}"];
language = "python";
logo32 = "${env}/${env.sitePackages}/ipykernel/resources/logo-32x32.png";
logo64 = "${env}/${env.sitePackages}/ipykernel/resources/logo-64x64.png";
});
python37Kernel = (
let env = pkgs.python37.withPackages(p: with p; [
ipykernel numpy scipy
]);
in {
displayName = "Python 3.7";
argv = ["${env.interpreter}" "-m" "ipykernel_launcher" "-f" "{connection_file}"];
language = "python";
logo32 = "${env}/${env.sitePackages}/ipykernel/resources/logo-32x32.png";
logo64 = "${env}/${env.sitePackages}/ipykernel/resources/logo-64x64.png";
});
python38Kernel = (
let env = pkgs.python38.withPackages(p: with p; [
ipykernel numpy scipy numba nixpkgs
]);
in {
displayName = "Python 3.8";
argv = ["${env.interpreter}" "-m" "ipykernel_launcher" "-f" "{connection_file}"];
language = "python";
logo32 = "${env}/${env.sitePackages}/ipykernel/resources/logo-32x32.png";
logo64 = "${env}/${env.sitePackages}/ipykernel/resources/logo-64x64.png";
});
cKernel = (
let env = pkgs.python3.withPackages(p: with p; [
jupyter-c-kernel
]);
in {
displayName = "C";
argv = ["${env.interpreter}" "-m" "jupyter_c_kernel" "-f" "{connection_file}"];
language = "c";
});
rustKernel = {
displayName = "Rust";
argv = ["${pkgs.evcxr}/bin/evcxr_jupyter" "--control_file" "{connection_file}"];
language = "Rust";
};
rKernel = (
let env = pkgs.rWrapper.override { packages = with pkgs.rPackages; [
IRkernel ggplot2
];};
in {
displayName = "R";
argv = ["${env}/bin/R" "--slave" "-e" "IRkernel::main()" "--args" "{connection_file}"];
language = "R";
});
ansibleKernel = (
let env = (pkgs.python3.withPackages(p: with p; [
ansible-kernel ansible
])).override (args: { ignoreCollisions = true; });
in {
displayName = "Ansible";
argv = ["${env.interpreter}" "-m" "ansible_kernel" "-f" "{connection_file}"];
language = "ansible";
});
bashKernel = (
let env = pkgs.python3.withPackages(p: with p; [
bash_kernel
]);
in {
displayName = "Bash";
argv = ["${env.interpreter}" "-m" "bash_kernel" "-f" "{connection_file}"];
language = "Bash";
});
} The password is |
adc973e
to
bd36e0e
Compare
@jonringer and @FRidh I'd say I'm complete with this PR and I've added the requested changes. Adding the nixos tests might be extremely tricky due to having to store a cookie and following redirects. The example I posted above should launch a multiuser jupyterhub cluster with 7 kernels. |
Not sure why the tests are failing |
it fails if ofborg doesn't post results, ofborg is just backed-up due to influx of PRs |
Result of 7 packages built:- python27Packages.jupyter-c-kernel - python37Packages.bash_kernel - python37Packages.jupyter-c-kernel - python37Packages.jupyterhub-systemdspawner - python38Packages.bash_kernel - python38Packages.jupyter-c-kernel - python38Packages.jupyterhub-systemdspawner |
bd36e0e
to
a6d8b9b
Compare
cc @infinisil does this look right to you? |
a6d8b9b
to
2033a83
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few minor nitpicks you may or may not find useful.
|
||
jupyterhubEnv = mkOption { | ||
type = types.package; | ||
default = (pkgs.python3.withPackages (p: with p; [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
defaultText
could be useful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
couldn't get this to work when I tried it. How is this properly used?
|
||
jupyterlabEnv = mkOption { | ||
type = types.package; | ||
default = (pkgs.python3.withPackages (p: with p; [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
defaultText
could be useful.
2033a83
to
b1b1235
Compare
@aanderse I have made your suggested changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
b1b1235
to
86b458a
Compare
I'm not going to delay this forever, LGTM |
Motivation for this change
I work at Quansight and we do a lot of python data science deployments with jupyterhub and related tooling that allows for scalable data science environments. This will be my first contributed nixos module so I am happy for feedback. The goal is to have an opinionated standard jupyterhub deployment (with
extraConfig
option so that you can do what you want). This opinionated service will use pam for authentication and systemd for launching processes for users. Though I will design this in mind so it is easily extensible to add oauth authentication for users and changing the spawner to use Docker, Kubernetes, etc.Once we have this component in place I see some interesting comparisons with this nixos module and https://tljh.jupyter.org/en/latest/ and https://zero-to-jupyterhub.readthedocs.io/en/latest/. I want to make reproducible data science environments. From using tooling such as terraform and ansible a lot for these types of deployments I am excited to see if we could replace those tools with this using nixops for example. Anyways that should be enough motiviation as to why I think this PR is important.
Things needing to be done for this PR to be complete:
Things done
sandbox
innix.conf
on non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
./result/bin/
)nix path-info -S
before and after)