nixos/gitlab-runner: support multiple services #84139

misuzu · 2020-04-02T22:00:26Z

Motivation for this change

A more declarative module for gitlab-runner with support for multiple services.
Inspired by @arianvp's module.

Main changes:

Multiple runner services can be registered.
Automatic registration via registration token so same config can be used on many machines.
Runner services is registered/unregistered only when needed (idempotent).
Dead runner services are removed from config on start/reload.
DynamicUser is used so no need for static user and group.
Support for running nix in docker via host nix-daemon.

What is missing:

~~Global configuration could only be changed by editing /var/lib/gitlab-runner/.gitlab-runner/config.toml~~. Fixed.

Please comment if you think that something else is missing or would be great to have.

Example config with four runner services:

{ config, pkgs, lib, ...}:
with lib;
{
  services.gitlab-runner = {
    enable = true;
    services = {
      # runner for building via nix in docker
      nix = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-nix-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token1>
        '';
        dockerImage = "alpine";
        dockerVolumes = [
          "/nix/store:/nix/store:ro"
          "/nix/var/nix/db:/nix/var/nix/db:ro"
          "/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
        ];
        dockerDisableCache = true;
        preBuildScript = pkgs.writeScript "setup-container" ''
          mkdir -p -m 0755 /nix/var/log/nix/drvs
          mkdir -p -m 0755 /nix/var/nix/gcroots
          mkdir -p -m 0755 /nix/var/nix/profiles
          mkdir -p -m 0755 /nix/var/nix/temproots
          mkdir -p -m 0755 /nix/var/nix/userpool
          mkdir -p -m 1777 /nix/var/nix/gcroots/per-user
          mkdir -p -m 1777 /nix/var/nix/profiles/per-user
          mkdir -p -m 0755 /nix/var/nix/profiles/per-user/root
          mkdir -p -m 0700 "$HOME/.nix-defexpr"

          . ${pkgs.nix}/etc/profile.d/nix.sh

          ${pkgs.nix}/bin/nix-env -i ${concatStringsSep " " (with pkgs; [ nix cacert git openssh ])}

          ${pkgs.nix}/bin/nix-channel --add https://nixos.org/channels/nixpkgs-unstable
          ${pkgs.nix}/bin/nix-channel --update nixpkgs
        '';
        environmentVariables = {
          ENV = "/etc/profile";
          USER = "root";
          NIX_REMOTE = "daemon";
          PATH = "/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/bin:/sbin:/usr/bin:/usr/sbin";
          NIX_SSL_CERT_FILE = "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt";
        };
        tagList = [ "nix" ];
      };
      # runner for building docker images
      docker-images = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-docker-images-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token2>
        '';
        dockerImage = "docker:stable";
        dockerVolumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ];
        tagList = [ "docker-images" ];
      };
      # runner for executing stuff on host system
      # make sure to add required packages (including git!)
      # to `environment.systemPackages`
      shell = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-shell-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token3>
        '';
        executor = "shell";
        tagList = [ "shell" ];
      };
      # runner for everything else
      default = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-default-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token4>
        '';
        dockerImage = "debian:stable";
      };
    };
  };
  environment.systemPackages = with pkgs; [ git ];
}

Things done

nixos/modules/services/continuous-integration/gitlab-runner-multi.nix

rissson · 2020-04-02T22:08:09Z

You probably want to document the method to use the host's nix-daemon in the runner somewhere

misuzu · 2020-04-02T22:10:32Z

You probably want to document the method to use the host's nix-daemon in the runner somewhere

Maybe adding a module for this would be even better?

nixos/modules/services/continuous-integration/gitlab-runner-multi.nix

rissson · 2020-04-02T22:32:26Z

Maybe adding a module for this would be even better?

You could add an option for this, with the proper warning that it exposes the host's store to anything running on that runner

EDIT: actually, as it is, it might cause problems for people with a custom store path

misuzu · 2020-04-07T19:53:18Z

I've added examples for services option, built manual with nix-build nixos/release.nix -A manualHTML.x86_64-linux and checked that there is no display issues.

flokli · 2020-04-07T22:15:35Z

@arianvp, could you review this?

misuzu · 2020-04-09T10:31:18Z

Global check_interval and concurrent is now configurable.

bachp · 2020-04-10T22:24:18Z

@misuzu Can't we combine this with the existing gitlab-runner module.

Both modules have some features the other one is missing. Here is a short list of what I think those are.

gitlab-runner-multi

multiple services
declartive
automatic registration 🍾

gitlab-runner

runner executable can be selected via services.gitlab-runner.package attribute
graceful termination where it can configured how long gitlab-runner should wait till jobs get killed if the runner service is stoped (e.g machine shutdown/reboot).
allows changing the working directory
better module name

My proposal would be:

Replace the current module with this one but keeping all the currently supported options.
Add the .services attribute as proposed here to gitlab-runner
Allow .configFile to override the config. I think that would cover most usecases of the current module.
Deprecate .configOptions
Maybe rename packages to extraPackages as proposed here.

What do you think?

I think it would be better to have only one gitlab-runner module in nixpkgs.

misuzu · 2020-04-10T22:41:24Z

Replace the current module with this one but keeping all the currently supported options.

workDir option will not work correctly with DynamicUser. Should DynamicUser be dropped?

Allow .configFile to override the config. I think that would cover most usecases of the current module.

I'll look into this.

I think it would be better to have only one gitlab-runner module in nixpkgs.

The idea was to drop current one since it's broken anyway, but suggested changes sounds reasonable.

flokli · 2020-04-11T11:01:05Z

I'm also much in favour of improving/merging the existing module.

@misuzu workDir would still work in combination with DynamicUser=true, as long as it's inside /var/lib/gitlab-runner (which is the default).

I'd prefer to restrict the runner to this location over introducing all sort of system-wide requirements.
We could add an assertion asking the user to move things before switching if workDir doesn't currently point into a subdirectory of /var/lib/gitlab-runner.

bachp · 2020-04-11T11:43:51Z

As far as I remember workDir is only useful in combination with the shell runner, which I have no experience with. So from my point of view we can also drop the workDir option if nobody has a usecase for it.

misuzu · 2020-04-11T11:46:20Z

@misuzu workDir would still work in combination with DynamicUser=true, as long as it's inside /var/lib/gitlab-runner (which is the default).

We could add an assertion asking the user to move things before switching if workDir doesn't currently point into a subdirectory of /var/lib/gitlab-runner.

What's the difference if workDir in /var/lib/gitlab-runner or /var/lib/gitlab-runner/some/other/path? Don't really see the point of this option in this case.

As far as I remember workDir is only useful in combination with the shell runner, which I have no experience with. So from my point of view we can also drop the workDir option if nobody has a usecase for it.

I agree.

flokli · 2020-04-11T12:04:43Z

Yeah, okay, let's drop it, and in the mkRemovedOption module mention state (if any) would need to be migrated to /var/lib/gitlab-runner.

nixos/modules/services/continuous-integration/gitlab-runner.nix

bachp

With my proposed change this modules works as a drop in replacement for the use cases I have tested.

All docker based runners tough.

nixos/modules/services/continuous-integration/gitlab-runner.nix

bachp

Thanks a lot for the improvements to the module. All good from my side now!

FYI @max-wittig

nixos/modules/services/continuous-integration/gitlab-runner.nix

misuzu · 2020-04-17T19:00:18Z

Added registrationFlags option for advanced use-cases.

misuzu · 2020-04-18T11:04:27Z

Added ability to change user for service using systemd.services.gitlab-runner.serviceConfig options. Configuration script will still be executed as root.
Tested changing User to root, unprivileged user and rolling back to DynamicUser, no manual chown's was needed.
Not sure if there should be dedicated options for this.

nixos-discourse · 2020-04-18T21:15:16Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/call-for-testing-gitlab-runner-module/6765/1

nixos/modules/services/continuous-integration/gitlab-runner.nix

arianvp · 2020-05-01T09:24:34Z

This looks great! what is left? I think this is ready to merge no?

nixos/modules/services/continuous-integration/gitlab-runner.nix

misuzu · 2020-05-01T09:58:19Z

This looks great! what is left? I think this is ready to merge no?

I think it's feature-complete and people are saying it's working fine so yes, i think it is.
New options should be easy to add if necessary.

flokli · 2020-05-01T10:37:23Z

I'm still not a fan of running the configuration script as root, but that can be improved on in a later PR.

flokli · 2020-05-01T10:37:37Z

Thanks everyone!

FRidh · 2020-05-02T07:40:54Z

I reverted this when merging master into staging-next to avoid the merge conflict. Please rebase it unto staging-next and submit again.

misuzu · 2020-05-02T09:50:42Z

I reverted this when merging master into staging-next to avoid the merge conflict. Please rebase it unto staging-next and submit again.

Done! #86561

misuzu requested a review from infinisil as a code owner April 2, 2020 22:00

ofborg bot added 6.topic: nixos 8.has: module (update) labels Apr 2, 2020

rissson reviewed Apr 2, 2020

View reviewed changes

nixos/modules/services/continuous-integration/gitlab-runner-multi.nix Outdated Show resolved Hide resolved

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Apr 2, 2020

flokli reviewed Apr 2, 2020

View reviewed changes

nixos/modules/services/continuous-integration/gitlab-runner-multi.nix Outdated Show resolved Hide resolved

misuzu force-pushed the gitlab-runner-multi branch 2 times, most recently from 44f3855 to 02d8bcf Compare April 7, 2020 19:49

misuzu force-pushed the gitlab-runner-multi branch from 02d8bcf to 24a5034 Compare April 9, 2020 10:30

misuzu force-pushed the gitlab-runner-multi branch 5 times, most recently from 882c765 to 93f86b9 Compare April 10, 2020 22:17

misuzu force-pushed the gitlab-runner-multi branch from 93f86b9 to f9010c8 Compare April 11, 2020 12:42

misuzu changed the title ~~nixos/gitlab-runner-multi: init~~ nixos/gitlab-runner: support multiple services Apr 11, 2020

misuzu force-pushed the gitlab-runner-multi branch from ce3aea3 to 221791f Compare April 11, 2020 18:21

flokli reviewed Apr 11, 2020

View reviewed changes

nixos/modules/services/continuous-integration/gitlab-runner.nix Outdated Show resolved Hide resolved

misuzu force-pushed the gitlab-runner-multi branch from 221791f to 4aff1f7 Compare April 11, 2020 20:11

bachp requested changes Apr 11, 2020

View reviewed changes

nixos/modules/services/continuous-integration/gitlab-runner.nix Outdated Show resolved Hide resolved

misuzu force-pushed the gitlab-runner-multi branch from 4aff1f7 to 7095937 Compare April 11, 2020 22:36

bachp approved these changes Apr 12, 2020

View reviewed changes

bachp reviewed Apr 13, 2020

View reviewed changes

nixos/modules/services/continuous-integration/gitlab-runner.nix Show resolved Hide resolved

misuzu force-pushed the gitlab-runner-multi branch 3 times, most recently from 9620830 to 3b80f04 Compare April 17, 2020 18:59

misuzu force-pushed the gitlab-runner-multi branch from 3b80f04 to ac8d006 Compare April 18, 2020 11:01

misuzu force-pushed the gitlab-runner-multi branch from ac8d006 to c4c855f Compare April 18, 2020 11:31

misuzu commented Apr 30, 2020

View reviewed changes

nixos/modules/services/continuous-integration/gitlab-runner.nix Outdated Show resolved Hide resolved

misuzu force-pushed the gitlab-runner-multi branch from c4c855f to d241f5a Compare April 30, 2020 15:26

arianvp reviewed May 1, 2020

View reviewed changes

nixos/modules/services/continuous-integration/gitlab-runner.nix Outdated Show resolved Hide resolved

arianvp reviewed May 1, 2020

View reviewed changes

nixos/modules/services/continuous-integration/gitlab-runner.nix Outdated Show resolved Hide resolved

nixos/gitlab-runner: support multiple services

3853c27

misuzu force-pushed the gitlab-runner-multi branch from d241f5a to 3853c27 Compare May 1, 2020 09:53

flokli merged commit 4e14ff6 into NixOS:master May 1, 2020

FRidh mentioned this pull request May 2, 2020

Staging next #86442

Merged

10 tasks

misuzu mentioned this pull request May 2, 2020

nixos/gitlab-runner: support multiple services #86561

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/gitlab-runner: support multiple services #84139

nixos/gitlab-runner: support multiple services #84139

misuzu commented Apr 2, 2020 •

edited

rissson commented Apr 2, 2020

misuzu commented Apr 2, 2020

rissson commented Apr 2, 2020 •

edited

misuzu commented Apr 7, 2020

flokli commented Apr 7, 2020

misuzu commented Apr 9, 2020

bachp commented Apr 10, 2020

misuzu commented Apr 10, 2020

flokli commented Apr 11, 2020

bachp commented Apr 11, 2020

misuzu commented Apr 11, 2020

flokli commented Apr 11, 2020

bachp left a comment

bachp left a comment

misuzu commented Apr 17, 2020

misuzu commented Apr 18, 2020 •

edited

nixos-discourse commented Apr 18, 2020

arianvp commented May 1, 2020

misuzu commented May 1, 2020

flokli commented May 1, 2020

flokli commented May 1, 2020

FRidh commented May 2, 2020

misuzu commented May 2, 2020

nixos/gitlab-runner: support multiple services #84139

nixos/gitlab-runner: support multiple services #84139

Conversation

misuzu commented Apr 2, 2020 • edited

Motivation for this change

Things done

rissson commented Apr 2, 2020

misuzu commented Apr 2, 2020

rissson commented Apr 2, 2020 • edited

misuzu commented Apr 7, 2020

flokli commented Apr 7, 2020

misuzu commented Apr 9, 2020

bachp commented Apr 10, 2020

misuzu commented Apr 10, 2020

flokli commented Apr 11, 2020

bachp commented Apr 11, 2020

misuzu commented Apr 11, 2020

flokli commented Apr 11, 2020

bachp left a comment

Choose a reason for hiding this comment

bachp left a comment

Choose a reason for hiding this comment

misuzu commented Apr 17, 2020

misuzu commented Apr 18, 2020 • edited

nixos-discourse commented Apr 18, 2020

arianvp commented May 1, 2020

misuzu commented May 1, 2020

flokli commented May 1, 2020

flokli commented May 1, 2020

FRidh commented May 2, 2020

misuzu commented May 2, 2020

misuzu commented Apr 2, 2020 •

edited

rissson commented Apr 2, 2020 •

edited

misuzu commented Apr 18, 2020 •

edited