nixos/gitlab-runner: support multiple services #86561

misuzu · 2020-05-02T09:18:40Z

Motivation for this change

This is a rebase of #84139

A more declarative module for gitlab-runner with support for multiple services.
Inspired by @arianvp's module.

Main changes:

Multiple runner services can be registered.
Automatic registration via registration token so same config can be used on many machines.
Runner services is registered/unregistered only when needed (idempotent).
Dead runner services are removed from config on start/reload.
DynamicUser is used so no need for static user and group.
Support for running nix in docker via host nix-daemon.

What is missing:

~~Global configuration could only be changed by editing /var/lib/gitlab-runner/.gitlab-runner/config.toml~~. Fixed.

Please comment if you think that something else is missing or would be great to have.

Example config with four runner services:

{ config, pkgs, lib, ...}:
with lib;
{
  services.gitlab-runner = {
    enable = true;
    services = {
      # runner for building via nix in docker
      nix = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-nix-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token1>
        '';
        dockerImage = "alpine";
        dockerVolumes = [
          "/nix/store:/nix/store:ro"
          "/nix/var/nix/db:/nix/var/nix/db:ro"
          "/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
        ];
        dockerDisableCache = true;
        preBuildScript = pkgs.writeScript "setup-container" ''
          mkdir -p -m 0755 /nix/var/log/nix/drvs
          mkdir -p -m 0755 /nix/var/nix/gcroots
          mkdir -p -m 0755 /nix/var/nix/profiles
          mkdir -p -m 0755 /nix/var/nix/temproots
          mkdir -p -m 0755 /nix/var/nix/userpool
          mkdir -p -m 1777 /nix/var/nix/gcroots/per-user
          mkdir -p -m 1777 /nix/var/nix/profiles/per-user
          mkdir -p -m 0755 /nix/var/nix/profiles/per-user/root
          mkdir -p -m 0700 "$HOME/.nix-defexpr"

          . ${pkgs.nix}/etc/profile.d/nix.sh

          ${pkgs.nix}/bin/nix-env -i ${concatStringsSep " " (with pkgs; [ nix cacert git openssh ])}

          ${pkgs.nix}/bin/nix-channel --add https://nixos.org/channels/nixpkgs-unstable
          ${pkgs.nix}/bin/nix-channel --update nixpkgs
        '';
        environmentVariables = {
          ENV = "/etc/profile";
          USER = "root";
          NIX_REMOTE = "daemon";
          PATH = "/nix/var/nix/profiles/default/bin:/nix/var/nix/profiles/default/sbin:/bin:/sbin:/usr/bin:/usr/sbin";
          NIX_SSL_CERT_FILE = "/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt";
        };
        tagList = [ "nix" ];
      };
      # runner for building docker images
      docker-images = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-docker-images-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token2>
        '';
        dockerImage = "docker:stable";
        dockerVolumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ];
        tagList = [ "docker-images" ];
      };
      # runner for executing stuff on host system
      # make sure to add required packages (including git!)
      # to `environment.systemPackages`
      shell = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-shell-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token3>
        '';
        executor = "shell";
        tagList = [ "shell" ];
      };
      # runner for everything else
      default = {
        registrationConfigFile = pkgs.writeText "gitlab-runner-default-registration" ''
          CI_SERVER_URL=https://gitlab.com/
          REGISTRATION_TOKEN=<token4>
        '';
        dockerImage = "debian:stable";
      };
    };
  };
  environment.systemPackages = with pkgs; [ git ];
}

Things done

max-wittig · 2020-05-05T12:42:38Z

@FRidh @misuzu Is there an option to define additional options? E.g. some things are missing:

S3 cache
Pull policy
Session server
Sentry DSN

misuzu · 2020-05-05T12:44:35Z

@FRidh @misuzu Is there an option to define additional options? E.g. some things are missing:

S3 cache

Pull policy

Session server

Sentry DSN

Yes, there is registrationFlags.
Session server and Sentry DSN should be easy to add to module, in the meantime you can add them manually to /var/lib/gitlab-runner/.gitlab-runner/config.toml.

max-wittig · 2020-05-05T13:23:33Z

@misuzu Thanks for the fast response. I can't really add anything manually, as this is an automated setup. Will the remaining complex options like s3 cache and session server also be added soon?

misuzu · 2020-05-05T13:52:53Z

@misuzu Thanks for the fast response. I can't really add anything manually, as this is an automated setup. Will the remaining complex options like s3 cache and session server also be added soon?

You can configure cache using registrationFlags:

   --cache-type value                                           Select caching method [$CACHE_TYPE]
   --cache-path value                                           Name of the path to prepend to the cache URL [$CACHE_PATH]
   --cache-shared                                               Enable cache sharing between runners. [$CACHE_SHARED]
   --cache-s3-server-address value                              A host:port to the used S3-compatible server [$CACHE_S3_SERVER_ADDRESS]
   --cache-s3-access-key value                                  S3 Access Key [$CACHE_S3_ACCESS_KEY]
   --cache-s3-secret-key value                                  S3 Secret Key [$CACHE_S3_SECRET_KEY]
   --cache-s3-bucket-name value                                 Name of the bucket where cache will be stored [$CACHE_S3_BUCKET_NAME]
   --cache-s3-bucket-location value                             Name of S3 region [$CACHE_S3_BUCKET_LOCATION]
   --cache-s3-insecure                                          Use insecure mode (without https) [$CACHE_S3_INSECURE]

I can try adding other global options.

misuzu · 2020-05-05T14:04:55Z

@max-wittig please take a look at #86946
See https://discourse.nixos.org/t/call-for-testing-gitlab-runner-module/6765 for how to quickly test it.

max-wittig · 2020-05-12T11:06:43Z

@misuzu I've tested it now with this MR, but I'm just getting the old gitlab-runner (12.4.1). Is there any way to specify the latest runner. I've seen the package option but I'm not sure how to bump something there. Should already be at pkgs.gitlab-runner

misuzu · 2020-05-12T11:39:28Z

@misuzu I've tested it now with this MR, but I'm just getting the old gitlab-runner (12.4.1). Is there any way to specify the latest runner. I've seen the package option but I'm not sure how to bump something there. Should already be at pkgs.gitlab-runner

12.4.1 is in 19.09. You can upgrade to 20.03 or unstable for newer version.

max-wittig · 2020-05-12T12:01:18Z

@misuzu I just noticed that, but ran into another problem. Every 2nd deployment or so, the config.toml stays empty:

gitlab-runner is happy with an empty config, but I'm not 😄

misuzu · 2020-05-12T12:05:54Z

the config.toml stays empty

Please share your config.

max-wittig · 2020-05-12T12:08:17Z

@misuzu Thanks for all the help!

services.gitlab-runner = {
    enable = true;
    checkInterval = 15;
    package = latestPkgs.gitlab-runner;
    #concurrent = ${vars.concurrent};
    services = {
      docker-images = {
        registrationFlags = [
          "--cache-type"
          "s3"
          "--cache-shared"
          "--cache-s3-server-address"
          "s3.amazonaws.com"
          "--cache-s3-access-key"
          "${vars.s3_access_key}"
          "--cache-s3-secret-key"
          "${vars.s3_secret_key}"
          "--cache-s3-bucket-name"
          "some-aws-bucket"
          "--cache-s3-bucket-location"
          "eu-central-1"
          "--docker-pull-policy"
          "always"
        ];
        registrationConfigFile = pkgs.writeText "gitlab-runner-nix-registration" ''
          CI_SERVER_URL=https://gitlab-url/
          REGISTRATION_TOKEN=${vars.token}
        '';
        #dockerAllowedImages = "vars.docker_whitelist}";
        dockerImage = "some-registry.com:some-image:stable";
        dockerVolumes = [
          "/mnt/projects/sstate-cache:/home/projects/sstate-cache:rw"
        ];
        executor = "docker";
        maximumTimeout = 86400;
        dockerDisableCache = true;
        environmentVariables = {
          CONCURRENCY_LEVEL = vars.concurrent;
        };
        #tagList = ["${vars.tags}"];
      };
    };
  };

misuzu · 2020-05-12T12:27:10Z

@max-wittig this config is working fine on my server. Please check the logs: journalctl -u gitlab-runner.service.

max-wittig · 2020-05-12T13:10:04Z

No errors in the journal. Looks good, but doesn't work.

misuzu · 2020-05-12T13:16:57Z

No errors in the journal. Looks good, but doesn't work.

Please share a full log: start journalctl -f -u gitlab-runner.service and in different console run systemctl restart gitlab-runner.service

max-wittig · 2020-05-12T13:19:55Z

Doesn't change anything, as it's the registration part that writes this empty config file that doesn't work, not the runner start part.

misuzu · 2020-05-12T13:23:44Z

Doesn't change anything, as it's the registration part that writes this empty config file that doesn't work, not the runner start part.

That's exactly what i want to see, there is no logs from gitlab-runner-configure in your screenshot.

max-wittig · 2020-05-12T13:26:07Z

Ah I guess I had some registration parameter wrong. Can we implement some sort of error for this? Like fail the nixos switch if this happens?

misuzu · 2020-05-12T13:29:15Z

Ah I guess I had some registration parameter wrong. Can we implement some sort of error for this? Like fail the nixos switch if this happens?

Maybe. I have to understand what happened to do that.

max-wittig · 2020-05-12T13:29:48Z

I think the registration failed. I had this message: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2862

max-wittig · 2020-05-12T13:37:42Z

@misuzu Thanks again for the great work you did here and sorry for my stupid questions 😉

max-wittig · 2020-05-12T13:43:00Z

@misuzu The registration error from the runner actually looks like this:

misuzu · 2020-05-12T13:43:21Z

@misuzu Thanks again for the great work you did here and sorry for my stupid questions

Don't worry. Thanks for testing!

misuzu · 2020-05-12T13:43:32Z

@misuzu The registration error from the runner actually looks like this:

I have updated pull request, now this should fail on activation.

max-wittig · 2020-05-12T13:47:57Z

Ah nice! Thanks. One more thing. The name of the runner cannot be overriden currently, right? Could that be configurable, too? 😄

misuzu · 2020-05-12T13:56:15Z

Ah nice! Thanks. One more thing. The name of the runner cannot be overriden currently, right? Could that be configurable, too?

You can use something else instead of docker-images for prefix but suffix is needed to check if runner must be re-registered when configuration is changed.

max-wittig · 2020-05-12T14:09:12Z

Ah I see. Thanks

SuperSandro2000 · 2021-02-03T11:38:53Z

This should have been mentioned in the release notes.

nixos/gitlab-runner: support multiple services

8ccc3b2

misuzu requested a review from flokli May 2, 2020 09:19

ofborg bot added 6.topic: nixos 8.has: module (update) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels May 2, 2020

misuzu mentioned this pull request May 2, 2020

nixos/gitlab-runner: support multiple services #84139

Merged

10 tasks

FRidh merged commit 0b0afca into NixOS:staging-next May 2, 2020

misuzu mentioned this pull request May 5, 2020

nixos/gitlab-runner: add more global options #86946

Merged

10 tasks

misuzu deleted the gitlab-runner-multi branch April 22, 2021 21:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos/gitlab-runner: support multiple services #86561

nixos/gitlab-runner: support multiple services #86561

misuzu commented May 2, 2020

max-wittig commented May 5, 2020

misuzu commented May 5, 2020 •

edited

max-wittig commented May 5, 2020

misuzu commented May 5, 2020

misuzu commented May 5, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020 •

edited

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

max-wittig commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

SuperSandro2000 commented Feb 3, 2021

nixos/gitlab-runner: support multiple services #86561

nixos/gitlab-runner: support multiple services #86561

Conversation

misuzu commented May 2, 2020

Motivation for this change

Things done

max-wittig commented May 5, 2020

misuzu commented May 5, 2020 • edited

max-wittig commented May 5, 2020

misuzu commented May 5, 2020

misuzu commented May 5, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020 • edited

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

max-wittig commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

misuzu commented May 12, 2020

max-wittig commented May 12, 2020

SuperSandro2000 commented Feb 3, 2021

misuzu commented May 5, 2020 •

edited

misuzu commented May 12, 2020 •

edited