buildbot: 0.9.0rc4 -> 0.9.0.post1 #19759

nand0p · 2016-10-21T21:12:42Z

buildbot: 0.9.0rc4 -> 0.9.0.post1

updates buildbot to version 9 release
adds nixos configuration module
fixes buildbot-www package deps
re-hardcode path to tail

nand0p · 2016-10-21T21:15:50Z

@FRidh @copumpkin @ryansydnor this PR is WIP, as still working thru the buildbot nixos configuration module.

Mic92 · 2016-10-21T23:09:50Z

nixos/modules/services/continuous-integration/buildbot/default.nix

+      script = "${pkgs.buildbot-ui}/bin/buildbot start master";
+
+      postStart = ''
+        until [[ $(${pkgs.curl}/bin/curl -s --head -w '\n%{http_code}' http://localhost:${toString cfg.port} | tail -n1) =~ ^(200|403)$ ]]; do


what is that for?

This runs curl until it returns a status code indicating it is running, to block "After"-dependent services.

Do we have any services with an "After" dependency on this?

buildbot worker would need buildbot master running to come up successfully

@nand0p would you run a worker on the same machine as a master?

in a dev/test environment, bb master and worker may run on same machine. This is probably a common setup.

Unlikely in production, although i can imagine a scenario where a dedicated builder/worker would need to run on master (to make use of instance profile, secrets, etc), but that would be an edge case, and there are probably better ways to set worker environment.

nand0p · 2016-10-27T14:17:32Z

@copumpkin @FRidh @grahamc @Mic92 ready_for_review

copumpkin · 2016-10-27T14:24:38Z

I haven't looked deeply yet, but two things jump to mind:

You need to include your module in the big list o' modules that NixOS includes (https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/module-list.nix)
I'd probably call the module buildbot-master or something similar, since it seems likely that people will eventually want a buildbot-worker module to handle worker configuration.

copumpkin · 2016-11-11T17:12:41Z

nixos/modules/services/continuous-integration/buildbot-master/default.nix

+      environment =
+        let
+          selectedSessionVars =
+            lib.filterAttrs (n: v: builtins.elem n [ "NIX_PATH" ])


lib should re-export user-facing builtins, so lib.elem would probably be nicer here.

copumpkin · 2016-11-11T17:13:44Z

nixos/modules/services/continuous-integration/buildbot-master/default.nix

+        ${pkgs.buildbot-ui}/bin/buildbot create-master ${cfg.buildbotDir}
+        sed -i '/www/,+1 d' ${cfg.buildbotDir}/master.cfg.sample
+        mv -v ${cfg.buildbotDir}/master.cfg.sample ${cfg.buildbotDir}/master.cfg
+        echo "c['www'] = dict(port=8010)" >> ${cfg.buildbotDir}/master.cfg


I think you mean ${cfg.port} here, right?

copumpkin · 2016-11-11T17:19:13Z

nixos/modules/services/continuous-integration/buildbot-master/default.nix

+        sed -i '/www/,+1 d' ${cfg.buildbotDir}/master.cfg.sample
+        mv -v ${cfg.buildbotDir}/master.cfg.sample ${cfg.buildbotDir}/master.cfg
+        echo "c['www'] = dict(port=8010)" >> ${cfg.buildbotDir}/master.cfg
+        cat ${cfg.buildbotDir}/master.cfg


It seems like it would be nice for master.cfg to exist in the store, to avoid the config from changing without Nix being aware of it (since it'll just overwrite it anyway).

Maybe use -c to create-master to point it at a master.cfg in the store?

copumpkin · 2016-11-16T22:35:21Z

nixos/modules/services/continuous-integration/buildbot-master/default.nix

+          selectedSessionVars //
+            { BUILDBOT_HOME = cfg.home;
+              BUILDBOT_DIR = cfg.buildbotDir;
+              NIX_REMOTE = "daemon";


Hmm, what's this about?

copumpkin · 2016-11-16T22:36:26Z

nixos/modules/services/continuous-integration/buildbot-master/default.nix

+  cfg = config.services.buildbot-master;
+  configFile = pkgs.writeText "master.cfg" ''
+
+from buildbot.plugins import *


There's no need to not indent this, by the way. The double quote thing looks for the smallest amount of leading whitespace and subtracts that from all lines, IIRC.

copumpkin · 2016-11-16T22:41:25Z

pkgs/development/tools/build-managers/buildbot/default.nix

  '';

  postFixup = ''
+    # re-hardcode path to tail
+    sed -i 's|/usr/bin/tail|${coreutils}/bin/tail|' $out/lib/python2.7/site-packages/buildbot/scripts/logwatcher.py


If you did this in the patchPhase or pre/postPatch, you wouldn't have to remove the .pyc file, right?

copumpkin · 2016-11-16T22:44:52Z

nixos/modules/services/continuous-integration/buildbot-master/default.nix

+
+      changeSource = mkOption {
+        default = " \\
+          \"changes.GitPoller('git://github.com/buildbot/pyflakes.git', workdir='gitpoller-workdir', branch='master', pollinterval=300)\" \\


These defaults seem more appropriate for example, than default. example just shows up in the documentation as possible values, whereas default will actually get set if you don't set some other value for it. Given that most people probably don't need to monitor the pyflakes repo, it makes more sense to me for the config to complain that this hasn't been set than to default to looking somewhere many users might not want to look.

copumpkin · 2016-11-16T22:47:05Z

nixos/modules/module-list.nix

@@ -134,6 +134,7 @@
  ./services/computing/torque/server.nix
  ./services/computing/torque/mom.nix
  ./services/computing/slurm/slurm.nix
+  ./services/continuous-integration/buildbot-master/default.nix


I'd probably either do ./services/continuous-integration/buildbot/master.nix (and then someday we might get a /worker.nix) or buildbot-master.nix. Most services don't use a default.nix or have supporting files.

- updates buildbot to version 9 release - adds nixos configuration module - fixes buildbot-www package deps - re-hardcode path to tail - builbot configuration via module vars

nand0p · 2016-11-21T23:16:40Z

ready_for_review

grahamc · 2016-11-28T01:49:31Z

Bump @copumpkin, @Mic92

nand0p · 2016-12-07T16:24:43Z

@copumpkin added both functionality of passing entire raw config and/or just override a few default variables. should be ready for merge?

Mic92 · 2016-12-08T07:36:58Z

will take a look at it later this day. In case I forget it, feel free to ping me again after 24h.
Update: I did not forget it, but did not find the time today. Hopefully tomorrow.

Mic92 · 2016-12-11T14:58:33Z

I tested the module now with "LocalWorker". I made some changes to the module
in this branch: https://github.com/Mic92/nixpkgs/commits/buildbot See commit message for details.

- allow to exchange package - better option types (lists instead of string) - remove environment variable (was unused)

nand0p · 2016-12-12T20:52:41Z

@Mic92 cherry-picked your buildbot refactor commit to this PR

Mic92 · 2016-12-13T09:55:30Z

nixos/modules/services/continuous-integration/buildbot/master.nix

@@ -233,19 +229,18 @@ in {
        User = cfg.user;
        Group = cfg.group;
        WorkingDirectory = cfg.home;
+        ExecStart = "${cfg.package}/bin/buildbot start ${cfg.buildbotDir}";
      };

      preStart = ''
        mkdir -vp ${cfg.buildbotDir}
        chown -c ${cfg.user}:${cfg.group} ${cfg.buildbotDir}


oh, this should have been "-R", or we need fixed uid/gid for buildbot!

Mic92 reviewed Oct 21, 2016

View reviewed changes

grahamc added 8.has: package (update) 8.has: module (new) 2.status: work-in-progress labels Oct 22, 2016

nand0p force-pushed the buildbot-0.9.0.post1 branch 3 times, most recently from 042fa9b to 63adc8d Compare October 25, 2016 21:21

nand0p force-pushed the buildbot-0.9.0.post1 branch 2 times, most recently from 619268e to 80ad35a Compare November 3, 2016 17:18

copumpkin reviewed Nov 11, 2016

View reviewed changes

nand0p force-pushed the buildbot-0.9.0.post1 branch 3 times, most recently from bff27c8 to 063626d Compare November 16, 2016 22:28

copumpkin reviewed Nov 16, 2016

View reviewed changes

nand0p force-pushed the buildbot-0.9.0.post1 branch 2 times, most recently from 6ea6e79 to f4e8d8d Compare November 18, 2016 15:20

buildbot: 0.9.0rc4 -> 0.9.0.post1

a4de1df

- updates buildbot to version 9 release - adds nixos configuration module - fixes buildbot-www package deps - re-hardcode path to tail - builbot configuration via module vars

nand0p force-pushed the buildbot-0.9.0.post1 branch from f4e8d8d to a4de1df Compare November 21, 2016 22:04

Mic92 self-assigned this Dec 8, 2016

buildbot: refactor options

1373725

- allow to exchange package - better option types (lists instead of string) - remove environment variable (was unused)

Mic92 closed this in 50466c2 Dec 13, 2016

Mic92 reviewed Dec 13, 2016

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildbot: 0.9.0rc4 -> 0.9.0.post1 #19759

buildbot: 0.9.0rc4 -> 0.9.0.post1 #19759

nand0p commented Oct 21, 2016 •

edited

nand0p commented Oct 21, 2016

Mic92 Oct 21, 2016

grahamc Oct 22, 2016

Mic92 Oct 22, 2016

nand0p Oct 24, 2016

copumpkin Oct 24, 2016

nand0p Oct 24, 2016

nand0p commented Oct 27, 2016

copumpkin commented Oct 27, 2016

copumpkin Nov 11, 2016

copumpkin Nov 11, 2016

copumpkin Nov 11, 2016

copumpkin Nov 16, 2016

copumpkin Nov 16, 2016

copumpkin Nov 16, 2016

copumpkin Nov 16, 2016

copumpkin Nov 16, 2016

nand0p commented Nov 21, 2016

grahamc commented Nov 28, 2016

nand0p commented Dec 7, 2016

Mic92 commented Dec 8, 2016 •

edited

Mic92 commented Dec 11, 2016

nand0p commented Dec 12, 2016

Mic92 Dec 13, 2016

buildbot: 0.9.0rc4 -> 0.9.0.post1 #19759

buildbot: 0.9.0rc4 -> 0.9.0.post1 #19759

Conversation

nand0p commented Oct 21, 2016 • edited

nand0p commented Oct 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nand0p commented Oct 27, 2016

copumpkin commented Oct 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nand0p commented Nov 21, 2016

grahamc commented Nov 28, 2016

nand0p commented Dec 7, 2016

Mic92 commented Dec 8, 2016 • edited

Mic92 commented Dec 11, 2016

nand0p commented Dec 12, 2016

Choose a reason for hiding this comment

nand0p commented Oct 21, 2016 •

edited

Mic92 commented Dec 8, 2016 •

edited