hydra: 2017-11-21 -> 2018-08-07 #44841

Ma27 · 2018-08-09T20:44:54Z

Motivation for this change

The first commit fixes a minor issue with the exported perl bindings for Nix.
The second commit bumps Hydra and applies some minor fixes to the derivation:
- Compatibility with Nix 2.0 and Nix 2.1 (there's a slight API incompatibility between the versions, please read the commit message for further reference).
- Dropped obsolete postUnpack hook.
- Added myself as maintainer to help out in case of further breakage.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

This causes collisions between the build outputs of `nix` when building in an environment with `nix1.perl-bindings` and `nix`: ``` collision between `/nix/store/aa4rrcj7dg2xj4rfkiclcmp745ibqng0-nix-2.0.4/lib/libnixstore.so' and `/nix/store/sp0sdi4bll80h58big1iy8kkh3qqxpw2-nix-1.11.16/lib/libnixstore.so' builder for '/nix/store/wgbccin107lhm8cv9imnnvkx1j2pgibc-hydra-perl-deps.drv' failed with exit code 25 ```

Ma27 · 2018-08-09T20:47:55Z

I played around with it locally, but I'll test the change with my own Hydra tomorrow, would be cool to get some feedback though:)

GrahamcOfBorg · 2018-08-09T21:07:12Z

Success on aarch64-linux (full log)

Attempted: nix1

The following builds were skipped because they don't evaluate on aarch64-linux: hydra

Partial log (click to expand)

  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


these paths will be fetched (0.75 MiB download, 3.18 MiB unpacked):
  /nix/store/9lvd1zk0h23vkn0bawqxz7zchkyv4338-perl-WWW-Curl-4.17
  /nix/store/fi43kyl9psyi71xrri4q4w4mykfv90kr-nix-1.11.16
copying path '/nix/store/9lvd1zk0h23vkn0bawqxz7zchkyv4338-perl-WWW-Curl-4.17' from 'https://cache.nixos.org'...
copying path '/nix/store/fi43kyl9psyi71xrri4q4w4mykfv90kr-nix-1.11.16' from 'https://cache.nixos.org'...
/nix/store/fi43kyl9psyi71xrri4q4w4mykfv90kr-nix-1.11.16

GrahamcOfBorg · 2018-08-09T21:08:16Z

Success on x86_64-linux (full log)

Attempted: hydra, nix1

Partial log (click to expand)

make[1]: Leaving directory '/build/source'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/x21viqrphphy918r96kf2x5p81mgv8ic-hydra-2018-08-07
shrinking /nix/store/x21viqrphphy918r96kf2x5p81mgv8ic-hydra-2018-08-07/bin/hydra-queue-runner
shrinking /nix/store/x21viqrphphy918r96kf2x5p81mgv8ic-hydra-2018-08-07/bin/hydra-eval-jobs
shrinking /nix/store/x21viqrphphy918r96kf2x5p81mgv8ic-hydra-2018-08-07/bin/hydra-evaluator
patching script interpreter paths in /nix/store/x21viqrphphy918r96kf2x5p81mgv8ic-hydra-2018-08-07
checking for references to /build in /nix/store/x21viqrphphy918r96kf2x5p81mgv8ic-hydra-2018-08-07...
finalPhase
build time elapsed:  0m0.684s 0m0.654s 1m36.858s 0m9.504s

srhb · 2018-08-12T07:42:34Z

pkgs/development/tools/misc/hydra/default.nix

+  nativeBuildInputs = [ autoreconfHook ];
+
+  patches = lib.optionals (!isGreaterNix20) [
+    (fetchpatch {


I think it would be nice to inline a short description of this patch here.

nlewo · 2018-08-12T08:40:28Z

@GrahamcOfBorg test hydra

GrahamcOfBorg · 2018-08-12T08:40:55Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.hydra

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-08-12T08:52:04Z

Failure on x86_64-linux (full log)

Attempted: tests.hydra

Partial log (click to expand)

cannot build derivation '/nix/store/r0p9ldkkkpdnjwina6wzy912q7gaspf5-system-units.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/g52b9jj44hrq9iiscrg0s7518bgirbmf-user-units.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5h6qbvf7q6dc6bm5jqr55yqsj3p4pqym-etc.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/x6zch7kv1zzski6kz2wi1hfmw9plwbrz-nixos-system-machine-18.09.git.3091d6b.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/ic0v9285621dp6rqdnk5fl4z7jd0bz3x-closure-info.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/j6sn1f7i8790bxzz5n7g076f9lza0w7q-run-nixos-vm.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/3cw69f9mr4wz2kqvpqqb49mfxhhd9h59-nixos-vm.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/4b6w0sld1d0v8iggwv2ywpfaj3b0x97h-nixos-test-driver-hydra-init-localdb.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/fpw0fi4adbz83wlzq305hz3c81fb6fz1-vm-test-run-hydra-init-localdb.drv': 1 dependencies couldn't be built
error: build of '/nix/store/fpw0fi4adbz83wlzq305hz3c81fb6fz1-vm-test-run-hydra-init-localdb.drv' failed

Ma27 · 2018-08-12T09:15:37Z

I'll look into the test and fix the review comments...

Ma27 · 2018-08-13T07:44:05Z

regarding the test: it seems as the build of the Hydra derivation fails when evaluating the test expression (not sure why though), but it works fine, when running nix-build -A hydra and after that the test (since the expression is in the Nix store then).
The test itself breaks (not sure why ATM, but the test jobset breaks). I started investigating with a VM and the interactive test driver and will continue with this tonight.

Ma27 · 2018-08-13T14:05:56Z

@srhb I fixed your comment and the test, would you mind having another look?

let's see if I can trigger VM tests on my own (I'm a known user at least) :D
@GrahamcOfBorg test hydra

GrahamcOfBorg · 2018-08-13T14:06:25Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.hydra

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-08-13T14:09:34Z

Success on x86_64-linux (full log)

Attempted: tests.hydra

Partial log (click to expand)

machine: exit status 1
syncing
machine: running command: sync
machine: exit status 0
test script finished in 37.89s
cleaning up
killing machine (pid 597)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/5nvb0s472822b4ddnmkxn5lfvyl7j54j-vm-test-run-hydra-init-localdb

GrahamcOfBorg · 2018-08-13T14:17:22Z

Success on aarch64-linux (full log)

Attempted: nix1

The following builds were skipped because they don't evaluate on aarch64-linux: hydra

Partial log (click to expand)

copying path '/nix/store/08ny5hf4m9zx700g2f59df6jc6bwldhk-libatomic_ops-7.6.4' from 'https://cache.nixos.org'...
copying path '/nix/store/wqavhs2f4xbm7g6wqb36wvbxnl4052cl-curl-7.59.0' from 'https://cache.nixos.org'...
copying path '/nix/store/q6nlysz1hhsy5rjlmcql3b81p5gyl346-libsodium-1.0.16' from 'https://cache.nixos.org'...
copying path '/nix/store/k9v709cxqkxd9rj19dngps7v5ax78rq2-perl-DBI-1.641' from 'https://cache.nixos.org'...
copying path '/nix/store/9lvd1zk0h23vkn0bawqxz7zchkyv4338-perl-WWW-Curl-4.17' from 'https://cache.nixos.org'...
copying path '/nix/store/izqp9vi2604chbylkyi17rjch4jn7hn3-boehm-gc-7.6.6' from 'https://cache.nixos.org'...
copying path '/nix/store/wdrp06zd2w40q0ipqxbqw4bij3dbaqwx-curl-7.59.0-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/1c8blbizrsdr8cavynwfchmigzhdypax-perl-DBD-SQLite-1.58' from 'https://cache.nixos.org'...
copying path '/nix/store/fi43kyl9psyi71xrri4q4w4mykfv90kr-nix-1.11.16' from 'https://cache.nixos.org'...
/nix/store/fi43kyl9psyi71xrri4q4w4mykfv90kr-nix-1.11.16

GrahamcOfBorg · 2018-08-13T14:17:31Z

Success on x86_64-linux (full log)

Attempted: hydra, nix1

Partial log (click to expand)

these paths will be fetched (0.79 MiB download, 3.21 MiB unpacked):
  /nix/store/1yjc0jb2x7ammsap25zksyvl0is4vnn7-perl-WWW-Curl-4.17
  /nix/store/rhjx1gm7cxgcb6gcvrb60iyrc7711lw8-nix-1.11.16
copying path '/nix/store/1yjc0jb2x7ammsap25zksyvl0is4vnn7-perl-WWW-Curl-4.17' from 'https://cache.nixos.org'...
copying path '/nix/store/rhjx1gm7cxgcb6gcvrb60iyrc7711lw8-nix-1.11.16' from 'https://cache.nixos.org'...
/nix/store/p17pxrinipf0p8znwm2l53h9hj909ssf-hydra-2018-08-07
/nix/store/rhjx1gm7cxgcb6gcvrb60iyrc7711lw8-nix-1.11.16

srhb · 2018-08-13T20:36:21Z

I approve, with one comment and the desire for one other reviewer, since it's Hydra and Hydra is scary. :P

Why does the test need these extra systemPackages jq and bash? Shouldn't the module be taking care of that anyway?

nlewo

Thanks for your fixes!
So, just some minor comments.

nlewo · 2018-08-13T20:39:32Z

nixos/tests/hydra/default.nix

@@ -7,9 +7,8 @@ let
     { trivial = builtins.derivation {
         name = "trivial";
         system = "x86_64-linux";
-         PATH = coreutils;


The with import <nix/config.nix>; line is then useless. Could you also remove it?

nlewo · 2018-08-13T20:42:36Z

nixos/tests/hydra/default.nix

-         PATH = coreutils;
-         builder = shell;
-         args = ["-c" "touch $out; exit 0"];
+         builder = "/bin/sh";


Regarding what you said in the commit message, I think /bin/sh comes from the busybox package which is bind mounted at the build environment creation (and not from the VM /bin/sh).

nlewo · 2018-08-13T20:42:58Z

nixos/tests/hydra/default.nix

@@ -37,7 +36,7 @@ in {
      virtualisation.memorySize = 1024;
      time.timeZone = "UTC";

-      environment.systemPackages = [ createTrivialProject pkgs.jq ];
+      environment.systemPackages = with pkgs; [ createTrivialProject jq bash ];


bash is useless

This bumps Hydra to the latest revision available. As Hydra doesn't have a release model (and therefore no tags) ATM, the derivation will pin against the actual git revision and the date of the commit in the derivation name. Additionally the following changes have been made: * Dropped `postUnpack` phase. It is useful when working with the Hydra source (and no dirty changes shall be used in `release.nix`, but is has no use in `nixpkgs`). * Added myself as maintainer to have more folks available in case of future breakage. * Implemented support for Nix 2.0 and `unstable` (currently 2.1): Since 1672bcd in NixOS/nix the evaluator differentiates between `settings` and `evalSettings`. Previously `restrictEval` in `hydra-eval-jobs.cc` has been set in `settings`, this doesn't work anymore in Nix 2.1 and is therefore incompatible to Nix 2.0 on an API level. To resolve this, the flag `isGreaterNix20` parses the version string of `pkgs.nix` and applies a patch if nix.version<=2.0. Furthermore the Hydra build with Nix 2.1 requires `boost` as build input which is not needed for Nix 2.0. To avoid unnecessary increase in the closure size this library will only used as build input for nix.version>2.0. * Fixed the NixOS test for `hydra`: disabled binary cache to allow sandbox builds (otherwise it would query `cache.nixos.org` during the Hydra build inside the test). Additionally the trivial.nix jobset required simplification (as done in NixOS/hydra, e.g. tests/api-test.nix) as bash is not available in the build sandbox as builder (even when adding pkgs.bash to systemPackages). The easiest workaround to confirm a the functionality of a jobset without importing nixpkgs is to use the default shell /bin/sh which is mounted from `pkgs.busybox` into the build env (NixOS#44841 (comment)) in the VM and a named pipe to create $out. Closes NixOS#44044

Ma27 · 2018-08-13T21:01:05Z

I approve, with one comment and the desire for one other reviewer, since it's Hydra and Hydra is scary. :P

agreed, I use it on my own, but the more reviews, the safer I feel with this patch :D

Why does the test need these extra systemPackages jq and bash? Shouldn't the module be taking care of that anyway?

bash was a hack of mine (which can be removed again as stated by @nlewo), jq is just used to parse/interpret the output of Hydra's JSON API in the test case and has no use for Hydra itself.`

@nlewo thanks for your feedback!

GrahamcOfBorg · 2018-08-13T21:12:12Z

Success on aarch64-linux (full log)

Attempted: nix1

The following builds were skipped because they don't evaluate on aarch64-linux: hydra

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/fi43kyl9psyi71xrri4q4w4mykfv90kr-nix-1.11.16

GrahamcOfBorg · 2018-08-13T21:14:18Z

Success on x86_64-linux (full log)

Attempted: hydra, nix1

Partial log (click to expand)

make[1]: Leaving directory '/build/source'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/p17pxrinipf0p8znwm2l53h9hj909ssf-hydra-2018-08-07
shrinking /nix/store/p17pxrinipf0p8znwm2l53h9hj909ssf-hydra-2018-08-07/bin/hydra-eval-jobs
shrinking /nix/store/p17pxrinipf0p8znwm2l53h9hj909ssf-hydra-2018-08-07/bin/hydra-evaluator
shrinking /nix/store/p17pxrinipf0p8znwm2l53h9hj909ssf-hydra-2018-08-07/bin/hydra-queue-runner
patching script interpreter paths in /nix/store/p17pxrinipf0p8znwm2l53h9hj909ssf-hydra-2018-08-07
checking for references to /build in /nix/store/p17pxrinipf0p8znwm2l53h9hj909ssf-hydra-2018-08-07...
finalPhase
build time elapsed:  0m0.846s 0m0.902s 2m4.568s 0m12.588s

nlewo · 2018-08-14T07:23:36Z

@GrahamcOfBorg test hydra

GrahamcOfBorg · 2018-08-14T07:24:07Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: tests.hydra

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-08-14T07:24:45Z

Success on x86_64-linux (full log)

Attempted: tests.hydra

Partial log (click to expand)

machine: exit status 1
syncing
machine: running command: sync
machine: exit status 0
test script finished in 41.48s
cleaning up
killing machine (pid 597)
vde_switch: EOF on stdin, cleaning up and exiting
vde_switch: Could not remove ctl dir '/build/vde1.ctl': Directory not empty
/nix/store/6a88pnvm89k4ka5yqz745i366kcivq2b-vm-test-run-hydra-init-localdb

nlewo · 2018-08-14T07:55:09Z

Thanks!

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Aug 9, 2018

srhb reviewed Aug 12, 2018

View reviewed changes

Ma27 changed the title ~~hydra: 2017-11-21 -> 2018-08-07~~ WIP hydra: 2017-11-21 -> 2018-08-07 Aug 13, 2018

Ma27 force-pushed the bump-hydra branch from 12fc0da to e0e9b5c Compare August 13, 2018 14:04

GrahamcOfBorg added the 6.topic: nixos label Aug 13, 2018

Ma27 changed the title ~~WIP hydra: 2017-11-21 -> 2018-08-07~~ hydra: 2017-11-21 -> 2018-08-07 Aug 13, 2018

nlewo requested changes Aug 13, 2018

View reviewed changes

Ma27 force-pushed the bump-hydra branch from e0e9b5c to 7682c2f Compare August 13, 2018 20:58

nlewo approved these changes Aug 14, 2018

View reviewed changes

nlewo merged commit f2cab31 into NixOS:master Aug 14, 2018

Ma27 deleted the bump-hydra branch August 14, 2018 11:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hydra: 2017-11-21 -> 2018-08-07 #44841

hydra: 2017-11-21 -> 2018-08-07 #44841

Ma27 commented Aug 9, 2018 •

edited

Ma27 commented Aug 9, 2018

GrahamcOfBorg commented Aug 9, 2018

GrahamcOfBorg commented Aug 9, 2018

srhb Aug 12, 2018

nlewo commented Aug 12, 2018

GrahamcOfBorg commented Aug 12, 2018

GrahamcOfBorg commented Aug 12, 2018

Ma27 commented Aug 12, 2018

Ma27 commented Aug 13, 2018

Ma27 commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

srhb commented Aug 13, 2018

nlewo left a comment

nlewo Aug 13, 2018

nlewo Aug 13, 2018

nlewo Aug 13, 2018

Ma27 commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

nlewo commented Aug 14, 2018

GrahamcOfBorg commented Aug 14, 2018

GrahamcOfBorg commented Aug 14, 2018

nlewo commented Aug 14, 2018

hydra: 2017-11-21 -> 2018-08-07 #44841

hydra: 2017-11-21 -> 2018-08-07 #44841

Conversation

Ma27 commented Aug 9, 2018 • edited

Motivation for this change

Things done

Ma27 commented Aug 9, 2018

GrahamcOfBorg commented Aug 9, 2018

GrahamcOfBorg commented Aug 9, 2018

srhb Aug 12, 2018

Choose a reason for hiding this comment

nlewo commented Aug 12, 2018

GrahamcOfBorg commented Aug 12, 2018

GrahamcOfBorg commented Aug 12, 2018

Ma27 commented Aug 12, 2018

Ma27 commented Aug 13, 2018

Ma27 commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

srhb commented Aug 13, 2018

nlewo left a comment

Choose a reason for hiding this comment

nlewo Aug 13, 2018

Choose a reason for hiding this comment

nlewo Aug 13, 2018

Choose a reason for hiding this comment

nlewo Aug 13, 2018

Choose a reason for hiding this comment

Ma27 commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

GrahamcOfBorg commented Aug 13, 2018

nlewo commented Aug 14, 2018

GrahamcOfBorg commented Aug 14, 2018

GrahamcOfBorg commented Aug 14, 2018

nlewo commented Aug 14, 2018

Ma27 commented Aug 9, 2018 •

edited