Add a post build hook #2995

thufschmitt · 2019-07-18T07:27:08Z

Add a new post-build-hook mechanism such that passing --option post-build-hook /foo/bar to a nix-* command will cause /foo/bar /nix/store/drv-that/has-been-built.drv -- out1 out2 ... outn to be executed after each build.

One specially interesting use-case for this is to upload all the builded artifacts to a binary cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD)

cc @grahamc

edolstra · 2019-07-18T09:12:20Z

src/libstore/build.cc

+            for (auto outputPath: outputPaths)
+                args.push_front(outputPath);
+            args.push_front("--");
+            args.push_front(drvPath);


Rather than using command line arguments, it would be nicer to use environment variables (e.g. OUTPUT_PATHS=... and DRV_PATH=...) because then the hook doesn't have to parse the command line.

I used cli arguments because that was the simplest to implement (because the runProgram2 interface doesn't allow to pass extra environment variables, so it requires extending it a bit) and to be more consistent with the pre-build-hook. But environment variables would make the hooks simpler indeed. I'll change that

(that being said the hook would then have to parse the OUTPUT_PATHS variable to get back a list. But that's probably already better)

src/libstore/globals.hh

doc/manual/command-ref/conf-file.xml

src/libstore/build.cc

thufschmitt · 2019-07-18T16:01:14Z

It looks like the test I've added doesn't work inside a nix-build. If I run make installcheck locally the test pass, but if I try nix-build release.nix -A build.x86_64-linux then the test fail with error: executing '/build/nix-2.3pre0_0000000/tests/push_to_store.sh': No such file or directory.

@edolstra @grahamc any idea of what could cause that? The file seems to be present in the sandbox but maybe there's something special about the way nix runs in here which prevents it from seeing the file

thufschmitt · 2019-07-19T05:07:55Z

It looks like the test I've added doesn't work inside a nix-build. If I run make installcheck locally the test pass, but if I try nix-build release.nix -A build.x86_64-linux then the test fail with error: executing '/build/nix-2.3pre0_0000000/tests/push_to_store.sh': No such file or directory.

@edolstra @grahamc any idea of what could cause that? The file seems to be present in the sandbox but maybe there's something special about the way nix runs in here which prevents it from seeing the file

Nvm, I was using /usr/bin/env bash (which is present on my machine but not in the build sandbox) as the shebang for that file, hence the error.

src/libstore/build.cc

doc/manual/advanced-topics/post-build-hook.xml

flokli

Really excited to see this - some comments :-)

doc/manual/command-ref/conf-file.xml

grahamc · 2019-07-30T21:13:07Z

I have some additional patches here around logging the output from the post build hook. @edolstra can we setup a time to do some pair programming to finish it off?

flokli · 2019-07-31T11:06:12Z

I gave this a try inside a CI environment, to sign and upload derivations after they're built, and it works just beautifully 😍!

We should add a note that this will only run for derivations being actually built, not those that are substituted from a binary cache.

In my CI environment, this shouldn't be a problem (as the nix copy command makes sure the full closure is uploaded), but it should probably still be documented.

Do we intend to add a similar hook for substituted ones?

grahamc

Do we intend to add a similar hook for substituted ones?

probably not for now :)

grahamc · 2019-08-02T14:29:42Z

doc/manual/command-ref/conf-file.xml

+
+      <itemizedlist>
+        <listitem><para>The hook executes after an evaluation-time build.</para></listitem>
+        <listitem><para>The hook does not execute on substituted paths.</para></listitem>


We should add a note that this will only run for derivations being actually built, not those that are substituted from a binary cache.

@flokli

Passing `--post-build-hook /foo/bar` to a nix-* command will cause `/foo/bar` to be executed after each build with the following environment variables set: DRV_PATH=/nix/store/drv-that-has-been-built.drv OUT_PATHS=/nix/store/...build /nix/store/...build-bin /nix/store/...build-dev This can be useful in particular to upload all the builded artifacts to the cache (including the ones that don't appear in the runtime closure of the final derivation or are built because of IFD). This new feature prints the stderr/stdout output to the `nix-build` and `nix build` client, and the output is printed in a Nix 2 compatible format: [nix]$ ./inst/bin/nix-build ./test.nix these derivations will be built: /nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv building '/nix/store/ishzj9ni17xq4hgrjvlyjkfvm00b0ch9-my-example-derivation.drv'... hello! bye! running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'... post-build-hook: + sleep 1 post-build-hook: + echo 'Signing paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Signing paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + echo 'Uploading paths' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: Uploading paths /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation post-build-hook: + sleep 1 post-build-hook: + printf 'very important stuff' /nix/store/qr213vjmibrqwnyp5fw678y7whbkqyny-my-example-derivation [nix-shell:~/projects/github.com/NixOS/nix]$ ./inst/bin/nix build -L -f ./test.nix my-example-derivation> hello! my-example-derivation> bye! my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Signing paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Signing paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + echo 'Uploading paths' /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> Uploading paths /nix/store/c263gzj2kb2609mz8wrbmh53l14wzmfs-my-example-derivation my-example-derivation (post)> + sleep 1 my-example-derivation (post)> + printf 'very important stuff' [1 built, 0.0 MiB DL] Co-authored-by: Graham Christensen <graham@grahamc.com> Co-authored-by: Eelco Dolstra <edolstra@gmail.com>

grahamc · 2019-08-02T14:48:57Z

I think this is ready to go!

grahamc · 2019-08-02T15:17:29Z

Here is a demo: https://asciinema.org/a/JGe4UHRYMfj554e34gE97h45v

edolstra · 2019-08-05T11:18:23Z

doc/manual/advanced-topics/post-build-hook.xml

+      shell to perform word splitting to make each output path its
+      own argument to <command>nix sign-paths</command>. Nix guarantees
+      the paths will only contain characters which are safe for word
+      splitting, and free of any globs.


This is not quite true. In particular paths can contain ? and +. So doing for i in $OUT_PATHS may cause globbing (and this has caused performance issues in the past, because globbing a path like ...-gtk+-... requires the entire Nix store to be read.

edolstra · 2019-08-05T11:20:09Z

doc/manual/command-ref/conf-file.xml

+        <varlistentry>
+          <term><envar>OUT_PATHS</envar></term>
+          <listitem>
+            <para>Output paths of the built derivation, separated by a space (<literal> </literal>) character.</para>


Drop <literal> </literal> since it doesn't show anything useful.

edolstra · 2019-08-05T11:21:49Z

doc/manual/command-ref/conf-file.xml

+        <listitem><para>The hook executes after an evaluation-time build.</para></listitem>
+        <listitem><para>The hook does not execute on substituted paths.</para></listitem>
+        <listitem><para>The hook's output always goes to the user's terminal.</para></listitem>
+        <listitem><para>If the hook fails, the build succeeds but no further builds execute.</para></listitem>


It's also worth noting that the hook is executed synchronously and so blocks other builds from progressing.

grahamc

I added a commit addressing the latest round of feedback. I think it is good again :)

edolstra reviewed Jul 18, 2019

View reviewed changes

src/libstore/globals.hh Outdated Show resolved Hide resolved

edolstra reviewed Jul 18, 2019

View reviewed changes

doc/manual/command-ref/conf-file.xml Outdated Show resolved Hide resolved

grahamc reviewed Jul 18, 2019

View reviewed changes

src/libstore/build.cc Outdated Show resolved Hide resolved

grahamc reviewed Jul 19, 2019

View reviewed changes

src/libstore/build.cc Outdated Show resolved Hide resolved

edolstra reviewed Jul 21, 2019

View reviewed changes

doc/manual/advanced-topics/post-build-hook.xml Outdated Show resolved Hide resolved

flokli reviewed Jul 30, 2019

View reviewed changes

doc/manual/command-ref/conf-file.xml Outdated Show resolved Hide resolved

doc/manual/command-ref/conf-file.xml Outdated Show resolved Hide resolved

grahamc force-pushed the post-build-hook branch 2 times, most recently from 985bae8 to 74461e9 Compare August 2, 2019 14:29

grahamc reviewed Aug 2, 2019

View reviewed changes

grahamc force-pushed the post-build-hook branch from 74461e9 to 7c55967 Compare August 2, 2019 14:48

grahamc requested a review from edolstra August 2, 2019 14:49

edolstra reviewed Aug 5, 2019

View reviewed changes

post-build-hook: docs fixup

363a2f6

grahamc reviewed Aug 6, 2019

View reviewed changes

edolstra merged commit 56df30c into NixOS:master Aug 7, 2019

adisbladis mentioned this pull request Aug 14, 2019

Auto-upload built binaries to Cachix nix-community/infra#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a post build hook #2995

Add a post build hook #2995

thufschmitt commented Jul 18, 2019

edolstra Jul 18, 2019

thufschmitt Jul 18, 2019

thufschmitt Jul 18, 2019

thufschmitt commented Jul 18, 2019

thufschmitt commented Jul 19, 2019

flokli left a comment

grahamc commented Jul 30, 2019

flokli commented Jul 31, 2019

grahamc left a comment

grahamc Aug 2, 2019

grahamc commented Aug 2, 2019

grahamc commented Aug 2, 2019

edolstra Aug 5, 2019

grahamc Aug 6, 2019

edolstra Aug 5, 2019

grahamc Aug 6, 2019

edolstra Aug 5, 2019

grahamc Aug 6, 2019

grahamc left a comment

Add a post build hook #2995

Add a post build hook #2995

Conversation

thufschmitt commented Jul 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thufschmitt commented Jul 18, 2019

thufschmitt commented Jul 19, 2019

flokli left a comment

Choose a reason for hiding this comment

grahamc commented Jul 30, 2019

flokli commented Jul 31, 2019

grahamc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamc commented Aug 2, 2019

grahamc commented Aug 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamc left a comment

Choose a reason for hiding this comment