breakpointHook: add for debugging failing builds #42371

Mic92 · 2018-06-22T13:09:53Z

Motivation for this change

Usage: Add breakpointHook to your buildInputs like this:

  stdenv.mkDerivation rec {
    # ...
    buildInputs = [ breakpointHook ];
  };

When the build fails as show in this example:

  pkgs.hello.overrideAttrs (old: {
    buildInputs = [ breakpointHook ];
    postPatch = ''
      false
    '';
  });

It will halt execution printing the following message:

build failed in patchPhase with exit code 1
To attach to this build run the following command as root:
    $ cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10

Installing cntr and running the command will provide shell access to the
build sandbox of failed build:

$ sudo cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10
WARNING: bad ownership on /nix/var/nix/profiles/per-user/root, should be 1000
[nixbld@localhost:/var/lib/cntr]$

At /var/lib/cntr the sandbox filesystem is mounted. All commands and
files of the system are still accessible within the shell.
To execute commands from the sandbox use the cntr exec subcommand.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Mic92 · 2018-06-22T13:12:23Z

Before merging this I would like to write nixpkgs documentation.

GrahamcOfBorg · 2018-06-22T13:45:01Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: breakpointHook, cntr

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-06-22T13:46:00Z

Failure on aarch64-linux (full log)

Attempted: breakpointHook, cntr

Partial log (click to expand)

              found type `*mut i8`

error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0308`.
error: Could not compile `cntr-nix`.

To learn more, run the command again with --verbose.
builder for '/nix/store/l3w9qhri50a6lq0mhilyl12an5ccm1f7-cntr-1.2.0.drv' failed with exit code 101
error: build of '/nix/store/l3w9qhri50a6lq0mhilyl12an5ccm1f7-cntr-1.2.0.drv' failed

orivej

Could you document -t and --type command in cntr attach --help and in cntr readme?

orivej · 2018-06-22T13:47:39Z

pkgs/build-support/setup-hooks/breakpoint-hook.sh

+    local no_color='\033[0m'
+
+    echo -e "${red}build failed in ${curPhase} with exit code ${exitCode}${no_color}"
+    printf "To attach install cntr into your profile and run the following command as root:\n\n"


"into your profile" is not necessary.

GrahamcOfBorg · 2018-06-22T13:47:57Z

Success on x86_64-linux (full log)

Attempted: breakpointHook, cntr

Partial log (click to expand)

installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/9vax8m1g8p085phynq8a2yl3nc7h74pq-cntr-1.2.0
shrinking /nix/store/9vax8m1g8p085phynq8a2yl3nc7h74pq-cntr-1.2.0/bin/cntr
strip is /nix/store/4mf2xm9p32lzrim927yk92xhx35yaz62-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/9vax8m1g8p085phynq8a2yl3nc7h74pq-cntr-1.2.0/bin
patching script interpreter paths in /nix/store/9vax8m1g8p085phynq8a2yl3nc7h74pq-cntr-1.2.0
checking for references to /build in /nix/store/9vax8m1g8p085phynq8a2yl3nc7h74pq-cntr-1.2.0...
/nix/store/k8y02g8iby243h420sh6wqsz9x2w3hnq-hook
/nix/store/9vax8m1g8p085phynq8a2yl3nc7h74pq-cntr-1.2.0

LnL7 · 2018-06-22T14:23:33Z

pkgs/build-support/setup-hooks/breakpoint-hook.sh

+    printf "To attach install cntr into your profile and run the following command as root:\n\n"
+    sh -c "echo '   cntr attach -t command cntr-${out}'; while true; do sleep 99999999; done"
+}
+failureHooks+=(breakpointHook)


Maybe this should also add an exitHook that fails the build to avoid accidental inclusion in a pr?

I would rather not do that because:

propagatedBuildInputs = [ breakpointHook ];

can be quite useful as well.
If this really becomes a problem we can still grep with ofBorg in the git diff.

right, nevermind then

GrahamcOfBorg · 2018-06-22T15:48:52Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: breakpointHook, cntr

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-06-22T15:48:54Z

Success on x86_64-linux (full log)

Attempted: breakpointHook, cntr

Partial log (click to expand)

these derivations will be built:
  /nix/store/cpr7gk7j3282lyvrkyirg5jscabimifz-hook.drv
building '/nix/store/cpr7gk7j3282lyvrkyirg5jscabimifz-hook.drv'...
/nix/store/cl03pc511w33wmrjaj08da5r7v532s2n-hook
/nix/store/9vax8m1g8p085phynq8a2yl3nc7h74pq-cntr-1.2.0

GrahamcOfBorg · 2018-06-22T15:49:08Z

Failure on aarch64-linux (full log)

Attempted: breakpointHook, cntr

Partial log (click to expand)

              found type `*mut i8`

error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0308`.
error: Could not compile `cntr-nix`.

To learn more, run the command again with --verbose.
builder for '/nix/store/l3w9qhri50a6lq0mhilyl12an5ccm1f7-cntr-1.2.0.drv' failed with exit code 101
error: build of '/nix/store/l3w9qhri50a6lq0mhilyl12an5ccm1f7-cntr-1.2.0.drv' failed

Mic92 · 2018-10-25T08:58:38Z

@GrahamcOfBorg build cntr

GrahamcOfBorg · 2018-10-25T08:59:07Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: cntr

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-10-25T09:02:55Z

Failure on aarch64-linux (full log)

Attempted: cntr

Partial log (click to expand)


   Compiling parking_lot v0.5.5
error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0308`.
error: Could not compile `cntr-nix`.

To learn more, run the command again with --verbose.
builder for '/nix/store/3z9l5bgbl07dycb0n03afcvnkgx5lgvw-cntr-1.2.0.drv' failed with exit code 101
error: build of '/nix/store/3z9l5bgbl07dycb0n03afcvnkgx5lgvw-cntr-1.2.0.drv' failed

GrahamcOfBorg · 2018-10-25T09:05:25Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: breakpointHook, cntr

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-10-25T09:05:36Z

Failure on aarch64-linux (full log)

Attempted: breakpointHook, cntr

Partial log (click to expand)


   Compiling parking_lot v0.5.5
error: aborting due to 3 previous errors

For more information about this error, try `rustc --explain E0308`.
error: Could not compile `cntr-nix`.

To learn more, run the command again with --verbose.
builder for '/nix/store/3z9l5bgbl07dycb0n03afcvnkgx5lgvw-cntr-1.2.0.drv' failed with exit code 101
error: build of '/nix/store/3z9l5bgbl07dycb0n03afcvnkgx5lgvw-cntr-1.2.0.drv' failed

GrahamcOfBorg · 2018-10-25T09:12:08Z

Success on x86_64-linux (full log)

Attempted: breakpointHook, cntr

Partial log (click to expand)

installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0
shrinking /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0/bin/cntr
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0/bin
patching script interpreter paths in /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0
checking for references to /build in /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0...
/nix/store/m8w9695shjg1zbh0frzyy5191f5fvr07-hook
/nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0

GrahamcOfBorg · 2018-10-25T09:14:35Z

Success on x86_64-linux (full log)

Attempted: cntr

Partial log (click to expand)


installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0
shrinking /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0/bin/cntr
strip is /nix/store/vcc4svb8gy29g4pam2zja6llkbcwsyiq-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0/bin
patching script interpreter paths in /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0
checking for references to /build in /nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0...
/nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0

Usuage: Add breakpointHook to your `buildInputs` like this: stdenv.mkDerivation rec { # ... buildInputs = [ breakpointHook ]; }); When the build fails as show in this example: pkgs.hello.overrideAttrs (old: { buildInputs = [ breakpointHook ]; postPatch = '' false ''; }); It will halt execution printing the following message: build failed in patchPhase with exit code 1 To attach to this build run the following command as root: cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10 Installing cntr and running the command will provide shell access to the build sandbox of failed build: sudo cntr attach -t command cntr-/nix/store/ynyb4n82x2r7sldd58pbb405jdqh5f00-hello-2.10 WARNING: bad ownership on /nix/var/nix/profiles/per-user/root, should be 1000 [nixbld@localhost:/var/lib/cntr]$ At /var/lib/cntr the sandbox filesystem is mounted. All commands and files of the system are still accessible within the shell. To execute commands from the sandbox use the `cntr exec` subcommand.

Mic92 · 2018-10-25T09:20:50Z

@GrahamcOfBorg build cntr

GrahamcOfBorg · 2018-10-25T09:21:02Z

No attempt on aarch64-linux (full log)

The following builds were skipped because they don't evaluate on aarch64-linux: cntr

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-10-25T09:21:03Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: cntr

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

GrahamcOfBorg · 2018-10-25T09:21:50Z

Success on x86_64-linux (full log)

Attempted: cntr

Partial log (click to expand)

/nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0

GrahamcOfBorg · 2018-10-25T09:33:15Z

Success on aarch64-linux (full log)

Attempted: breakpointHook

The following builds were skipped because they don't evaluate on aarch64-linux: cntr

Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.


/nix/store/fw4rz9lziqk14m7smcvf12w7mmm2y3ds-hook

GrahamcOfBorg · 2018-10-25T09:33:22Z

Success on x86_64-linux (full log)

Attempted: breakpointHook, cntr

Partial log (click to expand)

these derivations will be built:
  /nix/store/nl1wm051hy4kikvnvaf66fxr2zxmkcbx-hook.drv
building '/nix/store/nl1wm051hy4kikvnvaf66fxr2zxmkcbx-hook.drv'...
/nix/store/m8w9695shjg1zbh0frzyy5191f5fvr07-hook
/nix/store/xq1kqmcc4k24ng69j42svk4r22rjgd7a-cntr-1.2.0

GrahamcOfBorg · 2018-10-25T09:33:29Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: breakpointHook, cntr

Partial log (click to expand)


a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Mic92 · 2018-10-25T09:38:47Z

Just in time for the talk!

nixos-discourse · 2020-03-02T12:33:07Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/debugging-broken-builds/3138/5

Mic92 requested a review from Ericson2314 as a code owner June 22, 2018 13:09

Mic92 force-pushed the nix-breakpoints branch from 65c7711 to 34cfab1 Compare June 22, 2018 13:30

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Jun 22, 2018

orivej reviewed Jun 22, 2018

View reviewed changes

LnL7 approved these changes Jun 22, 2018

View reviewed changes

Mic92 force-pushed the nix-breakpoints branch from 34cfab1 to 8ec50dc Compare June 22, 2018 15:25

zimbatm mentioned this pull request Sep 25, 2018

Document Nix usage Mic92/cntr#10

Open

Mic92 force-pushed the nix-breakpoints branch from 8ec50dc to d075499 Compare October 25, 2018 08:53

GrahamcOfBorg added the 8.has: documentation label Oct 25, 2018

Mic92 added 3 commits October 25, 2018 10:19

cntr: init at 1.2.0

84e915a

doc/breakpointHook: add documentation

325a569

Mic92 force-pushed the nix-breakpoints branch from d075499 to 325a569 Compare October 25, 2018 09:19

Mic92 merged commit 926d6c0 into NixOS:master Oct 25, 2018

Mic92 deleted the nix-breakpoints branch October 25, 2018 09:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

breakpointHook: add for debugging failing builds #42371

breakpointHook: add for debugging failing builds #42371

Mic92 commented Jun 22, 2018 •

edited

Mic92 commented Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

orivej left a comment

orivej Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

LnL7 Jun 22, 2018

Mic92 Jun 22, 2018

LnL7 Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

Mic92 commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

Mic92 commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

Mic92 commented Oct 25, 2018

nixos-discourse commented Mar 2, 2020

breakpointHook: add for debugging failing builds #42371

breakpointHook: add for debugging failing builds #42371

Conversation

Mic92 commented Jun 22, 2018 • edited

Motivation for this change

Things done

Mic92 commented Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

orivej left a comment

Choose a reason for hiding this comment

orivej Jun 22, 2018

Choose a reason for hiding this comment

GrahamcOfBorg commented Jun 22, 2018

LnL7 Jun 22, 2018

Choose a reason for hiding this comment

Mic92 Jun 22, 2018

Choose a reason for hiding this comment

LnL7 Jun 22, 2018

Choose a reason for hiding this comment

GrahamcOfBorg commented Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

GrahamcOfBorg commented Jun 22, 2018

Mic92 commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

Mic92 commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

GrahamcOfBorg commented Oct 25, 2018

Mic92 commented Oct 25, 2018

nixos-discourse commented Mar 2, 2020

Mic92 commented Jun 22, 2018 •

edited