libvirt: Avoid patchShebangs in the darwin build #33084

johbo · 2017-12-26T20:01:08Z

Motivation for this change

Noticed that libvirt did not build on darwin. Tracked the problem down to the
call to "patchShebangs" in the hook "preConfigure".

libvirt is a new dependency in NixOPS, that's why I found it.

The error during the build looked as follows:

Can't exec "no": No such file or directory at ./rpc/genprotocol.pl line 45.
cannot run no -h locking/lock_protocol.x: No such file or directory at ./rpc/genprotocol.pl line 45.
make[2]: *** [Makefile:12060: locking/lock_protocol.h] Error 2
make[2]: Leaving directory '/private/var/folders/v2/kx2sg5693tb1h84zc2hmjjgr0000gn/T/nix-build-libvirt-3.10.0.drv-0/libvirt-3.10.0/src'
make[1]: *** [Makefile:2119: all-recursive] Error 1
make[1]: Leaving directory '/private/var/folders/v2/kx2sg5693tb1h84zc2hmjjgr0000gn/T/nix-build-libvirt-3.10.0.drv-0/libvirt-3.10.0'
make: *** [Makefile:2012: all] Error 2
builder for ‘/nix/store/m0riralw5x2dwj9p1fgfq2rihysbvahc-libvirt-3.10.0.drv’ failed with exit code 2
cannot build derivation ‘/nix/store/3ij4hh2b5ikn1ndss5hi0s5i0nyrzsnz-python2.7-libvirt-3.10.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/3ij4hh2b5ikn1ndss5hi0s5i0nyrzsnz-python2.7-libvirt-3.10.0.drv’ failed

When investigating via make --trace I found the following details:

Makefile:12060: update target 'locking/lock_protocol.h' due to: locking/lock_protocol.x rpc/genprotocol.pl
echo "  GEN     " locking/lock_protocol.h;/nix/store/204x8f9ii18sszpfqmq35rqdpmzm482w-perl-5.24.3/bin/perl -w ./rpc/genprotocol.pl no -h \
       locking/lock_protocol.x ./locking/lock_protocol.h
  GEN      locking/lock_protocol.h
Can't exec "no": No such file or directory at ./rpc/genprotocol.pl line 45.
cannot run no -h locking/lock_protocol.x: No such file or directory at ./rpc/genprotocol.pl line 45.

I did notice by accident that things were working when I forgot to run the hook preConfigure.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

grahamc · 2017-12-26T20:34:46Z

@GrahamcOfBorg build libvirt

GrahamcOfBorg

Success for system: aarch64-linux

copying path '/nix/store/cz76yg431fh92i25g6m7qcidnp1k7bal-libpciaccess-0.14' from 'https://cache.nixos.org'...
copying path '/nix/store/685zfm904y82qbzm1srm1qyxya2h1v38-numactl-2.0.10' from 'https://cache.nixos.org'...
copying path '/nix/store/2yj6w4ls43h8miims3if9vvjh9vyhp3a-numad-0.5' from 'https://cache.nixos.org'...
copying path '/nix/store/48qxncwafxwg3b43k38dyijd61yjs3z3-parted-3.2' from 'https://cache.nixos.org'...
copying path '/nix/store/jrd4wi7qg1jrr6lbbwnn67xi68d1fnmr-yajl-2.1.0' from 'https://cache.nixos.org'...
copying path '/nix/store/sr8536mljvjkm429r4mpm58rrxlzynng-zfs-user-0.7.4' from 'https://cache.nixos.org'...
copying path '/nix/store/5ja0x47hza8niddyzxhpan3rihvb3zwc-dbus-1.10.24' from 'https://cache.nixos.org'...
copying path '/nix/store/iws9x1jjfy6b7rhgh63572if7s7sbkz9-pm-utils-1.4.1' from 'https://cache.nixos.org'...
copying path '/nix/store/ly14zsg5703jnxrc5a3m04pv7zzmrirk-libvirt-3.10.0' from 'https://cache.nixos.org'...
/nix/store/ly14zsg5703jnxrc5a3m04pv7zzmrirk-libvirt-3.10.0

GrahamcOfBorg

Success for system: x86_64-darwin

Removing /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/share/gtk-doc
rmdir: failed to remove '/nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/share': Directory not empty
gzipping man pages under /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/share/man/
stripping (with flags -S) in /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/lib  /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/libexec  /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/bin  /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/sbin
patching script interpreter paths in /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0
/nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/bin/virt-pki-validate: interpreter directive changed from "/bin/sh" to "/nix/store/hiv85an0qyzj6ps88f5pvkbbdj8238nj-bash-4.4-p12/bin/sh"
/nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/bin/virt-xml-validate: interpreter directive changed from "/bin/sh" to "/nix/store/hiv85an0qyzj6ps88f5pvkbbdj8238nj-bash-4.4-p12/bin/sh"
/nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/libexec/libvirt-guests.sh: interpreter directive changed from "/bin/sh" to "/nix/store/hiv85an0qyzj6ps88f5pvkbbdj8238nj-bash-4.4-p12/bin/sh"
moving /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/sbin/* to /nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0/bin
/nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0

GrahamcOfBorg

Failure for system: x86_64-linux

cannot build derivation ‘/nix/store/snn1c2g9395apg9m0bnlgmnm0wp77ygg-yaml-0.8.25.drv’: 17 dependencies couldn't be built
cannot build derivation ‘/nix/store/ig7gi36gxa21h2wvfn8kgkh9cjsnmn0c-x509-system-1.6.6.drv’: 6 dependencies couldn't be built
cannot build derivation ‘/nix/store/h8f6mdfmskcl45b5h5hxfpyvqap9piki-x509-validation-1.6.9.drv’: 15 dependencies couldn't be built
cannot build derivation ‘/nix/store/vixjq0z2vncrg818pvklvij763jb40yk-tls-1.4.0.drv’: 14 dependencies couldn't be built
cannot build derivation ‘/nix/store/3hjqfnikz92994zk5nc5rwsv9b310sr1-connection-0.2.8.drv’: 11 dependencies couldn't be built
cannot build derivation ‘/nix/store/79y9yka3nc0hpamc6mv7f6xpv1vmi6v4-http-client-tls-0.3.5.1.drv’: 14 dependencies couldn't be built
cannot build derivation ‘/nix/store/0w7fzaykgyyc6cvwba15m1iwf34qz3jg-pandoc-2.0.5.drv’: 42 dependencies couldn't be built
cannot build derivation ‘/nix/store/faansijryr9m4nwsr4h6h98bhdijxx37-xen-4.5.5.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/dhhm1z5nrs3drcdgbyccv1sxnq1svn2l-libvirt-3.10.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/dhhm1z5nrs3drcdgbyccv1sxnq1svn2l-libvirt-3.10.0.drv’ failed

grahamc · 2017-12-26T23:50:09Z

@GrahamcOfBorg build libvirt

GrahamcOfBorg

Success for system: aarch64-linux

/nix/store/ly14zsg5703jnxrc5a3m04pv7zzmrirk-libvirt-3.10.0

GrahamcOfBorg

Success for system: x86_64-darwin

/nix/store/l9pyyrga3zwkka9ik02c88fb6pnw67lj-libvirt-3.10.0

GrahamcOfBorg

Failure for system: x86_64-linux

cannot build derivation ‘/nix/store/snn1c2g9395apg9m0bnlgmnm0wp77ygg-yaml-0.8.25.drv’: 17 dependencies couldn't be built
cannot build derivation ‘/nix/store/ig7gi36gxa21h2wvfn8kgkh9cjsnmn0c-x509-system-1.6.6.drv’: 6 dependencies couldn't be built
cannot build derivation ‘/nix/store/h8f6mdfmskcl45b5h5hxfpyvqap9piki-x509-validation-1.6.9.drv’: 15 dependencies couldn't be built
cannot build derivation ‘/nix/store/vixjq0z2vncrg818pvklvij763jb40yk-tls-1.4.0.drv’: 14 dependencies couldn't be built
cannot build derivation ‘/nix/store/3hjqfnikz92994zk5nc5rwsv9b310sr1-connection-0.2.8.drv’: 11 dependencies couldn't be built
cannot build derivation ‘/nix/store/79y9yka3nc0hpamc6mv7f6xpv1vmi6v4-http-client-tls-0.3.5.1.drv’: 14 dependencies couldn't be built
cannot build derivation ‘/nix/store/0w7fzaykgyyc6cvwba15m1iwf34qz3jg-pandoc-2.0.5.drv’: 42 dependencies couldn't be built
cannot build derivation ‘/nix/store/faansijryr9m4nwsr4h6h98bhdijxx37-xen-4.5.5.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/dhhm1z5nrs3drcdgbyccv1sxnq1svn2l-libvirt-3.10.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/dhhm1z5nrs3drcdgbyccv1sxnq1svn2l-libvirt-3.10.0.drv’ failed

grahamc · 2017-12-27T00:51:46Z

Hmm @johbo any idea why it is failing on Linux?

johbo · 2017-12-27T08:31:11Z

That's odd indeed, "works on my machine" syndrome.

Maybe it's because I did try from my mac?

$ nix-build -A libvirt --argstr system x86_64-linux
/nix/store/gs6lz3bm9lh62jlkqf89nr7fx322y0nh-libvirt-3.10.0

I'll give it a try in a NixOS VM.

johbo · 2017-12-27T08:31:56Z

It's actually interesting, I assumed that the linux version would be unchanged by my tweak. Must have missed something.

FRidh · 2017-12-27T09:18:05Z

pkgs/development/libraries/libvirt/default.nix

@@ -43,7 +43,9 @@ stdenv.mkDerivation rec {
    # do not use "''${qemu_kvm}/bin/qemu-kvm" to avoid bound VMs to particular qemu derivations
    substituteInPlace src/lxc/lxc_conf.c \
      --replace 'lxc_path,' '"/run/libvirt/nix-emulators/libvirt_lxc",'
-  '' + ''
+  ''
+  # "patchShebangs" causes issues in the Darwin build


What issues? Either explain here or refer to where they are explained (maybe this PR).

good point, I'll add details into the description and point to the PR. Or if anyone can spot the cause, we could also apply a proper fix ;)

Noticed that libvirt did not build on darwin. Tracked the problem down to the call to "patchShebangs" in the hook "preConfigure".

johbo · 2017-12-27T09:34:34Z

I did rebase the change. To my surprise the build in the linux vm did work as well.

lukateras · 2017-12-30T02:19:53Z

I think we should fix patchShebangs proper, given that we maintain it, instead of working around it.

johbo · 2018-01-01T14:13:18Z

@yegortimoshenko your comment made me think, I guess I figured out the reason why it fails due to that 😄

By applying patchShebangs we change the modification time of genprotocol.pl and this causes that make will actually run it.

genprotocol.pl depends on rpcgen being available. On darwin we don't have it available and this leads to the value "no" being set for RPCGEN inside of the Makefile.

If my conclusions are correct, then there is nothing to fix in patchShebangs. As long as we don't have rpcgen available on darwin, we should not apply patchShebangs to make it possible to build libvirt there.

Details of how I came to this conclusion:

Makefile.am contains the following:

%protocol.c: %protocol.x %protocol.h $(srcdir)/rpc/genprotocol.pl
        $(AM_V_GEN)$(PERL) -w $(srcdir)/rpc/genprotocol.pl $(RPCGEN) -c \
               $< $(srcdir)/$(subst $(srcdir)/,,$@)

%protocol.h: %protocol.x $(srcdir)/rpc/genprotocol.pl
        $(AM_V_GEN)$(PERL) -w $(srcdir)/rpc/genprotocol.pl $(RPCGEN) -h \
               $< $(srcdir)/$(subst $(srcdir)/,,$@)

The Makefile which is generated contains this setting:

RPCGEN = no

Causing this issue:

Can't exec "no": No such file or directory at ./rpc/genprotocol.pl line 45.
cannot run no -h locking/lock_protocol.x: No such file or directory at ./rpc/genprotocol.pl line 45.

config.status of the failed build contains these fragments:

S["HAVE_RPCGEN_FALSE"]="#"
S["HAVE_RPCGEN_TRUE"]=""
S["RPCGEN"]="no"

lukateras · 2018-01-01T17:09:13Z

Thanks for thorough debugging!

I've made a fix in #33084, but I can't test it right now (only in a few days).

johbo · 2018-01-02T11:39:01Z

Cool, now I get your point about "fixing" patchShebangs, that's indeed an approach which should work as well.

vcunat · 2018-02-03T10:05:44Z

Well, patchShebangs should be fixed now, but Hydra won't confirm fixing the libvirt issue on Darwin, due to missing libintl in a dependency.

vcunat · 2018-02-10T07:50:40Z

It builds now.

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 0 labels Dec 26, 2017

GrahamcOfBorg reviewed Dec 26, 2017

View reviewed changes

GrahamcOfBorg reviewed Dec 27, 2017

View reviewed changes

FRidh reviewed Dec 27, 2017

View reviewed changes

johbo force-pushed the libvirt-darwin branch from 02b3d7d to e7004c3 Compare December 27, 2017 09:28

libvirt: Avoid patchShebangs in the darwin build

53405eb

Noticed that libvirt did not build on darwin. Tracked the problem down to the call to "patchShebangs" in the hook "preConfigure".

johbo force-pushed the libvirt-darwin branch from e7004c3 to 53405eb Compare December 27, 2017 09:33

lukateras added a commit that referenced this pull request Jan 1, 2018

patchShebangs: preserve times, resolves #33084

d98f4d7

lukateras mentioned this pull request Jan 1, 2018

patchShebangs: preserve times, resolves #33084 #33281

Closed

8 tasks

vcunat closed this in 4b1b6ee Feb 3, 2018

vcunat reopened this Feb 3, 2018

vcunat closed this Feb 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libvirt: Avoid patchShebangs in the darwin build #33084

libvirt: Avoid patchShebangs in the darwin build #33084

johbo commented Dec 26, 2017 •

edited

grahamc commented Dec 26, 2017

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

grahamc commented Dec 26, 2017

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

grahamc commented Dec 27, 2017

johbo commented Dec 27, 2017

johbo commented Dec 27, 2017

FRidh Dec 27, 2017

johbo Dec 27, 2017

johbo commented Dec 27, 2017

lukateras commented Dec 30, 2017

johbo commented Jan 1, 2018

lukateras commented Jan 1, 2018

johbo commented Jan 2, 2018

vcunat commented Feb 3, 2018

vcunat commented Feb 10, 2018

libvirt: Avoid patchShebangs in the darwin build #33084

libvirt: Avoid patchShebangs in the darwin build #33084

Conversation

johbo commented Dec 26, 2017 • edited

Motivation for this change

Things done

grahamc commented Dec 26, 2017

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

grahamc commented Dec 26, 2017

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

grahamc commented Dec 27, 2017

johbo commented Dec 27, 2017

johbo commented Dec 27, 2017

FRidh Dec 27, 2017

Choose a reason for hiding this comment

johbo Dec 27, 2017

Choose a reason for hiding this comment

johbo commented Dec 27, 2017

lukateras commented Dec 30, 2017

johbo commented Jan 1, 2018

lukateras commented Jan 1, 2018

johbo commented Jan 2, 2018

vcunat commented Feb 3, 2018

vcunat commented Feb 10, 2018

johbo commented Dec 26, 2017 •

edited