Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help? Working on Hydra release builds for master x64 Deb/RPM #1769

Closed
wants to merge 1 commit into from

Conversation

bhipple
Copy link
Contributor

@bhipple bhipple commented Dec 30, 2017

Hydra has been broken since 4cde04f due to some strange behavior with the
sandboxing, and somewhere during that time a debug symbol problem also crept
into the RPM builds. Strangely, neither problem shows up on 32-bit builds.

This is hideous but it gets both Ubuntu and Fedora building again:
$ nix-build release.nix -A rpm_fedora25x86_64 -A deb_ubuntu1604x86_64

The first issue seems to be that /bin/bash doesn't get mounted in the chroot,
but I'm not sure why that is. I verified outside the sandbox call that it is
there and does exist. Not sure on the debug symbols issue. I'll keep digging
around but figured I'd show some hotfixes to demonstrate these are the only
problems and see if anyone else knows more?

@bhipple
Copy link
Contributor Author

bhipple commented Dec 30, 2017

Not really intending for this to be merged as-is (hoping someone has some insight into the VM tools/debug symbols/chroot issues and can enlighten me), though it would be nice to get the Hydra release build in the green again.

Hydra has been broken since 4cde04f due to some strange behavior with the
sandboxing, and somewhere during that time a debug symbol problem also crept
into the RPM builds. Strangely, neither problem shows up on 32-bit builds.

This is hideous but it gets both Ubuntu and Fedora building again:
  $ nix-build release.nix -A rpm_fedora25x86_64 -A deb_ubuntu1604x86_64
  <passes everything>

The first issue seems to be that /bin/bash doesn't get mounted in the chroot,
but I'm not sure why that is. I verified outside the sandbox call that it is
there and does exist. Not sure on the debug symbols issue. I'll keep digging
around but figured I'd show some hotfixes to demonstrate these are the only
problems and see if anyone else knows more?
@dtzWill
Copy link
Member

dtzWill commented Dec 31, 2017

Not sure what to do about the debuginfo problem, but your fix seems reasonable if it's a known issue with the tools shipping with released Fedora.

Searching around...

FWIW, here's a log snippet from the Fedora failure:

+ find /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64 -name '*.la' -exec rm -f '{}' ';'
+ mkdir -p /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/store
+ chmod 1775 /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/store
+ for d in profiles gcroots
+ mkdir -p /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/var/nix/profiles/per-user
+ chmod 1777 /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/var/nix/profiles/per-user
+ for d in profiles gcroots
+ mkdir -p /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/var/nix/gcroots/per-user
+ chmod 1777 /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/nix/var/nix/gcroots/per-user
+ chmod -x /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/etc/profile.d/nix.sh
+ rm -f /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/share/doc/nix-doc-1.12pre5828_56591f6/README
+ rm -rf /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/etc/init
+ /usr/lib/rpm/find-debuginfo.sh --strict-build-id -m --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 /tmp/rpmout/BUILD/nix-1.12pre5828_56591f6
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/libexec/nix/build-remote
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/libexec/nix/buildenv
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/lib64/libnixmain.so
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/lib64/libnixexpr.so
extracting debug info from /tmp/rpmout/BUILDROOT/nix-1.12pre5828_56591f6-2.fc25.x86_64/usr/lib64/libnixstore.so
/usr/lib/rpm/find-debuginfo.sh: line 127:  3683 Bus error               eu-strip --remove-comment $r $g -f "$1" "$2"
error: Bad exit status from /var/tmp/rpm-tmp.r9d216 (%install)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.r9d216 (%install)
[  338.625208] reboot: Power down

@bhipple
Copy link
Contributor Author

bhipple commented Jan 3, 2018

I can confirm that the fix in #1770 ALSO fixes the RPM sandbox test cases! Good catch on /lib64 @dtzWill!

That just leaves the debug symbols for the RPMs. That ticket hints that eu-strip throws that error when it runs out of memory while separating debug symbols, and sure enough, if we increase our VM memory from 1028mb to 2048mb we get working RPM builds again for Fedora! Awesome! I opened #1773 which has this fix, and assuming that the extra VM memory isn't a problem for Hydra it should get our builds back in the green.

@bhipple bhipple closed this Jan 3, 2018
@dtzWill
Copy link
Member

dtzWill commented Jan 3, 2018

Oh, hooray! Good call on VM memory, looking forward to seeing some green builds! 😁

@vcunat
Copy link
Member

vcunat commented Jan 3, 2018

bus error? I suppose they use mmap...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants