[WIP] Recursive debian packages #58777

matthiasbeyer · 2019-04-02T11:33:06Z

Motivation

We (as in "my employer and I") need to package software for debianoids (and RPM, but we're only looking into debian here).
I am therefore exploring the possibilities nix/nixpkgs gives us for building different versions for different customers with different patches and so on and so forth. Everything that is easy with nixpkgs (like custom compileflags, etc etc).

But, we need to ship .deb/.rpm packages. That is obviously not that easy with nixpkgs. There is support for building debian packages, which is releaseTools.debBuild - which only "kinda" works, as it leaves /nix artifacts in the resulting .deb - but that is a different topic.

So what am I doing here is: I try to build a whole dependency tree (all transitive dependencies) of a package as debian packages.

Prerequisites

I want to work with only nixpkgs and nix, no other tooling required

Expected outcome

I can build a nixpkgs package and I get all required dependencies as .deb.
That means that I build multitail and get ncurses.deb and multitail.deb or I build octaveFull and I get <long list of .deb files here>.

Current status

My functions can generate the attribute sets for the builder function (releaseTools.debBuild) pretty well... it works ~95% of the time.

Unfortunately, I can not do the full job:

$ nix repl # in nixpkgs clone
:l ./default.nix
pkgs.buildRecDebpkg.debPackageFor pkgs.multitail "multitail-deb"
error: cannot coerce a set to a string, at /home/user/nixpkgs/pkgs/build-support/release/debian-build.nix:34:5

And I just do not get where this error comes from.

My request

So if some can have a look at my patches, I would welcome

Feedback on the general approach
Pointers on how to improve the approach or whether it is completely insane/bogus or whether there's a better way to do this
Pointers on where my error comes from and what to do about it

Note

This is WIP and should not be considered for inclusion.

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

Required for debugging when developing with the debBuild function. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

This is a WIP patch and should not hit nixpkgs as-is. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

matthiasbeyer · 2019-04-03T06:44:44Z

Carefully pinging some previous authors of the files I touched here: @abbradar @rbvermaa @Profpatsch

Profpatsch · 2019-04-03T09:42:45Z

I’ll look at it later, maybe I can see where the error comes from.

tomberek · 2019-04-10T19:25:10Z

This line [0] does not seem to do the right thing.

Replacing it with this seems to do what you want. There is still an bug with multiple layers of debs, but perhaps this would help track down the error.


      prePhases = "installExtraDebsPhase sysInfoPhase";
+      inherit src diskImage stdenv checkinstall fsTranslation debProvides debRequires;
      name = name + "-" + diskImage.name + (if src ? version then "-" + src.version else "");

Side note: in the past i've used https://github.com/matthewbauer/nix-bundle as a way to solve a similar problem (used the appimage approach). It made the .deb installs completely independent of the debian host - which was convenient.

[0]

nixpkgs/pkgs/build-support/release/debian-build.nix

Line 31 in 6ca571c

// removeAttrs args ["vmTools"] //

matthiasbeyer · 2019-04-11T11:10:41Z

Now I get to build something, yay.

But the output hangs for hours (it literally has been about 1.5 hrs right now) at

[1/0/3 built, 1 copied (0.1/0.1 MiB), 0.1 MiB DL] building ncurses-6.1-20190112-debian-8.11-jessie-amd64 (installPhase): ======================== Installation successful ==========================

When building with

nix repl
$ :l ./default.nix
$ :b pkgs.buildRecDebpkg.debPackageFor pkgs.multitail

matthiasbeyer · 2019-04-11T13:33:32Z

The latest patch does not even work. I don't know what is going on. What I tried:

$ nix repl
> :l ./default.nix
> pkgs.buildRecDebpkg.makeDeb { pkg = pkgs.multitail; }

# fails with
error: cannot coerce a set to a string, at /home/user/nixpkgs/pkgs/build-support/release/debian-build.nix:33:5

> pkgs.buildRecDebpkg.debPackageFor { pkg = pkgs.multitail; name = "multitail-deb"; }
_lambda @ /home/user/nixpkgs/pkgs/build-support/build-debian/default.nix:42:24_

# but it should yield a package, not a lambda! At least from what I see.

matthiasbeyer · 2019-04-15T13:29:33Z

So I constantly get

error: cannot coerce a set to a string, at /home/user/nixpkgs/pkgs/build-support/release/debian-build.nix:32:3

No matter what I try. I even hard-coded "name" in pkgs/build-support/release/debian-build.nix to be a string and got the very same error. I don't know what is happening!

matthiasbeyer · 2019-04-16T07:39:32Z

So the last patch rewrites some debian-build infrastructure (which was in nixpkgs before I started this PR) and now it compiles for me (only a no-dependency-package):

$ nix repl
> :l ./default.nix
> :b (builtins.elemAt (pkgs.buildRecDebpkg.makeDeb pkgs.multitail) 1) 

this derivation produced the following outputs:
  out -> /nix/store/m7a0yxdc7ab0fd3xjg4g2y3784k4y8jh-debian-build-debian-8.11-jessie-amd64

But as you can see, only the dependency of multitail builds (ncurses, which has no dependencies in this case). I cannot build multitail itself, though:

> :b (builtins.elemAt (pkgs.buildRecDebpkg.makeDeb pkgs.multitail) 0) 
builder for '/nix/store/8dr0yyv6mcccvnfap3b13n187c14if2z-debian-build-debian-8.11-jessie-amd64.drv' failed with exit code 2; last 10 log lines:
  build flags: SHELL=/bin/sh
  gcc -funsigned-char -D`uname` -DVERSION=\"6.4.2\" -DCONFIG_FILE=\"/etc/multitail.conf\" -DUTF8_SUPPORT -D_FORTIFY_SOURCE=2 -O3   -c -o utils.o utils.c
  In file included from utils.c:34:0:
  mt.h:63:29: fatal error: ncursesw/panel.h: No such file or directory
    #include <ncursesw/panel.h>
                               ^
  compilation terminated.
  <builtin>: recipe for target 'utils.o' failed
  make: *** [utils.o] Error 1
  [   29.134518] reboot: Power down
[0 built (1 failed)]
error: build of '/nix/store/8dr0yyv6mcccvnfap3b13n187c14if2z-debian-build-debian-8.11-jessie-amd64.drv' failed

That's because multitail does not get the dependency ncurses in the qemu-vm that builds the package. My question: How to solve this?

This commit rewrites the debian-build expression because it was cluttered with uneccessary attribute sets. All sets were merged into one set. The attribute "name" was renamed so it can more easily be distinguished from the "name" attribute of the resulting set. This somehow fixes the build of debian packages in `pkgs.buildRecDebpkg.makeDeb`. I don't know why.

matthiasbeyer · 2019-04-16T08:30:16Z

So from what I see, the ncurses package is _in the /tmp/nix-build-debian-build-debian-8.11-jessie-amd64.drv-1/disk-image.qcow2, but the install process of multitail does not find it.

I'm pinging someone now because I really don't know how to debug this: @tomberek @rbvermaa @abbradar

Does not work. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

matthiasbeyer · 2019-04-26T09:04:19Z

So what I debugged:

the deb files I build as a dependency (ncurses for multitail in my case) are not installed in the image I use for building the package (multitail).
When manually booting the image I can see the dependency only with

find / -name nix-support
cat /nix/store/0jkrxryvfcbbwsjn8zs2q92finlrvwla-debian-build-debian-8.11-jessie-amd64/nix-support/hydra-build-products

But then trying to dpkg -i what I found in the hydra-build-products file:

 dpkg -i /nix/store/0jkrxryvfcbbwsjn8zs2q92finlrvwla-debian-build-debian-8.11-jessie-amd64/debs/ncurses_0.0.0-1_amd64.deb
dpkg: warning: 'ldconfig' not found in PATH or not executable
dpkg: warning: 'start-stop-daemon' not found in PATH or not executable
dpkg: error: 2 expected programs not found in PATH or not executable
Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin

So I'm not sure where the error is here, but I assume that the image is missing some stuff. Someone with more knowledge about the whole setup should probably help me investigate because this might be totally wrong.

stale · 2020-06-01T19:51:50Z

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the
related code and @ mention them in a comment.
Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
irc.freenode.net.

GrahamcOfBorg added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Apr 2, 2019

matthiasbeyer added 3 commits April 2, 2019 14:09

build-support/vm: centos7-x86_64: Update to 7.6.1810

Verified

This commit was signed with the committer’s verified signature.

renovate-bot Mend Renovate

GPG key ID: 81C63AE1C224182B

Verified
Learn about vigilant mode

706fce6

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

debian build: Add assertions

8ff936f

Required for debugging when developing with the debBuild function. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

[WIP] Add recursive debian builder function(s)

6ca571c

This is a WIP patch and should not hit nixpkgs as-is. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

WIP: Fix debian-build

1ee59ae

Try to catch nonexisting attributes

4484b61

Try to bring dependencies (deb files) into scope of build job

a379b8b

Does not work. Signed-off-by: Matthias Beyer <mail@beyermatthias.de>

Add pathes to link, which apparently does not help at all

64d7724

samueldr added the 2.status: work-in-progress label Jul 27, 2019

stale bot added the 2.status: stale label Jun 1, 2020

matthiasbeyer closed this Jun 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Recursive debian packages #58777

[WIP] Recursive debian packages #58777

matthiasbeyer commented Apr 2, 2019

matthiasbeyer commented Apr 3, 2019

Profpatsch commented Apr 3, 2019

tomberek commented Apr 10, 2019

matthiasbeyer commented Apr 11, 2019 •

edited

Loading

matthiasbeyer commented Apr 11, 2019

matthiasbeyer commented Apr 15, 2019

matthiasbeyer commented Apr 16, 2019

matthiasbeyer commented Apr 16, 2019 •

edited

Loading

matthiasbeyer commented Apr 26, 2019 •

edited

Loading

stale bot commented Jun 1, 2020

[WIP] Recursive debian packages #58777

[WIP] Recursive debian packages #58777

Conversation

matthiasbeyer commented Apr 2, 2019

Motivation

Prerequisites

Expected outcome

Current status

My request

Note

matthiasbeyer commented Apr 3, 2019

Profpatsch commented Apr 3, 2019

tomberek commented Apr 10, 2019

matthiasbeyer commented Apr 11, 2019 • edited Loading

matthiasbeyer commented Apr 11, 2019

matthiasbeyer commented Apr 15, 2019

matthiasbeyer commented Apr 16, 2019

matthiasbeyer commented Apr 16, 2019 • edited Loading

matthiasbeyer commented Apr 26, 2019 • edited Loading

stale bot commented Jun 1, 2020

matthiasbeyer commented Apr 11, 2019 •

edited

Loading

matthiasbeyer commented Apr 16, 2019 •

edited

Loading

matthiasbeyer commented Apr 26, 2019 •

edited

Loading