Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Recursive debian packages #58777

Closed

Conversation

matthiasbeyer
Copy link
Contributor

Motivation

We (as in "my employer and I") need to package software for debianoids (and RPM, but we're only looking into debian here).
I am therefore exploring the possibilities nix/nixpkgs gives us for building different versions for different customers with different patches and so on and so forth. Everything that is easy with nixpkgs (like custom compileflags, etc etc).

But, we need to ship .deb/.rpm packages. That is obviously not that easy with nixpkgs. There is support for building debian packages, which is releaseTools.debBuild - which only "kinda" works, as it leaves /nix artifacts in the resulting .deb - but that is a different topic.

So what am I doing here is: I try to build a whole dependency tree (all transitive dependencies) of a package as debian packages.

Prerequisites

I want to work with only nixpkgs and nix, no other tooling required

Expected outcome

I can build a nixpkgs package and I get all required dependencies as .deb.
That means that I build multitail and get ncurses.deb and multitail.deb or I build octaveFull and I get <long list of .deb files here>.

Current status

My functions can generate the attribute sets for the builder function (releaseTools.debBuild) pretty well... it works ~95% of the time.

Unfortunately, I can not do the full job:

$ nix repl # in nixpkgs clone
:l ./default.nix
pkgs.buildRecDebpkg.debPackageFor pkgs.multitail "multitail-deb"
error: cannot coerce a set to a string, at /home/user/nixpkgs/pkgs/build-support/release/debian-build.nix:34:5

And I just do not get where this error comes from.

My request

So if some can have a look at my patches, I would welcome

  1. Feedback on the general approach
  2. Pointers on how to improve the approach or whether it is completely insane/bogus or whether there's a better way to do this
  3. Pointers on where my error comes from and what to do about it

Note

This is WIP and should not be considered for inclusion.

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
Required for debugging when developing with the debBuild function.

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
This is a WIP patch and should not hit nixpkgs as-is.

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
@matthiasbeyer
Copy link
Contributor Author

Carefully pinging some previous authors of the files I touched here: @abbradar @rbvermaa @Profpatsch

@Profpatsch
Copy link
Member

I’ll look at it later, maybe I can see where the error comes from.

@tomberek
Copy link
Contributor

This line [0] does not seem to do the right thing.

Replacing it with this seems to do what you want. There is still an bug with multiple layers of debs, but perhaps this would help track down the error.


      prePhases = "installExtraDebsPhase sysInfoPhase";
+      inherit src diskImage stdenv checkinstall fsTranslation debProvides debRequires;
      name = name + "-" + diskImage.name + (if src ? version then "-" + src.version else "");

Side note: in the past i've used https://github.com/matthewbauer/nix-bundle as a way to solve a similar problem (used the appimage approach). It made the .deb installs completely independent of the debian host - which was convenient.

[0]

// removeAttrs args ["vmTools"] //

@matthiasbeyer
Copy link
Contributor Author

matthiasbeyer commented Apr 11, 2019

Now I get to build something, yay.

But the output hangs for hours (it literally has been about 1.5 hrs right now) at

[1/0/3 built, 1 copied (0.1/0.1 MiB), 0.1 MiB DL] building ncurses-6.1-20190112-debian-8.11-jessie-amd64 (installPhase): ======================== Installation successful ==========================

When building with

nix repl
$ :l ./default.nix
$ :b pkgs.buildRecDebpkg.debPackageFor pkgs.multitail

@matthiasbeyer
Copy link
Contributor Author

The latest patch does not even work. I don't know what is going on. What I tried:

$ nix repl
> :l ./default.nix
> pkgs.buildRecDebpkg.makeDeb { pkg = pkgs.multitail; }

# fails with
error: cannot coerce a set to a string, at /home/user/nixpkgs/pkgs/build-support/release/debian-build.nix:33:5

> pkgs.buildRecDebpkg.debPackageFor { pkg = pkgs.multitail; name = "multitail-deb"; }
_lambda @ /home/user/nixpkgs/pkgs/build-support/build-debian/default.nix:42:24_

# but it should yield a package, not a lambda! At least from what I see.

@matthiasbeyer
Copy link
Contributor Author

So I constantly get

error: cannot coerce a set to a string, at /home/user/nixpkgs/pkgs/build-support/release/debian-build.nix:32:3

No matter what I try. I even hard-coded "name" in pkgs/build-support/release/debian-build.nix to be a string and got the very same error. I don't know what is happening!

@matthiasbeyer
Copy link
Contributor Author

So the last patch rewrites some debian-build infrastructure (which was in nixpkgs before I started this PR) and now it compiles for me (only a no-dependency-package):

$ nix repl
> :l ./default.nix
> :b (builtins.elemAt (pkgs.buildRecDebpkg.makeDeb pkgs.multitail) 1) 

this derivation produced the following outputs:
  out -> /nix/store/m7a0yxdc7ab0fd3xjg4g2y3784k4y8jh-debian-build-debian-8.11-jessie-amd64

But as you can see, only the dependency of multitail builds (ncurses, which has no dependencies in this case). I cannot build multitail itself, though:

> :b (builtins.elemAt (pkgs.buildRecDebpkg.makeDeb pkgs.multitail) 0) 
builder for '/nix/store/8dr0yyv6mcccvnfap3b13n187c14if2z-debian-build-debian-8.11-jessie-amd64.drv' failed with exit code 2; last 10 log lines:
  build flags: SHELL=/bin/sh
  gcc -funsigned-char -D`uname` -DVERSION=\"6.4.2\" -DCONFIG_FILE=\"/etc/multitail.conf\" -DUTF8_SUPPORT -D_FORTIFY_SOURCE=2 -O3   -c -o utils.o utils.c
  In file included from utils.c:34:0:
  mt.h:63:29: fatal error: ncursesw/panel.h: No such file or directory
    #include <ncursesw/panel.h>
                               ^
  compilation terminated.
  <builtin>: recipe for target 'utils.o' failed
  make: *** [utils.o] Error 1
  [   29.134518] reboot: Power down
[0 built (1 failed)]
error: build of '/nix/store/8dr0yyv6mcccvnfap3b13n187c14if2z-debian-build-debian-8.11-jessie-amd64.drv' failed

That's because multitail does not get the dependency ncurses in the qemu-vm that builds the package. My question: How to solve this?

This commit rewrites the debian-build expression because it was
cluttered with uneccessary attribute sets.

All sets were merged into one set.

The attribute "name" was renamed so it can more easily be distinguished
from the "name" attribute of the resulting set.

This somehow fixes the build of debian packages in
`pkgs.buildRecDebpkg.makeDeb`. I don't know why.
@matthiasbeyer
Copy link
Contributor Author

matthiasbeyer commented Apr 16, 2019

So from what I see, the ncurses package is _in the /tmp/nix-build-debian-build-debian-8.11-jessie-amd64.drv-1/disk-image.qcow2, but the install process of multitail does not find it.

I'm pinging someone now because I really don't know how to debug this: @tomberek @rbvermaa @abbradar

Does not work.

Signed-off-by: Matthias Beyer <mail@beyermatthias.de>
@matthiasbeyer
Copy link
Contributor Author

matthiasbeyer commented Apr 26, 2019

So what I debugged:

  • the deb files I build as a dependency (ncurses for multitail in my case) are not installed in the image I use for building the package (multitail).
  • When manually booting the image I can see the dependency only with
find / -name nix-support
cat /nix/store/0jkrxryvfcbbwsjn8zs2q92finlrvwla-debian-build-debian-8.11-jessie-amd64/nix-support/hydra-build-products

But then trying to dpkg -i what I found in the hydra-build-products file:

 dpkg -i /nix/store/0jkrxryvfcbbwsjn8zs2q92finlrvwla-debian-build-debian-8.11-jessie-amd64/debs/ncurses_0.0.0-1_amd64.deb
dpkg: warning: 'ldconfig' not found in PATH or not executable
dpkg: warning: 'start-stop-daemon' not found in PATH or not executable
dpkg: error: 2 expected programs not found in PATH or not executable
Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin

So I'm not sure where the error is here, but I assume that the image is missing some stuff. Someone with more knowledge about the whole setup should probably help me investigate because this might be totally wrong.

@stale
Copy link

stale bot commented Jun 1, 2020

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

  1. Search for maintainers and people that previously touched the
    related code and @ mention them in a comment.
  2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
    irc.freenode.net.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 2.status: work-in-progress 10.rebuild-darwin: 0 10.rebuild-linux: 0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants