Remove uses of libOnly #39683

matthewbauer · 2018-04-29T20:12:36Z

We failed to get any sort of consensus on #39515 so I am opening this spinoff PR that focuses on the worst overrides offenders. Basically, a few packages rely on this legacy "libOnly" flag that tells them to only build the "library" part of the package.

What's wrong with libOnly

libOnly is an attempt to be more precise on what a package builds. The goal is to decrease closure size. Unfortunately, it's usage also means that you have to build each package twice - once for the libraries and another for the actual runtime (even though the runtime will still build the library). Since multiple outputs have been released, this is no longer the correct way to decrease closure size. Outputs like "lib" and "dev" should be used instead and mean you only have to build things once.

Current offenders

libpulseaudio
libjack2
libheimdal
libkrb5
libffado

Backwards compatibility

This PR retains aliases for each of the libOnly overrides so old references to "libpulseaudio", and "libjack", etc. will continue to work.

It looks like libkrb5 is still needed for bootstrapping.

Overrides should only be on the "default" package not the other way around.

GrahamcOfBorg · 2018-04-30T00:46:28Z

Failure on aarch64-linux (full log)

Attempted: krb5, libkrb5, pulseaudio

Partial log (click to expand)

make: *** [Makefile:1523: all-recurse] Error 1
builder for '/nix/store/5wh2j5gzqp3g36lzz30l2jzvbbj7nzkg-krb5-1.15.2.drv' failed with exit code 2
cannot build derivation '/nix/store/6a451k4z3fn78v5v1zg7mm43ybfhx9ks-curl-7.59.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/3gzd1q9244gcfsjxddnpx5hi3l3dsv2s-cyrus-sasl-2.1.26.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/wjls65bjzvxbxwnvd9jm4xsgh0x1g886-libmicrohttpd-0.9.59.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ccl6a6j3ddyasm678b3jpr7s92n1xa6d-openldap-2.4.45.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/19ybmj7zd1ah8f9b910dhhfhlgi19kd2-krb5-1.15.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/7mwd9sdj1hgr3nwyxaqd80zldxiid4jq-systemd-238.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/wpq0l3pc1wx31vp34pi7wdqky8r7p2s4-pulseaudio-11.1.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/19ybmj7zd1ah8f9b910dhhfhlgi19kd2-krb5-1.15.2.drv', '/nix/store/5wh2j5gzqp3g36lzz30l2jzvbbj7nzkg-krb5-1.15.2.drv', '/nix/store/wpq0l3pc1wx31vp34pi7wdqky8r7p2s4-pulseaudio-11.1.drv' failed

GrahamcOfBorg · 2018-04-30T00:47:00Z

Failure on x86_64-darwin (full log)

Attempted: krb5, libkrb5, pulseaudio

Partial log (click to expand)

/nix/store/51722z52v6mawpxvxagclrv9srl7blf9-pulseaudio-11.1/lib/pulse-11.1/modules/libprotocol-http.dylib: fixing dylib
/nix/store/51722z52v6mawpxvxagclrv9srl7blf9-pulseaudio-11.1/lib/pulse-11.1/modules/libprotocol-native.dylib: fixing dylib
/nix/store/51722z52v6mawpxvxagclrv9srl7blf9-pulseaudio-11.1/lib/pulse-11.1/modules/libprotocol-simple.dylib: fixing dylib
/nix/store/51722z52v6mawpxvxagclrv9srl7blf9-pulseaudio-11.1/lib/pulse-11.1/modules/librtp.dylib: fixing dylib
/nix/store/51722z52v6mawpxvxagclrv9srl7blf9-pulseaudio-11.1/lib/pulseaudio/libpulsecommon-11.1.dylib: fixing dylib
/nix/store/51722z52v6mawpxvxagclrv9srl7blf9-pulseaudio-11.1/lib/pulseaudio/libpulsecore-11.1.dylib: fixing dylib
strip is /nix/store/kdff2gim6417493yha769kh00n63lnrw-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/vbhj3x9sika75w0vgz5xqq9pmq5f6ifc-pulseaudio-11.1-dev/lib
patching script interpreter paths in /nix/store/vbhj3x9sika75w0vgz5xqq9pmq5f6ifc-pulseaudio-11.1-dev
error: build of '/nix/store/adacsch7b2afxbr5q7nvnsjk13xd88vp-krb5-1.15.2.drv', '/nix/store/y7hi7jrxp1xszj72zm85i9b875ynprf8-krb5-1.15.2.drv' failed

GrahamcOfBorg · 2018-04-30T00:47:54Z

Failure on x86_64-linux (full log)

Attempted: krb5, libkrb5, pulseaudio

Partial log (click to expand)

make: *** [Makefile:1523: all-recurse] Error 1
builder for '/nix/store/hkk5hrpfmjwk98rfqnv4cyjk94yzqah6-krb5-1.15.2.drv' failed with exit code 2
cannot build derivation '/nix/store/mpkzcp5j8wl49jaqac7f798r1pkcwndg-curl-7.59.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/2jsl5bg45820m5n2hicms2d6vwar5sw5-cyrus-sasl-2.1.26.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/j33rv0kdygb9cwp84ghia7j7047dfhxq-libmicrohttpd-0.9.59.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/pc9z91l5sjqc8yscp4ldvyh02m8qj33s-openldap-2.4.45.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/9cy10g8wm8b8by4sghfqqw0zflryxw63-krb5-1.15.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/c2y7qv64xr1ypj3f3d2di42l8m7pcbal-systemd-238.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/v5ldgwar7cfdabbklwwznn3hxsbxr1p0-pulseaudio-11.1.drv': 1 dependencies couldn't be built
error: build of '/nix/store/9cy10g8wm8b8by4sghfqqw0zflryxw63-krb5-1.15.2.drv', '/nix/store/hkk5hrpfmjwk98rfqnv4cyjk94yzqah6-krb5-1.15.2.drv', '/nix/store/v5ldgwar7cfdabbklwwznn3hxsbxr1p0-pulseaudio-11.1.drv' failed

Ericson2314

Excellent! Hopefully after these no-brainer split-outs we can get back to tackling the original problem.

matthewbauer · 2018-04-30T19:24:07Z

@Ericson2314 It looks like we are getting infinite recursion with krb5 (curl depends on it, so anything using fetchurl needs it). I wonder if this is a case where we really need "libOnly"?

There are a few similar "bootstrapping" derivations that are needed for similar reasons like:

fetchurlBoot
libkrb5
utillinuxMinimal
maybe more?

It would be nice to hide these so people don't accidentally use them directly. Not sure if that is possible though.

Ericson2314 · 2018-05-01T01:04:11Z

ugh this bootstrapping stuff is no fun. These days I'm not sure why we don't use the corepkg fetchurl more.

When we redo bootstrapping to use the cross stuff, we can finally get rid of these derivations. Making them *Boot in the meantime like you did might help us keep track of them, shrug.

dezgeg · 2018-05-01T13:53:09Z

Something similar is being done in #38494, might want to coordinate to avoid duplicate work.

oxij · 2018-05-01T14:23:09Z

Well, thinking some more about this, I'm now unsure removing all of `libOnly` is a good idea anyway. I agree it shouldn't be overused, but `libOnly` is useful, not only for bootstrapping. E.g. in case of `pulseaudio` and `jack`, those are pretty big pieces of software, but their libraries are pretty small, so, until we build against `libcardiacarrest` and (yet non-existent) `libemptynest` (or whatever) by default, having `libOnly` makes sense. It shouldn't be set by default, when possible, for hydra builds, I agree, but it would be nice to keep the option for independent builders (like me) that could then override things they don't need binaries for with `libOnly`. (Actually, now what I think about it, it would be cool to have a maintained library of standard override profiles, like "everything minimal", "everything full", "suckless only", etc.) What I think is totally reasonable part of motivation behind this PR is a homogeneous derivations for both values of `libOnly`. I.e. enabling `libOnly` should only produce `lib` (and `dev`, etc) outputs, and disabling would produce all of them. In short: I think we should keep `libOnly`, but set it to `false` by default, and make derivation outputs homogeneous.

matthewbauer · 2018-05-01T14:43:54Z

Closing because #38494 is further along.

oxij · 2018-05-01T14:51:27Z

Oops, I got carried away and forgot that `Only` in `libOnly` means something different from my memory model, hence I should've used different terms. What I wanted to say is I want derivations to become monoids. So, I think, `libOnly` should be split into `withBinaries` and `withLib` (or similar), because both are useful. (The rest of the previous comment is ok.) Then, we can continue from the parent issue on the usage of `.override`s in derivation themselves for specifying features they expect from dependencies. Clearly, derivations with use-flags are monoids (`libOnly` is not a use-flag, while `withLib` and `withBinaries` are). Hence, when nixpkgs finally implements use-flags we could make packages specify expected features of their dependencies and export that information, and then merge all those requests using `bool any` (see discussion in #37252) monoid (or something more interesting, if we want use-_options_ instead of use-_flags_) and build `pkgs` using that. E.g., you gather all "use-flag requests" from packages in `systemPackages`, build use-flag specification, supply it to `pkgs`, now you have the perfect package set. Merge some "hydra-use-flags" into there, and now you only rebuild things that hydra didn't build. Note that the gathering part is exactly isomorphic to `options + config` of NixOS modules, while the rest is exactly isomorphic to monoidal services (see the "Service composition" thread from 2014-12-26 on nix-dev ML by @esoeylemez and #35900).

matthewbauer added 5 commits April 29, 2018 14:56

heimdal: remove libOnly

b0dd289

libffado: remove libOnly

8d0dd5b

krb5: remove libOnly

06baa4e

jack2: remove libOnly

d4819ba

pulseaudio: remove libOnly

390a509

matthewbauer requested review from dezgeg, edolstra and wkennington April 29, 2018 20:12

libheimdal: remove lingering reference

cd7bc9a

matthewbauer force-pushed the remove-libonly branch from 31045c8 to 8cc82a6 Compare April 29, 2018 20:57

libkrb5: restore override

12c528d

It looks like libkrb5 is still needed for bootstrapping.

matthewbauer force-pushed the remove-libonly branch from 8cc82a6 to 12c528d Compare April 29, 2018 21:07

matthewbauer added 4 commits April 29, 2018 16:20

libv4l: add alias

673d5e2

poppler_utils: move to poppler.biin

68a7495

all-packages: normalize overrides

89405ec

Overrides should only be on the "default" package not the other way around.

ffado: resolve recursion

db7356c

matthewbauer force-pushed the remove-libonly branch from 3a02b6b to db7356c Compare April 29, 2018 22:06

matthewbauer changed the title ~~libOnly: considered harmful~~ Remove uses of libOnly Apr 29, 2018

matthewbauer mentioned this pull request Apr 29, 2018

Add top-level/overrides.nix for package overrides #39515

Closed

pulseaudio: fixup

2416bf0

GrahamcOfBorg added 8.has: clean-up 8.has: package (new) 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Apr 30, 2018

matthewbauer force-pushed the remove-libonly branch from 01f492e to b92e221 Compare April 30, 2018 03:39

Ericson2314 approved these changes Apr 30, 2018

View reviewed changes

libkrb5: resolve infinite recursion

a7a6c2d

matthewbauer force-pushed the remove-libonly branch from b92e221 to a7a6c2d Compare April 30, 2018 22:27

matthewbauer closed this May 1, 2018

matthewbauer mentioned this pull request May 1, 2018

[WIP] Refactor libprefixed packages to multioutput #38494

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove uses of libOnly #39683

Remove uses of libOnly #39683

matthewbauer commented Apr 29, 2018 •

edited

GrahamcOfBorg commented Apr 30, 2018

GrahamcOfBorg commented Apr 30, 2018

GrahamcOfBorg commented Apr 30, 2018

Ericson2314 left a comment

matthewbauer commented Apr 30, 2018

Ericson2314 commented May 1, 2018 •

edited

dezgeg commented May 1, 2018

oxij commented May 1, 2018 via email

matthewbauer commented May 1, 2018

oxij commented May 1, 2018 via email

Remove uses of libOnly #39683

Remove uses of libOnly #39683

Conversation

matthewbauer commented Apr 29, 2018 • edited

What's wrong with libOnly

Current offenders

Backwards compatibility

GrahamcOfBorg commented Apr 30, 2018

GrahamcOfBorg commented Apr 30, 2018

GrahamcOfBorg commented Apr 30, 2018

Ericson2314 left a comment

Choose a reason for hiding this comment

matthewbauer commented Apr 30, 2018

Ericson2314 commented May 1, 2018 • edited

dezgeg commented May 1, 2018

oxij commented May 1, 2018 via email

matthewbauer commented May 1, 2018

oxij commented May 1, 2018 via email

matthewbauer commented Apr 29, 2018 •

edited

Ericson2314 commented May 1, 2018 •

edited