Yes, so once it's established that the default attribute (say, imagemagick) should refer to a version with "reasonable" amount of support, and let's say now your personal project needs a minimal imagemagick with only PNG support for your Docker container or whatever...

So I think this is a good example of premature optimization. Unless your project will break with a larger version of imagemagick, it should just refer to imagemagick.override { enablePNG = true; }. The problem is going to come up when there's another package in your Docker container that needs to have a bigger imagemagick. Nix doesn't know that your package could have used the larger version of imagemagick so you'll wind up with BOTH imagemagick-light and imagemagick-full. I suppose we could add some concept of "replaceable" overrides in Nix but I don't think anything like that has been proposed.

If you want your docker container to avoid the "big" version of imagemagick because of space constraints, etc., you should do that in an override. This will only work correctly if you're sure that nothing in your closure needs the larger version of imagemagick (otherwise you'll get double like the above). I'm also in favor of having a flag to "imagemagick" that is something like "light" if the code duplication mentioned is too much. My main objection is putting "light" in the attribute name.

So I definitely think this PR is the right way to go - but I understand that people feel uncomfortable with it. Importantly, though, this won't break overrides right now but just try to contain some of the ugliness in one place.

A key insight to all of this is each top-level attribute in all-packages.nix should have a unique "src" attribute. This policy IMO would remove lots of the ugliness in all-packages.nix. The on exception I see is in bootstrapping where things like fetchurlBoot and utillinuxMinimal are probably unavoidable (though perhaps can be hidden). I'd like to make this an policy official.

So I think this is a good example of premature optimization. Unless your project will break with a larger version of imagemagick, it should just refer to imagemagick.override { enablePNG = true; }

If I were to not care about the closure size, why would I be thinking about using overrides at all? First I would just write imagemagick (just like it's done in ~200 places in Nixpkgs already) and see if it includes the necessary stuff I need or not. If it does work and the closure size of that is already not too bad for my purposes I won't be concerning my mind of whether imagemagick even supports disabling various formats in the first place.

The problem is going to come up when there's another package in your Docker container that needs to have a bigger imagemagick. Nix doesn't know that your package could have used the larger version of imagemagick so you'll wind up with BOTH imagemagick-light and imagemagick-full.

But that problem becomes even more likely to happen when overrides are used instead of using things like imagemagickBig. E.g. if Nixpkgs has a package foo which needs say, imagemagick with ghostscript support and bar which needs imagemagick with ffmpeg support, and I have a personal project baz which needs both foo and bar, I would be better off if foo and bar both just depended on imagemagickBig instead each of them depending on imagemagick.override { ghostscriptSupport = true; } and imagemagick.override { ffmpegSupport = true; }.

My main objection is putting "light" in the attribute name. [..] A key insight to all of this is each top-level attribute in all-packages.nix should have a unique "src" attribute.

Why so? Even without this -light business and the ones you mentioned there are several other counterexamples. What about linux_hardened (not the copperhead variant)? What about ubootBeagleboneBlack and ubootRaspberryPi and other U-Boot builds made from the same upstream source? What about libopcodes and libbfd (IIRC)?

Also as mentioned, in the current state of Nixpkgs, the existence of things like imagemagickBig are discoverable with either nix-env or the Packages page on the website, while all those enableFoo flags are not except by reading the source. With things like emacs25-nox the user has a guarantee that they can get the package from the binary cache and don't need to compile the package from source, with .override there is no such guarantee. Until these issues are solved first, deprecating this -light business is a reduction in functionality and should not be done, IMO.

But that problem becomes even more likely to happen when overrides are used instead of using things like imagemagickBig. E.g. if Nixpkgs has a package foo which needs say, imagemagick with ghostscript support and bar which needs imagemagick with ffmpeg support, and I have a personal project baz which needs both foo and bar, I would be better off if foo and bar both just depended on imagemagickBig instead each of them depending on imagemagick.override { ghostscriptSupport = true; } and imagemagick.override { ffmpegSupport = true; }.

It's more likely at least when the default imagemagick doesn't have ffmpegSupport or ghostscriptSupport. But with package overrides we can change the default to optimize our closure:

{
  packageOverrides = pkgs: rec {
    imagemagick = pkgs.defaultOverrides.imagemagickBig;
  };
}

Our new imagemagick will be identical to imagemagick.override { ghostscriptSupport = true; } and imagemagick.override { ffmpegSupport = true; }. AFAICT this only works correctly when packages are being honest about what they really need from imagemagick and not trying to optimize with underspecified concepts like imagemagickMinimal and imagemagickFull.

Why so? Even without this -light business and the ones you mentioned there are several other counterexamples. What about linux_hardened (not the copperhead variant)? What about ubootBeagleboneBlack and ubootRaspberryPi and other U-Boot builds made from the same upstream source? What about libopcodes and libbfd (IIRC)?

So-

• Because linux_hardened is derived from "Linux", it should not be used as a dependency directly. IMO putting multiple Linux derivatives in all-packages.nix invites users to depend on it in their packages and should probably be somewhere else.
• Hopefully some day we can handle the uboot stuff entirely within cross compilation but it probably is not possible right now. Since no package depends on them directly they're probably fine to leave in for legacy purposes (although in principle they should be in their own attribute sets).
• libopcodes and libbfd might have a legit reason for reusing binutils source. I'm not sure why they are packaged like that. From what I understand though, binutils already builds both of them so it should be possible to put them into a separate output under binutils (in theory at least- bootstrapping can get awfully complicated).

Also as mentioned, in the current state of Nixpkgs, the existence of things like imagemagickBig are discoverable with either nix-env or the Packages page on the website, while all those enableFoo flags are not except by reading the source. With things like emacs25-nox the user has a guarantee that they can get the package from the binary cache and don't need to compile the package from source, with .override there is no such guarantee. Until these issues are solved first, deprecating this -light business is a reduction in functionality and should not be done, IMO.

Yes, discoverability is definitely an issue with flags. I wouldn't necessarily say that the status quo is very discoverable either though (for the longest time, I thought emacs25-nox was somehow related to @madjar's nox). One thing that's important to realize though is that we can use things from the binary cache without referring to them directly. Nix's hashing means that it can't tell the difference between:

foo = callPackage ./foo {
  emacs = emac25-nox;
};

and

foo = callPackage ./foo {
  emacs = emacs25.override { withX = false; };
};

and substitution will occur in both.

Part of what is missing from all of this is a good way for Hydra to put overlays into the binary cache. I am looking into doing something like this in #39580 & I think something like a default "headless" overlay in the binary cache could be really useful. For now, though, this PR keeps our overrides in the binary cache (and we can certainly add more).

Ok closing for now. Definitely does not look like there is a consensus to do this. I still think it is worth doing but it could definitely break things.

I have split off parts of this into #39683 and #39686.