Many derivations don't produce a binary, but several libraries or
scripts. Therefore the bullet point about testing shouldn't focus on
testing binaries only, but on everything which lives in ./result

Yes and no. Testing libraries may be much harder, so the scope really is /bin. In that case, replace "binaries" with "executables"?

A longer term observation: for many libraries we have a primary executable reverse dependency in Nixpkgs, though. And testing just the main use case is definitely faster than a full nox-review wip. Should we open an issue to collect ideas about codification of that and the related guidelines? (Maybe after the release is officially announced).

Not sure if I made my point clear enough, hence another try:

I've been bothered by this item on the checklist for some time now, whenever I patch the derivation of a library (lately several python packages), I also check this item when I loaded the module into a REPL and confirmed that the main API is still functional and no executable in ./result/bin was available. I submitted a simple patch to show what I'd like to change and start a discussion about this to see whether or not I'm the only one.

Testing libraries may be much harder, so the scope really is /bin. In that case, replace "binaries" with "executables"?

ACK. My main point was that the phrasing might be confusing in case of a lib-only derivation.
How do you fell with adding to separate points (tested executables, tested build outputs (e.g. libraries, properly built man pages, ...))?

Should we open an issue to collect ideas about codification of that and the related guidelines

👍 I think finding a suitable solution everyone is happy with might even reduce the amount of breackage because of insufficient testing :-)

I just talked to @fpletz to get further opinions some days ago.
Despite the (absolutely valid argument regarding the effort to test single libraries I already answered too) he doesn't see the need as the problem can/will be solved thanks to the PR bot which is able to check possible rebuilds as well.

As three core maintainers basically have the same position I think that it's absolutely fine to close it for now and continue the discussion in case more people are interested...