Looks good!

One thing to check for is how this affects load times for the web page. I'm not sure how good JSON performance is in web browsers. If its unacceptably slow, we can definitely create a “packages-all.json” or similar specifically for Repology.

Other potential sets to add:

• emacsPackagesNg
• pythonPackages
• nodePackages
• perlPackages

A few more such sets and we'll top the repology list :-D (e.g. texlive, off the top of my head)

Is it worth listing all the libraries? As a user the primary packages that I am interested in are the programs. It doesn't do much for me if there are 10k libraries which I need to filter through to find interesting programs.

If we can (programatically) determine whether a package is software or library (what are fonts? what are themes? what are libraries with /bin/ paths? can /libexec/ be software?), they could be tagged in the json so they can be filtered out (by default) and added back in as needed. They could also be shown in a different way, not only library packages, but e.g. font packages could link to the fonts documentation, icon themes too.

Why add them as needed? Sometimes it is necessary to find the name of a library. This would reduce the amount of noise when searching while still making them available.

This is probably something for another issue, though.

Yes introducing a category attribute in meta might be nice.

Is it worth listing all the libraries?

I think that conversation is off-topic for this particular PR. We currently do include all kinds of libraries in the package list -- this is by no means a new phenomenon these patches create. If you'd like to change that policy and constrain the index to executable-only packages, then I'd prefer that you open a new issue to discuss that, because I'd rather not see this PR derailed by a discussion that's actually not related to its subject matter.

Maybe just have checkboxes for the categories, if we're not constrained with too big json. But I agree it belongs into another thread.

Sure, I didn't mean to derail the conversation. This is already an improvement so it should be merged in.

/cc @edolstra @domenkozar who have write access

How big does the JSON file become with this?

Before:

$ ls -lh nixpkgs/packages*json
-rw-r--r-- 1 simons users 14M Apr 20 14:41 nixpkgs/packages.json
-rw-r--r-- 1 simons users 19M Apr 20 14:41 nixpkgs/packages-unstable.json

After:

$ ls -lh nixpkgs/packages*json
-rw-r--r-- 1 simons users 35M Apr 20 14:39 nixpkgs/packages.json
-rw-r--r-- 1 simons users 59M Apr 20 14:40 nixpkgs/packages-unstable.json

At some point we might want to split away some of these big sets and provide a combo-box for the user to select which set to search. I expect it's still bearable – it's gzipped, currently 1.5 MiB and I guess it will rise to ~3 MiB.

While derailing this discussion a bit, this situation may be solved by relying on a server-side component for the search. This could mean either a custom-bit of code that does the search, returning server-side rendered pages and (possibly) API-consumable endpoints. Server-side rendering could allow search-engine indexation of the results, and would allow non-javascript users [e.g. dillo, elinks] to search packages. Though, this introduces server-side moving parts to the (elegantly) simpler static pages.

Alternatively, we could use services like Algolia which are built for search. Though this introduces a third-party to the website, which I understand if it is not wanted. (While unusual, it is still possible to do server-side rendering with Algolia results, meaning no state is strictly needed on the server if the feature is wanted.)

Both solutions could allow indexing much more data about the packages, while ensuring no long downloads are needed.

While this is all possible, It may be necessary to continue providing the .json file for other users (e.g. repology).

If this approach is desired, or a discussion is needed, an issue can and should be opened.

@peti if that's too big for package.html, why not just have something like nixpkgs/packages-all.json, and we can give that URL to repology to parse?

I suppose we can, but someone else will have to worry about it, because I am not going to pursue this topic.

It sucks that this PR has been lying around dead for over a month by now. I have no clear feedback about what's wrong with it. I have no feedback about what I'd have to change to get it merged either. So I'd rather not waste any more time on this issue than I already have.

FWIW a 3 MB JSON file seems okay to me.

@edolstra: so we merge this as it is, right? (I don't have write access.)

Pinging on this. Not that it's a critical issue or anything - it just doesn't seem like there's any objections.

This moved our channels to the very top in repology :-D Of course, such counts don't really tell much about a distribution...