gnome3: Package all the Gnome extensions #108414

piegamesde · 2021-01-04T18:41:31Z

Motivation for this change

An automatic way to do this that scales up and requires little manual intervention is really needed. It works by scraping extensions.gnome.org with a python script,
that writes all relevant information into the extensions.json. Every attribute of besaid file can be built into a package using buildShellExtension.

Extensions are grouped by Gnome shell version for practical reasons. Only extensions for Gnome 3.36 and 3.38 were added for now. If desired, we can always add older versions later on.
The extensions are exposed as an attrset, pkgs.gnome36Extensions and pkgs.gnome38Extensions respectively. The package name of each extensions is generated automatically
from its name and also contains its running index ID (e.g. pkgs.gnome36Extensions.system-monitor-120) to avoid collisions.

The old, manually packaged extensions under pkgs.gnomeExtensions are left as they are for now for backwards compatibility. We could link them to the new ones and then remove
the manual packaging, or keep them, or delete them alltogether. We could also link pkgs.gnomeExtensions to pkgs.gnome38Extensions or whatever the most recent version is (similar to what is done for pythonPackages).

You can have a look at all the extensions in my NUR.

Things done

mweinelt · 2021-02-01T16:45:05Z

@GrahamcOfBorg eval

andir

I did a quick pass of the code. There are a few things we can and probably should improve when we want this to land.

I wonder what the usage guidelines of the gnome API are and I worry a bit about licenses of extensions.

pkgs/desktops/gnome-3/extensions/buildGnomeExtension.nix

pkgs/desktops/gnome-3/extensions/update-extensions.py

pkgs/desktops/gnome-3/extensions/default.nix

cole-h · 2021-02-21T04:19:07Z

@ofborg eval

worldofpeace · 2021-02-23T13:58:42Z

I'm not going to suggest my changes on the diff because I think it's more helpful to address this conceptually. I'm going to start with the points that are going to influence how this code looks the most.

Transitioning our current packaging to this

There are a few issues:

Every package needs a maintainer ideally
How the builder works is not going to be, and I don't expect it to be, enough for every package to work in nixos
index ID in package name to avoid collisions?

I'm going to try to explain these in detail and suggest what I think we should do.

Number 1&2

1&2 are linked so I will answer them similarly.
To help me explain this I'm going to go over how I see nodePackages working since it's kinda similar to the issue we have here, except that packaging generates nix and here we don't.
If you already know about nodePackages in detail then sorry in advance since I'll probably be reexplaining things you already understand.
In node-packages/default.nix, there is a detailed file of overrides, this is achievable because buildNodePackage works like many of the other buildXPackage functions in nixpkgs.
So as a first tangible goal there's:

Make buildShellExtension flexible

We can then have overrides for some of these packages so the more complicated extensions can actually work, and have maintainer fields. I would then request that we should split out the overrides into separate files, and not one giant one.

Number 3

There is an alternative to having the index ID be in the name. This would be having every package number by their UUID... I have to say that, to a user, adding gnomeExtensions."dash-to-dock@micxgx.gmail.com" seems a bit more sensible than a "strange number". So second tangible goal:

Name packages after UUID instead of dash separated names and index ID

For now I would say that good examples to see how this works in nixpkgs is buildPythonPackage and node-env.nix buildNodePackage.

Tangents

Another parallel that we can have with nodePackages is node-packages.json. The process of packaging a node-package is to add the program you want to the file and regenerate using the node2nix tool. They don't package all of node, and they probably shouldn't and couldn't 😀 But here I think it's acceptable to expose all of them,
my tab completion said "300 lines". The issue is that it could be interpreted that all 300 packages are going to work. This is difficult because you'll only know... exactly inside gnome-shell. It's not like the haskell packaging where it can be marked as broken right away and someone can file a bug. So maybe a second goal is finding a way to tell the user that "nixpkgs has all the gnome-extensions, most should work but some may be broken". Ideally with documentation and something in the nix output.

I'm going to steer away from documentation being a goal for now because we're just at a point where things might change a lot, but the other point can be implemented in nix code.

I could suggest:

a blocklist and an allow list

meta.broken is exactly this. Though I have the feeling someone might disagree on this, since I believe there has been a discussion about "broken build & broken at runtime", and this has yet to be sorted out. The alternative would be to do nothing and let people report them as broken. I'm thinking likely this will be what we have to do.

Closing

Thank you soo much for your PR @piegamesde ✨
I've been hoping for something like this for a while and you did it.
Lately my availability has been a bit better now, and I'm only choosing to focus on a few PRs and this is one of them. I don't want this to go unnoticed. Also, thank you @andir for the preliminary review.

✌️

piegamesde · 2021-02-23T17:45:57Z

Primary keys

Let's get #⁠3 out of the way first. Every extension has multiple unique identifiers. The UUID is one, and the number at the end of a name is one as well. I chose the $name-$number schema because because it can literally be copied out of the extension's URL and it's better to read IMO. But in the end, I don't really care. We can easily switch to another primary key, I'd be even fine with both. The only thing I don't want is manual names, or some other manual maintenance overhead.

Maintenance

Speaking of which, as long as there is little to no manual overhead to the individual extensions, I'd be willing to maintain them all.

How the builder works is not going to be, and I don't expect it to be, enough for every package to work in nixos

The extensions are installed exactly the way as it would be done through Gnome UI. And except for those that do some platform-specific magic foo (I haven't encountered any yet), all those that work on "normal" Gnome should work as well. But I agree that we need some kind of override mechanism for custom packaging.

Moving on

I propose to implement the following things:

Make buildGnomeExtension overrideable. All custom packaging needs can be done in the default.nix by overriding the specified extensions. In heavy cases, we can completely overwrite the derivation with custom packaging.
The blocklist can be implemented by overriding the broken attribute of the affected extensions.
I don't quite see what the allow list is good for. I think the number of up to date Gnome extensions will be manageable in the foreseeable future. As Nix evaluation is inherently lazy, overwritten extensions simply won't be evaluated.

An alternative proposal would be to only package the buildGnomeExtension and the JSON index together with some convenience tooling, so that everybody effectively assembles the parts themselves. I'm slightly against this, but at least it would solve the "communicate that things may break" issue.

worldofpeace · 2021-02-26T14:11:48Z

An alternative proposal would be to only package the buildGnomeExtension and the JSON index together with some convenience tooling, so that everybody effectively assembles the parts themselves. I'm slightly against this, but at least it would solve the "communicate that things may break" issue.

Yeah, I'm against this as well. As mentioned there is just too few extensions for it to be worth doing that.

Speaking of which, as long as there is little to no manual overhead to the individual extensions, I'd be willing to maintain them all.

Considering you mentioned someway of creating automated testing, I think we're all in good hands 😁

Overall the proposed changes are fine. But I would still harp on the index-id -> uuid change, because out of context it just feels weird to have some numbers with no apparent reason (to the enduser) in the package name. UUID's are typically suited for this purpose and somewhat ubiquitous with certain people.

worldofpeace · 2021-02-26T14:12:57Z

I don't quite see what the allow list is good for. I think the number of up to date Gnome extensions will be manageable in the foreseeable future. As Nix evaluation is inherently lazy, overwritten extensions simply won't be evaluated.

Oops, the allow list is simply what people put in their configuration (to build a broken package), and this already exists.

piegamesde · 2021-02-28T00:34:19Z

@worldofpeace I've thought about this again, and I really don't like using the UUIDs as primary identifiers. These will be the names that people will see (and have to look for) in the search. They are bulky, inconsistent in their casing and comparatively hard to discover. So I somewhat retract my claim about not caring about this.

My proposal for using the informal ID from the extension's URL still stands. But if you prefer, I propose to try remove the number from the name unless there are some collisions.

piegamesde · 2021-02-28T22:40:24Z

I went ahead and renamed all the packages according to my previous comment. I really like the new names a lot more. I did a quick check and there don't seem to be any name collisions so far.

andir

Did another pass through the code again. I am not a gnome user but I want to ensure that we have a decent level of code quality within Nixpkgs. At least on all those PRs that I have time & energy to review.

pkgs/desktops/gnome-3/extensions/update-extensions.py

pkgs/desktops/gnome-3/extensions/buildGnomeExtension.nix

pkgs/desktops/gnome-3/extensions/default.nix

pkgs/desktops/gnome-3/extensions/update-extensions.py

piegamesde · 2021-03-11T01:01:00Z

I did a quick check and there don't seem to be any name collisions so far.

Sadly, I have to take that back. After fixing some embarassing bug, it now finds quite a few collisions. I honestly can't tell if this is fine, or if there are too many and we should go back to one of the other approaches

pkgs/desktops/gnome-3/extensions/update-extensions.py

maralorn

I think this is really great. I am looking forward to using this.

I have a few minor nitpicks but nothing big (and I am no expert in modern python).

Please don‘t merge just because of my approval though. I‘d like to see one or two more.

pkgs/desktops/gnome-3/extensions/default.nix

pkgs/desktops/gnome-3/extensions/update-extensions.py

andir · 2021-03-18T12:42:06Z

pkgs/desktops/gnome-3/extensions/update-extensions.py

+                # If our page isn't "full", it must have been the last one
+                if responseLength < 25:
+                    logging.debug(
+                        f"\tThis page only has {responseLength} entries, so it must be the last one."


Logging should never use f-strings. The idea is that the expensive operation of substituting some strings in there should only be done if the log level requires it and otherwise it should be constant cost to skip this line.

I'll not be pedantic on this here as well this isn't a real application anyway...

Thanks for the notice, but I think changing this would not improve the readability of the code, and the performance impact of this is absolutely negligible.

It isn't about performance but consistent pythonic code style.

pkgs/desktops/gnome-3/extensions/update-extensions.py

An automatic way to do this that scales up and requires little manual intervention is really needed. It works by scraping extensions.gnome.org with a python script, that writes all relevant information into the `extensions.json`. Every attribute of besaid file can be built into a package using `buildShellExtension`. Extensions are grouped by Gnome shell version for practical reasons. Only extensions for Gnome 3.36 and 3.38 were added, as we don't support legacy Gnome versions. The extensions are exposed as an attrset, `pkgs.gnome36Extensions` and `pkgs.gnome38Extensions` respectively. The package name of each extensions is generated automatically from its UUID. The attribute `pkgs.gnomeExtensions` contains the officially packaged and supported extensions set. It contains all the automatically packaged extensions for the current Gnome Shell version, which are overwritten by manually packaged ones where needed. Unlike gnomeXYExtensions, the names are not UUIDs, but automatically generated human- friendly names. Naming collisions – which are tracked in collisions.json – need to be manually resolved in the `extensionRenames` attrset.

nixos-discourse · 2021-03-21T03:56:33Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-are-your-goals-for-21-05/11559/17

pkgs/desktops/gnome-3/extensions/update-extensions.py

Co-authored-by: Andreas Rammhold <andreas@rammhold.de>

pkgs/desktops/gnome-3/extensions/buildGnomeExtension.nix

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

mweinelt · 2021-03-31T23:40:47Z

Sorry, I mixed up the tabs and merged the wrong pull request. Reverted in #118184. Please reopen, @piegames. You might need to create a new branch for GitHub to allow you to create a new PR.

piegamesde · 2021-04-01T11:40:13Z

Moved to #118232

ofborg bot added the 6.topic: GNOME GNOME desktop environment and its underlying platform label Jan 4, 2021

piegamesde force-pushed the gnome-extensions branch from 84e38bf to 5b1cb27 Compare January 4, 2021 18:47

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Feb 1, 2021

andir requested changes Feb 1, 2021

View reviewed changes

ofborg bot added the ofborg-internal-error label Feb 17, 2021

cole-h removed the ofborg-internal-error label Feb 21, 2021

piegamesde force-pushed the gnome-extensions branch 3 times, most recently from f899451 to 7ebc6c7 Compare February 21, 2021 12:25

piegamesde marked this pull request as ready for review February 21, 2021 13:00

piegamesde requested a review from andir February 21, 2021 13:00

piegamesde force-pushed the gnome-extensions branch 5 times, most recently from fb4c962 to a38f0a8 Compare February 27, 2021 23:59

piegamesde force-pushed the gnome-extensions branch from e7ffc63 to 3b266a9 Compare February 28, 2021 22:33

andir requested changes Mar 10, 2021

View reviewed changes

andir reviewed Mar 10, 2021

View reviewed changes

pkgs/desktops/gnome-3/extensions/update-extensions.py Outdated Show resolved Hide resolved

andir reviewed Mar 11, 2021

View reviewed changes

pkgs/desktops/gnome-3/extensions/update-extensions.py Outdated Show resolved Hide resolved

maralorn approved these changes Mar 17, 2021

View reviewed changes

ofborg bot removed the 2.status: merge conflict label Mar 17, 2021

piegamesde force-pushed the gnome-extensions branch 2 times, most recently from e412c7b to bfe7a0c Compare March 17, 2021 22:35

ofborg bot added 10.rebuild-linux: 101-500 and removed 10.rebuild-linux: 501+ 10.rebuild-linux: 501-1000 labels Mar 17, 2021

andir reviewed Mar 18, 2021

View reviewed changes

pkgs/desktops/gnome-3/extensions/update-extensions.py Outdated Show resolved Hide resolved

andir reviewed Mar 18, 2021

View reviewed changes

pkgs/desktops/gnome-3/extensions/update-extensions.py Outdated Show resolved Hide resolved

andir reviewed Mar 18, 2021

View reviewed changes

pkgs/desktops/gnome-3/extensions/update-extensions.py Outdated Show resolved Hide resolved

piegamesde force-pushed the gnome-extensions branch from 75334dd to 345886b Compare March 18, 2021 14:16

maintainers: add piegames

7f0ada8

piegamesde force-pushed the gnome-extensions branch from 345886b to 52bfb84 Compare March 18, 2021 18:16

piegamesde force-pushed the gnome-extensions branch from 52bfb84 to 74463a4 Compare March 18, 2021 18:18

piegamesde requested a review from jtojnar March 19, 2021 20:46

andir reviewed Mar 21, 2021

View reviewed changes

pkgs/desktops/gnome-3/extensions/update-extensions.py Outdated Show resolved Hide resolved

Improve error handling

ed6ea66

Co-authored-by: Andreas Rammhold <andreas@rammhold.de>

SuperSandro2000 reviewed Mar 21, 2021

View reviewed changes

pkgs/desktops/gnome-3/extensions/buildGnomeExtension.nix Outdated Show resolved Hide resolved

pkgs/desktops/gnome-3/extensions/buildGnomeExtension.nix Show resolved Hide resolved

Improve formatting

838a6ea

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>

piegamesde force-pushed the gnome-extensions branch from e447f7b to 838a6ea Compare March 22, 2021 20:13

mweinelt requested a review from a team March 31, 2021 23:28

mweinelt merged commit f764863 into NixOS:master Mar 31, 2021

mweinelt mentioned this pull request Mar 31, 2021

Revert "gnome3: Package all the Gnome extensions" #118184

Merged

mweinelt removed the request for review from a team March 31, 2021 23:41

piegamesde mentioned this pull request May 21, 2021

gnome: Package all the Gnome extensions … again?! #118232

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gnome3: Package all the Gnome extensions #108414

gnome3: Package all the Gnome extensions #108414

piegamesde commented Jan 4, 2021 •

edited

mweinelt commented Feb 1, 2021

andir left a comment

cole-h commented Feb 21, 2021

worldofpeace commented Feb 23, 2021

piegamesde commented Feb 23, 2021

worldofpeace commented Feb 26, 2021

worldofpeace commented Feb 26, 2021

piegamesde commented Feb 28, 2021

piegamesde commented Feb 28, 2021

andir left a comment

piegamesde commented Mar 11, 2021

maralorn left a comment

andir Mar 18, 2021

piegamesde Mar 18, 2021

andir Mar 21, 2021

nixos-discourse commented Mar 21, 2021

mweinelt commented Mar 31, 2021

piegamesde commented Apr 1, 2021

gnome3: Package all the Gnome extensions #108414

gnome3: Package all the Gnome extensions #108414

Conversation

piegamesde commented Jan 4, 2021 • edited

Motivation for this change

Things done

mweinelt commented Feb 1, 2021

andir left a comment

Choose a reason for hiding this comment

cole-h commented Feb 21, 2021

worldofpeace commented Feb 23, 2021

Number 1&2

Number 3

Tangents

Closing

piegamesde commented Feb 23, 2021

Primary keys

Maintenance

Moving on

worldofpeace commented Feb 26, 2021

worldofpeace commented Feb 26, 2021

piegamesde commented Feb 28, 2021

piegamesde commented Feb 28, 2021

andir left a comment

Choose a reason for hiding this comment

piegamesde commented Mar 11, 2021

maralorn left a comment

Choose a reason for hiding this comment

andir Mar 18, 2021

Choose a reason for hiding this comment

piegamesde Mar 18, 2021

Choose a reason for hiding this comment

andir Mar 21, 2021

Choose a reason for hiding this comment

nixos-discourse commented Mar 21, 2021

mweinelt commented Mar 31, 2021

piegamesde commented Apr 1, 2021

piegamesde commented Jan 4, 2021 •

edited