IMO we'd just continue using packageOverrides in this case. Re-importing nixpkgs is very ugly, and all the user overlays will be lost inside of the reimport, not to mention system and configuration settings.

packageOverrides is to override the package set from outside of it. Overlays are considered the superior solution and it is thus replacing it.

In this case, what is needed is a new scope. All the sub package sets use this as well, see e.g. Qt5:

libsForQt59 = lib.makeScope qt59.newScope mkLibsForQt5;

In order to make it possible to modify some of the Sage-specific overrides, one could offer this package set and allow a self: super: {}. See e.g. the Python package set.

Re-importing nixpkgs is very ugly, and all the user overlays will be lost inside of the reimport, not to mention system and configuration settings.

Yeah that's what I feared. I wish it was as easy as it is with the python package set.

In this case, what is needed is a new scope.

Can you help me with that? Unfortunately it seems like that isn't documented anywhere. I would need a scope that has all of the regular nixpkgs, but with some overrides. All the scopes I'm aware of only have new packages, not the rest of nixpkgs.

In order to make it possible to modify some of the Sage-specific overrides

That's probably overkill. At best there shouldn't be any sage specific overrides at all, whenever possible I try to backport compatibility patches instead.

@timokau see FRidh@c6a649a

The withDoc part will need to be fixed though.

Might as well finish it as well:
FRidh@c6e27de

Please test.

Thanks! Thats a lot to read through, I'll do that and test it tomorrow.

Unfortunately that does break the build:

Error compiling Cython file:
------------------------------------------------------------
...
from sage.libs.gmp.mpz cimport *
from sage.libs.flint.fmpz_poly cimport *
from sage.libs.flint.nmod_poly cimport *
from sage.libs.flint.ulong_extras cimport *

from cypari2.paridecl cimport (GEN, cgetg, t_POL, set_gel, gel, stoi, lg,
^
------------------------------------------------------------

sage/schemes/elliptic_curves/descent_two_isogeny.pyx:33:0: 'cypari2/paridecl/hyperellratpoints.pxd' not found

I'll figure out the details when I have more time.

Regarding the changes in general:

lib.fix' (lib.extends overrides (lib.extends packages (self: pkgs // {callPackage = pkgs.newScope self;})))

What does that do? I know what lib.fix is supposed to do, but I don't know about the rest.

sagePackages = callPackage ../applications/science/math/sage { };

Will this show up in search results?

Unfortunately that does break the build:

I think there is something not OK with the callPackage here, or we have to explicitly pass in things with self.

lib.fix' (lib.extends overrides (lib.extends packages (self: pkgs // {callPackage = pkgs.newScope self;})))

Create a new package set that extends pkgs with a callPackage that looks into the current self when passing arguments to function calls:

self: pkgs // {callPackage = pkgs.newScope self;}

Take our "overlay" of modifications and extend our new package set with it:

lib.extends packages (self: pkgs // {callPackage = pkgs.newScope self;})

Extend once more with overrides "overlay" and then fix the final set:

lib.fix' (lib.extends overrides (lib.extends packages (self: pkgs // {callPackage = pkgs.newScope self;})))

Will this show up in search results?

No, we don't recurse into that attribute set so it won't show up in say nix-env -q or nix search. But, you can access the attributes.

The issue is that cypari2 doesn't use our version of pari. So, something is not yet correct with this new method.

This should do the trick, although I'm not sure whether that can go in as I expect it to take up some memory.

Create a new package set that extends pkgs with a callPackage that looks into the current self when passing arguments to function calls:

Thanks for your explanation. The comments in lib/fixed-points.nix are pretty helpful too.

No, we don't recurse into that attribute set so it won't show up in say nix-env -q or nix search. But, you can access the attributes.

That's perfect.

This should do the trick, although I'm not sure whether that can go in as I expect it to take up some memory.

Okay, lets wait for @peti's answer.

That still doesn't quite build, but thats a different problem. We have to explicitly pass the nixpkgs version of pynac to sagelib again to keep it from using the python version. Then it works (without any rebuild even, proving that it is equivalent to my re-import overlay solution).

Should those overlaps in namespaces be considered an issue and fixed? They lead to pretty subtle bugs...

Also apart from the "overlay" changes I see that you're splitting the python and non-python parts further apart. For example moving python package to its own file (making it necessary to inherit them again) and using python.pkgs instead of passing them as arguments like regular pkgs.

Are there reasons for this? Wouldn't it be nicer to treat python pkgs and "regular" pkgs as uniformly as possible?

Also if fix -> fix' isn't acceptable, can't we use the same mechanism overlays use to make it possible to do deep overrides from within nixpkgs? Its not a unique sage problem, sage just needs it at a bigger scale than most packages. pythonPackages offers it, why not nixpkgs?

Should those overlaps in namespaces be considered an issue and fixed? They lead to pretty subtle bugs...

They're annoying, but package names eventually will overlap. That's one reason why we use namespaces.

Are there reasons for this? Wouldn't it be nicer to treat python pkgs and "regular" pkgs as uniformly as possible?

Python libraries belong in the Python package set. That way we're also preventing possible naming collisions.

Also if fix -> fix' isn't acceptable, can't we use the same mechanism overlays use to make it possible to do deep overrides from within nixpkgs? Its not a unique sage problem, sage just needs it at a bigger scale than most packages. pythonPackages offers it, why not nixpkgs?

We are using the same mechanism: the package set we've defined here is exactly the same. The issue is that we need to re-evaluate the main package set. In order to do that, we need to access it, and there's two ways: the method you originally proposed, and attaching it to the evaluated package set as __unfix__ with fix'.

Are there any updates on this pull request, please?

Still blocked on something like #44196.

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

I marked this as stale due to inactivity. → More info