1. Impressive.

2. I think among the things to do before removing [WIP] is making sure you don't commit any clearly temporary files.

Thanks :)

Oh right the .orig files. I put the PR up in a bit of a hurry and missed those. Removed.

I'm not sure about the ofBorg error, it appears to have something to do with the node packages. I'm not sure about those anyways, I'm still hoping for some feedback from someone who knows more about node packaging.

@FRidh I am not sure if it is fully correct to call such packages unmaintained, as the Sage team is clearly capable and quite likely to perform maintenance on these packages if unavoidable (of course they can also drop the dependency, in which case we would find out the packages were indeed unmaintained along the way…)

On the other hand, maybe they are not worth putting in the global Python namespace, just as liborcus for LibreOffice lived together with it because nothing else was likely to ever use it.

@FRidh I've moved flask-oldsessions, flask-openid and python-openid under the sage scope.

Status update:

I have removed all unnecessary dependencies and made lots of minor improvements. I'm
currently polishing patches and upstreaming them when possible and appropriate.

I'll also have to figure out how makeScope and newScope work because I think I
should use those in default.nix.

Other than that all the dependencies except the python patches have been merged. Also
sage 8.2 final was released.

Can somebody (@7c6f434c, @FRidh?) help me figure out how to avoid all the inherit xyz in default.nix? I think I need some of newScope, makeScope, import pkgs.path { overlays = [ (self: super: {...})] } and python.pkgs.override. But whatever I try, it either eats all my RAM in endless recursion, doesn't recognize the overrides or doesn't allow me to define new packages.

Could someone give me a minimal example how to create a new scope that overrides one python package and one regular pacakge? Something like:

pkgs = nixpkgs.giveMeNewScope (self: super: {
    glpk = super.glpk.overrideDerivation (...); # regular package
    numpy = super.python2.pkgs.numpy.override {}; # python package
    fantasy-py = self.python2.callPackage ./fantasy-py.nix {}; # new python package
});

That defines a function right? But where do I call that function / give it as an argument?

I am not sure — reimporting with overlays sound dirty, packageOverrides are deprecated…

Yeah reimporting with overlays is what the manual suggested, but I think thats probably a different use-case. Do you know who to ask?

That would be @grahamc in doc/functions.xml and @nbp in doc/language-frameworks/python.section.md. Any help would be much appreciated.

Alright I think this is ready for review now. The only remaining blocker for merge is #39555. Once thats merged and through staging I will rebase this PR.

If I can clarify anything, let me know.

thanks that's an awesome contribution. I wanted to use sage at some point, had some problems, wanted to fix it but gave up when facing the huge task (plus sage developers don't seem too receptive about packaging concerns). I am no maintainer but I believe merging the new packages in one or several PRs would make things easier. I've seen that a maintainer didn't want merging one (old) python package but I would advocate that sage is worth it ?

thanks that's an awesome contribution.

You're welcome :)

I wanted to use sage at some point, had some problems, wanted to fix it but gave up when facing the huge task (plus sage developers don't seem too receptive about packaging concerns).

There is some work ongoing upstream, but its not as much of an priority as I would like, true. I have also suggested all my relevant changes upstream[1] and at least none of them have been flat out rejected yet. So hopefully it will get better in the future.

I am no maintainer but I believe merging the new packages in one or several PRs would make things easier. I've seen that a maintainer didn't want merging one (old) python package but I would advocate that sage is worth it ?

I have already split most new packages, package fixes and package updates in seperate PRs (lots of them). They've all been merged by now. The only things remaining are the pythong packages @FRidh doesn't want in the main tree, packages that don't make any sense without sage (like pybrial) and sage itself.

I think more or less the only thing blocking this PR is #39555, which is itself only blocked on the staging branch being a mess right now (not sure if thats still the case).

So once staging is cleaned up, #39555 can be merged into staging. Then we have to wait for staging to be merged into master and then this can be merged.

[1] https://trac.sagemath.org/query?status=closed&status=needs_info&status=needs_review&status=needs_work&status=new&status=positive_review&reporter=gh-timokau&order=priority

Finally all the dependencies are in master. I've cleaned up the history and rebased on nixpkgs-unstable for easier testing.

I think this is ready for test/merge.

@GrahamcOfBorg build sage.lib

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: sage.lib

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Failure on aarch64-linux (full log)

Attempted: sage.lib

make[1]: Leaving directory '/build/source'
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/v92icw9c0xqhml86n5xl1yj7rsdd4q9q-pynac-0.7.16
shrinking /nix/store/v92icw9c0xqhml86n5xl1yj7rsdd4q9q-pynac-0.7.16/lib/libpynac.so.17.1.0
strip is /nix/store/0pjsgkxz0rp5baycq5sp2s72lrr5q9sg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/v92icw9c0xqhml86n5xl1yj7rsdd4q9q-pynac-0.7.16/lib
patching script interpreter paths in /nix/store/v92icw9c0xqhml86n5xl1yj7rsdd4q9q-pynac-0.7.16
checking for references to /build in /nix/store/v92icw9c0xqhml86n5xl1yj7rsdd4q9q-pynac-0.7.16...
cannot build derivation '/nix/store/82rrk0bx5hjv91hbjsj9qfbyyrvjnv82-python2.7-sagelib-8.2.drv': 6 dependencies couldn't be built
error: build of '/nix/store/82rrk0bx5hjv91hbjsj9qfbyyrvjnv82-python2.7-sagelib-8.2.drv' failed

Failure on x86_64-linux (full log)

Attempted: sage.lib

100  1340  100  1340    0     0   1413      0 --:--:-- --:--:-- --:--:--  1413
building '/nix/store/c1614jmjiipk1aaswckmjwxhlsrchg3c-zero_division_error_formatting.patch.drv'...

trying https://git.sagemath.org/sage.git/patch/?h=f79070ddd09fa0ad6b340b097bd8d690a7aa35f0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1631    0  1631    0     0   3294      0 --:--:-- --:--:-- --:--:--  3294
cannot build derivation '/nix/store/vsvv2dadn6mbdf4vzy9h4p09f59zksma-sage-src-8.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ivpj81ax61c5gvp31xj3icidsrg9bzil-python2.7-sagelib-8.2.drv': 4 dependencies couldn't be built
error: build of '/nix/store/ivpj81ax61c5gvp31xj3icidsrg9bzil-python2.7-sagelib-8.2.drv' failed

Oh that must've been a mistake while fixing some merge conflict because I assumed that was my version of python-openid. I've rebased on latest master. Since python-openid for python2 is removed there, the eval should succeed now.

@timokau re: batch updates: when upstream of the language ecosystem has a package manager, it makes sense to track it via batch updates because the outcome is easier to reason about when comparing to the upstream situation.

Why is that? Python packages can depend on specific version (ranges) and if a update builds in nixpkgs and doesn't break any reverse-dependencies, I see no reason not to include it right then.

It's simply that just a small percentage of the python packages would be updated at all if we didn't have batch upgrades. And since the batch upgrades only happen on the unstable channel, people encountering broken builds should expect that this happens from time to time. Before a new release, we try to stabilize the python package set.
Btw, @FRidh proposed https://groups.google.com/forum/#!topic/nix-devel/koLvsCVjQcA, which would allow maintainers to opt out of batch upgrades.

Lets continue the python discussion here: https://discourse.nixos.org/t/continuation-discussion-about-python-infrastructure/441.

Back to topic: I've added another commit that un-pins two dependencies. I think that can remain its own commit, since it touches the arb packages as well and is a good example of the kind of maintenance work that will be needed to keep sage working.

Since this has been open and ready for review without major changes or objections for a while now, can we merge this? Maybe someone else can verify that sage builds for them before that. But I've rebuild from scratch after garbage-collecting, so I would be surprised if someone else got different results.

And a semi-related note: I think you should ask to get commit rights; while it might make sense to keep a Sage update in a PR while it is a WIP, and to discuss with @FRidh if you want a non-default version of a python package from a maintained branch, it is quite likely that reviewing your sage changes is useless because you have learnt about Sage much more than everyone else counted together. Also, I do think you have enough experience that your opinion about a PR «ah, it's just an uncontroversial update and it is ready to merge» is as good as anyone's.

Congrats ! thanks for the awesome work to all involved.

Thanks for merging! @teto you seem very interested in this package. Please don't hesitate to make improvements, fix bugs or ask questions if something is unclear! It would be nice to have a bus factor > 1.

I think you should ask to get commit rights

Sounds good :) What is the process for that?

@rbvermaa @domenkozar I think it is a good idea to give @timokau write access.

@timokau in the meantime, do you have 2FA enabled on GitHub?

Yes I have.

@rbvermaa @domenkozar was this ever discussed?