sage: 8.2 -> 8.3 #43281

timokau · 2018-07-10T13:02:44Z

Motivation for this change

The first release candidate for 8.3 is out. It will probably still be some time until the proper release. Its easier to keep track on upstream changes by maintaining an unstable sage version. Since working in the open is always better and maybe somebody cares or wants to test/use it, I'm opening this WIP PR for it. If somebody thinks this shouldn't be a PR, feel free to close it.

Notes:

the openblas fix is cherry-picked from openblas: backport fix #43234
the ipython patch will hopefully be in the latest upstream release by the time sage 8.3 is released

Known issues:

transient test failure in hyperelliptic_finite_field.py https://groups.google.com/d/msg/sage-release/zSCoOwWazww/CtQqMXprBgAJ

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

7c6f434c · 2018-07-10T13:34:50Z

I think given the amount of complexity in upstream sage packaging it is a nice idea to have such a PR in the open.

PS. added some labels, found out that GitHub obviously cannot do a merge of independent label subset operations.

FRidh · 2018-07-10T13:49:57Z

pkgs/development/libraries/science/math/openblas/default.nix

@@ -117,7 +118,12 @@ stdenv.mkDerivation rec {
    ] ++ stdenv.lib.optional (stdenv.hostPlatform.libc == "musl") "NO_AFFINITY=1"
    ++ mapAttrsToList (var: val: var + "=" + val) config;

-  patches = []; # TODO: Remove on next mass-rebuild
+    patches = [


Why? Please add a comment.

Thats the openblas backport I mentioned in the PR description, #43234

FRidh · 2018-07-10T13:50:01Z

pkgs/development/python-modules/ipython/5.nix

@@ -38,6 +39,14 @@ buildPythonPackage rec {
    substituteInPlace setup.py --replace "'gnureadline'" " "
  '';

+  patches = [
+    (fetchpatch {


Why? Please add a comment.

Because sage applied that patch and relies on it. As I said in the PR description, I'm guessing/hoping there will be a new ipython release before the final sage 8.3. So that patch will likely never make it into nixpkgs master.

Also see https://groups.google.com/forum/#!topic/sage-packaging/rUYd5M5tPTM

timokau · 2018-07-10T14:04:55Z

I think given the amount of complexity in upstream sage packaging it is a nice idea to have such a PR in the open.

Yah that's the idea :)

PS. added some labels, found out that GitHub obviously cannot do a merge of independent label subset operations.

What do you mean by label subset operations? Race conditions with ofborg adding labels? (Not all that related, but I'm really looking forward to NixOS/ofborg#164.)

timokau · 2018-07-10T14:50:17Z

In case anybody is testing this and the tests fail, thats because of a transient test failure and not caused by the nix package: https://groups.google.com/d/msg/sage-release/zSCoOwWazww/CtQqMXprBgAJ

7c6f434c · 2018-07-10T15:06:47Z

What do you mean by label subset operations? Race conditions with ofborg adding labels?

Race condition between two disjoint additions!

(Not all that related, but I'm really looking forward to NixOS/ofborg#164 .)

I look forward to Microsoft running GitHub into ground… I mean, if we need ofborg for everything, maybe Github is a part of the problem, not part of a solution?

timokau · 2018-07-11T20:01:54Z

>What do you mean by label subset operations? Race conditions with ofborg adding labels? Race condition between two disjoint additions!

Well thats just stupid.

>(Not all that related, but I'm really looking forward to NixOS/ofborg#164 .) I look forward to Microsoft running GitHub into ground… I mean, if we need ofborg for everything, maybe Github is a part of the problem, not part of a solution?

I wouldn't be all that saddened. But at gitlab wouldn't solve this problem either. And since it seems like the move is not going to happen before the frog is boiled, we might as well make ourselves comfortable. I dislike a lot of things about the sage development model. But the one thing I miss in nixpkgs is the workflow where every ticket has a status that is one of (among others) "needs:work" and "needs:review".

7c6f434c · 2018-07-11T20:41:33Z

Well, Gitlab is better because you can have a NixOS test for ofborg-Gitlab integration! Of course, having a bot generate and upload summaries out of mailing list would probably work better than Github if only because scraping a mailing list enforces a proper model of operation.

I think most of PRs are need:review by default, the rest are either [WIP] or match some needs: tag. And we don't have a clear idea what to do with issues, that's true. And of course we cannot even try a lot of nice workflows because «oh and run the full testsuite» requires insane amount of buildpower…

timokau · 2018-07-11T20:54:42Z

Well, Gitlab is better because you can have a NixOS test for ofborg-Gitlab integration! Of course, having a bot generate and upload summaries out of mailing list would probably work better than Github if only because scraping a mailing list enforces a proper model of operation.

What do you mean by generating summaries out of mailinglists?

I think _most_ of PRs are need:review by default, the rest are either [WIP] or match some needs: tag. And we don't have a clear idea what to do with issues, that's true.

Well they all *start* with needs:review, but I think many are in the "needs:work" phase of the review. If we had consistent tagging reviewers could just look at the needs:review list when they have time and we could properly judge how bad our PR-problem is.

And of course we cannot even try a lot of nice workflows because «oh and run the full testsuite» requires insane amount of buildpower…

Not sure what you mean by that. The full ofBorg testsuite?

7c6f434c · 2018-07-11T21:05:53Z

What do you mean by generating summaries out of mailinglists?

Patches to mailing list sometimes get lost. A summary «we have this Git URL submitted as a pull request, these people approved, these are ofborg evaluation result» and a global overview of such summaries would probably become strictly more informative than Github after three months.

Well they all *start* with needs:review, but I think many are in the "needs:work" phase of the review. If we had consistent tagging reviewers could just look at the needs:review list when they have time and we could properly judge how bad our PR-problem is.

Dismissing changes-requested reviews when no longer applicable could also be an option… And WIP labels are used from time to time. I would guess that the PR-problem is worse than you think, though. Then again, there is a nontrivial number of stuck PRs by project members (which are almost always a completely different kind of problem). Well, if we used an actual issue tracker we would have separate trackers for separate things, but oh well.

Not sure what you mean by that. The full ofBorg testsuite?

No, checking that a minor package update doesn't break any reverse dependencies.

timokau · 2018-07-11T21:41:16Z

On 18-07-11 14:05, Michael Raskin wrote: >What do you mean by generating summaries out of mailinglists? Patches to mailing list sometimes get lost. A summary «we have this Git URL submitted as a pull request, these people approved, these are ofborg evaluation result» and a global overview of such summaries would probably become strictly more informative than Github after three months.

Ah. Not sure what I think of the mailing list approach. It does have its merits for sure.

>Well they all *start* with needs:review, but I think many are in the "needs:work" phase of the review. If we had consistent tagging reviewers could just look at the needs:review list when they have time and we could properly judge how bad our PR-problem is. Dismissing changes-requested reviews when no longer applicable could also be an option…

Yeah that has always bothered me.

And WIP labels are used from time to time.

But usually not in the normal back-and-forth between reviewer and contributor.

I would guess that the PR-problem is worse than you think, though. Then again, there is a nontrivial number of stuck PRs by project members (which are almost always a completely different kind of problem).

Yeah the non-trivial, infrastructure changing PRs are something different entirely.

Well, if we used an actual issue tracker we would have separate trackers for separate things, but oh well.

I prefer tagging over separate trackers, but thats basically the same thing. By the way could someone merge #43234? This is causing failures and considering that it'll have to go through staging I would like to merge that as quickly as possible. The patch is already upstream and it is a bug fix so that should be pretty uncontroversial.

timokau · 2018-07-11T21:54:23Z

No, checking that a minor package update doesn't break any reverse dependencies.

Ah right. But that shouldn't stop us from improving our current workflow.

By the way in relevant sage related news: There is work upstream to make it possible to run tests in limited time. That would make it possible to test most of sage in a fraction of the time. I was thinking about completely splitting the tests from the main package anyways (while still making them a dependency). So then there could be sage.tests (which would be a dependency of the sage package) and sage.quicktests which would run maybe 10 minutes and could be used for reverse-dependency checks.

7c6f434c · 2018-07-12T04:48:11Z

Good news for Sage tests, but Sage is still a non-negligible rebuild per se, and then it is not the only one.

As for workflow — some structural differences mean that many tradeoffs between using tooling and spending human attention become unaffodable to us, and handling Nixpkgs PRs en masse is a burn out risk when spending too much of human attention.

But of course, there are a lot of good ideas that do not require making extra decisions or ensuring that nobody forgets about a PR.

timokau · 2018-08-03T13:33:35Z

8.3 was released 🎉

I've rebased and squashed the intermediate rc updates. The final build is still running but in the meantime:

@bjornfor @jgeerds @FRidh: Unfortunately there was no new ipython release. Is it okay to backport that patch in 86f058c? It has been accepted upstream.

Edit: Tests pass.

FRidh · 2018-08-03T15:31:50Z

@timokau I think so. Note I don't maintain the Python 2 version of IPython.

Now possible because of the lcalc c++11 patch.

timokau · 2018-08-04T12:24:50Z

I've added a patch to mark a known "random" failure (always happens on my laptop, never on my desktop). This is ready for merge.

timokau requested a review from FRidh as a code owner July 10, 2018 13:02

GrahamcOfBorg added 6.topic: python 8.has: documentation labels Jul 10, 2018

GrahamcOfBorg added 10.rebuild-darwin: 101-500 10.rebuild-linux: 501+ labels Jul 10, 2018

FRidh reviewed Jul 10, 2018

View reviewed changes

GrahamcOfBorg added 2.status: merge conflict and removed 2.status: merge conflict labels Jul 15, 2018

GrahamcOfBorg added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 and removed 10.rebuild-darwin: 101-500 labels Jul 23, 2018

GrahamcOfBorg removed the 10.rebuild-linux: 501+ label Jul 23, 2018

GrahamcOfBorg added 2.status: merge conflict and removed 2.status: merge conflict labels Jul 30, 2018

python2Packages.ipython: apply upstream signature patch

86f058c

timokau force-pushed the sage-8.3 branch 2 times, most recently from 916d683 to daeb8f9 Compare August 3, 2018 13:27

timokau changed the title ~~[WIP] sage: 8.2 -> 8.3~~ sage: 8.2 -> 8.3 Aug 3, 2018

timokau added 6 commits August 4, 2018 14:22

sage: 8.2 -> 8.3

840b64f

singular: backport ntl error handling patch

2891733

sage: don't pin singular

e82c26a

sage: don't pin pari

fa34f6d

lcalc: apply c++11 patch

e550e5d

sage: don't pin ntl

834a24e

Now possible because of the lcalc c++11 patch.

timokau force-pushed the sage-8.3 branch from daeb8f9 to 834a24e Compare August 4, 2018 12:23

7c6f434c merged commit b79dbfe into NixOS:master Aug 4, 2018

timokau deleted the sage-8.3 branch August 4, 2018 13:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sage: 8.2 -> 8.3 #43281

sage: 8.2 -> 8.3 #43281

timokau commented Jul 10, 2018 •

edited

7c6f434c commented Jul 10, 2018 •

edited

FRidh Jul 10, 2018

timokau Jul 10, 2018

FRidh Jul 10, 2018

timokau Jul 10, 2018

timokau Jul 10, 2018

timokau commented Jul 10, 2018

timokau commented Jul 10, 2018

7c6f434c commented Jul 10, 2018 via email

timokau commented Jul 11, 2018 via email

7c6f434c commented Jul 11, 2018

timokau commented Jul 11, 2018 via email

7c6f434c commented Jul 11, 2018 via email

timokau commented Jul 11, 2018 via email

timokau commented Jul 11, 2018

7c6f434c commented Jul 12, 2018

timokau commented Aug 3, 2018 •

edited

FRidh commented Aug 3, 2018

timokau commented Aug 4, 2018

sage: 8.2 -> 8.3 #43281

sage: 8.2 -> 8.3 #43281

Conversation

timokau commented Jul 10, 2018 • edited

Motivation for this change

Things done

7c6f434c commented Jul 10, 2018 • edited

FRidh Jul 10, 2018

Choose a reason for hiding this comment

timokau Jul 10, 2018

Choose a reason for hiding this comment

FRidh Jul 10, 2018

Choose a reason for hiding this comment

timokau Jul 10, 2018

Choose a reason for hiding this comment

timokau Jul 10, 2018

Choose a reason for hiding this comment

timokau commented Jul 10, 2018

timokau commented Jul 10, 2018

7c6f434c commented Jul 10, 2018 via email

timokau commented Jul 11, 2018 via email

7c6f434c commented Jul 11, 2018

timokau commented Jul 11, 2018 via email

7c6f434c commented Jul 11, 2018 via email

timokau commented Jul 11, 2018 via email

timokau commented Jul 11, 2018

7c6f434c commented Jul 12, 2018

timokau commented Aug 3, 2018 • edited

FRidh commented Aug 3, 2018

timokau commented Aug 4, 2018

timokau commented Jul 10, 2018 •

edited

7c6f434c commented Jul 10, 2018 •

edited

timokau commented Aug 3, 2018 •

edited