python-2.7.14: Fix #27177 and #25750 #39555

timokau · 2018-04-26T14:14:16Z

Motivation for this change

I originally wanted to include this in the sage PR, but since a world-rebuilding python PR (#39517 by @dtzWill) was just merged into staging maybe its better to only rebuild all python2.7 packages on linux once.

Python Issue 27177: Regex match.group doesn't support index, only standard integers. This was fixed, but not backported to 2.7. sage needs that because it uses its own integers by default. I tried to monkey-patch around this, but apparently you can't monkey-patch re.

Python Issue 25750 leads to occasional crashes, especially on python 2.7. There is an upstream PR in the making.

As requested previously, there is now a branch with the sage package. This is not yet PR-ready though, contains unneeded patches, duplication, TODOs etc. I will rewrite history on that branch.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

dtzWill · 2018-04-26T14:27:04Z

pkgs/development/interpreters/python/cpython/2.7/default.nix

@@ -60,6 +60,24 @@ let

      ./properly-detect-curses.patch

+      # Fix python bug #27177 (https://bugs.python.org/issue27177)
+      # The issue is that `match.group` only recognizes python integers
+      # instead of everything taht has `__index__`.


nit: "taht" -> "that"

dtzWill · 2018-04-26T14:31:48Z

pkgs/development/interpreters/python/cpython/2.7/default.nix

+      # Fix python bug #27177 (https://bugs.python.org/issue27177)
+      # The issue is that `match.group` only recognizes python integers
+      # instead of everything taht has `__index__`.
+      # This bug was fixed upstream, but not backported to 2.7


Do any of the other python versions (3.4, 3.5, ..) in-tree need this?

Yes, everything <= 3.6 should have this issue. I only applied the patch for 2.7 because I only need it for 2.7 (for sage) and I wanted to minimize rebuilds. Since no 3.4/3.5 software seems to need it, I don't think its strictly necessary. Should I add it to 3.4 and 3.5 anyways?

dtzWill · 2018-04-26T14:35:16Z

pkgs/development/interpreters/python/cpython/2.7/default.nix

+
+      # "`type_getattro()` calls `tp_descr_get(self, obj, type)` without actually owning a reference to "self".
+      # In very rare cases, this can cause a segmentation fault if "self" is deleted by the descriptor."
+      # https://github.com/python/cpython/pull/6118


Similarly, reading that PR suggests this should be applied to all python versions (although it does say backporting to 2.7 is the most important since it's most problematic there).

Thats true, I also only applied this to 2.7 to minimize rebuilds. But since this is a crashing bug, here it might really be a good idea to add it to all versions. Especially since the other PR triggers rebuilds across all versions anyways -- but that only matters if this gets merged before those builds run (which is a bit optimistic).

Minimizing mass rebuilds isn't really needed, I believe. On staging we have a large rebuild almost every week, and python2 by itself is probably the worst part.

dtzWill · 2018-04-26T15:34:12Z

I'm not sure, sorry. Hopefully our python maintainer or someone else can lend us some insight or make the judgement call.

FRidh · 2018-04-26T16:35:15Z

I'm on holidays now and won't be able to review until Monday earliest.

timokau · 2018-04-26T17:12:31Z

Okay, there is absolutely no hurry. I just wanted to avoid a double rebuild, but rebuilds probably aren't that rare anyways.

Thanks for taking the time out of your holidays to let me know.

timokau · 2018-05-06T13:09:37Z

@FRidh could you have a look at this? Or do you prefer it to be included in #39981?

Merging this separately would make it easier to review and test #39981.

FRidh · 2018-05-06T14:19:31Z

Sorry for the wait. This looks good to me. It's a pity the OP of the first one did not push that one further.

@vcunat can you take this into account in the 2.7.15 bump. I've been without broadband for some days now and it looks like it will be for a bit longer.

vcunat · 2018-05-06T15:12:50Z

This apparently doesn't conflict with #40005 in any way (except trivial conflict in our patch list). Neither issue from this PR seems to have a fix upstreamed to 2.7 yet.

With 2.7.15, I'm not sure about the risks, but perhaps it's safe enough to just merge to staging directly.

timokau · 2018-05-06T20:06:34Z

It's a pity the OP of the first one did not push that one further.

Yeah the "well technically not a bug" answer is not very helpful. python2 apparently really is on the last days of life support.

This apparently doesn't conflict with #40005 in any way (except trivial conflict in our patch list).

I guess @FRidh only meant to merge both PRs into staging at the same time.

vcunat · 2018-05-07T10:09:05Z

Yes, I think so. The real problem in staging ATM is that it already contains several thousand build regressions for weeks, so someone will have to resolve them...

jdemeyer · 2018-05-09T12:10:40Z

It's a pity the OP of the first one did not push that one further.

It was rejected pretty explicitly. No point in fighting a lost battle.

jdemeyer · 2018-05-09T12:13:15Z

For the record: we have been using both patches for quite some time in SageMath without any problems. The first one just adds some functionality which didn't exist before and the second one fixes an obvious bug but doesn't change any functionality.

27177 was merged but not backported to 2.7. There is currently an open PR for 25750.

timokau · 2018-05-28T12:01:34Z

Looks like #40005 is in master now (and can be closed). I have rebased. Can we merge this?

GrahamcOfBorg · 2018-05-28T12:16:56Z

Success on x86_64-linux (full log)

Attempted: python

Partial log (click to expand)

shrinking /nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15/lib/python2.7/lib-dynload/_locale.so
gzipping man pages under /nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15/share/man/
strip is /nix/store/92d2ifxcni4n3zx9s8wnkcjlvnx5ajlc-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15/lib  /nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15/bin
patching script interpreter paths in /nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15
/nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15/lib/python2.7/config/makesetup: interpreter directive changed from " /bin/sh" to "/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/sh"
/nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15/lib/python2.7/config/install-sh: interpreter directive changed from "/bin/sh" to "/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/sh"
/nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15/lib/python2.7/ctypes/macholib/fetch_macholib: interpreter directive changed from "/bin/sh" to "/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/k87hjw6klpxg957mllvh7zvr14hw2v1w-python-2.7.15...
wrong ELF type

GrahamcOfBorg · 2018-05-28T12:27:13Z

Success on aarch64-linux (full log)

Attempted: python

Partial log (click to expand)

gzipping man pages under /nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15/share/man/
strip is /nix/store/8yfik687kfccisxnad42j19lfb7ij9b4-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15/lib  /nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15/bin
patching script interpreter paths in /nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15
/nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15/lib/python2.7/config/install-sh: interpreter directive changed from "/bin/sh" to "/nix/store/vnb8q2h7951gd551nm2vq2g6n8296g5b-bash-4.4-p19/bin/sh"
/nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15/lib/python2.7/config/makesetup: interpreter directive changed from " /bin/sh" to "/nix/store/vnb8q2h7951gd551nm2vq2g6n8296g5b-bash-4.4-p19/bin/sh"
/nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15/lib/python2.7/ctypes/macholib/fetch_macholib: interpreter directive changed from "/bin/sh" to "/nix/store/vnb8q2h7951gd551nm2vq2g6n8296g5b-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15...
wrong ELF type
/nix/store/3zc1jxxz3xg93bvsp0gapm9pib9s054l-python-2.7.15

FRidh · 2018-05-28T12:27:30Z

@timokau I've added it to #41177.

GrahamcOfBorg · 2018-05-28T12:48:12Z

Failure on x86_64-darwin (full log)

Attempted: python

Partial log (click to expand)

cannot build derivation '/nix/store/x6mzbx1ay2lzi5yjpbn3ymw862711g6f-re_match_index.patch.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/66hngykd2k9pk0wkwvnhm97ys524v1gc-type_getattro.patch.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/66yg6w5crh04v6s2g3859bzwiccsvbba-flex-2.6.4.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/gmrwvzv1bw7012f6zrycvd58gmxradzh-python-2.7.15.drv': 16 dependencies couldn't be built
cannot build derivation '/nix/store/6j60v5b8cs7g33bsa1s72qhkahdxybdj-bootstrap_cmds-dev-tools-7.0.drv': 4 dependencies couldn't be built
cannot build derivation '/nix/store/jd9xb7lwas4xkyws8m13ynn9lz821rqg-xnu-osx-10.11.6.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/n5c1sm0a3lg9sr9zmg6djl93xbndlpcy-IOKit-osx-10.11.6.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/f70a8fnffw2gkypa089rihw85ni5al55-configd-osx-10.8.5.drv': 9 dependencies couldn't be built
cannot build derivation '/nix/store/a4frgav02bkqngk23psf4kk0drff6a7i-python-2.7.15.drv': 17 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/a4frgav02bkqngk23psf4kk0drff6a7i-python-2.7.15.drv' failed

timokau · 2018-05-28T13:40:52Z

Thanks! (ofBorg failure is a timeout)

timokau requested a review from FRidh as a code owner April 26, 2018 14:14

timokau changed the base branch from master to staging April 26, 2018 14:14

timokau force-pushed the python-patches branch 2 times, most recently from cea8788 to dad5c4a Compare April 26, 2018 14:17

GrahamcOfBorg added 6.topic: python 10.rebuild-darwin-stdenv 10.rebuild-darwin: 501+ 10.rebuild-linux: 501+ labels Apr 26, 2018

dtzWill reviewed Apr 26, 2018

View reviewed changes

timokau force-pushed the python-patches branch from dad5c4a to 66f866a Compare April 26, 2018 14:44

FRidh self-assigned this Apr 26, 2018

timokau mentioned this pull request May 4, 2018

Sage on nixos #39981

Merged

8 tasks

timokau mentioned this pull request May 12, 2018

python: 2.7.14 -> 2.7.15 (bugfix + security) #40005

Closed

8 tasks

python: Fix upstream bugs NixOS#27177 and NixOS#25750

b645950

27177 was merged but not backported to 2.7. There is currently an open PR for 25750.

timokau force-pushed the python-patches branch from 66f866a to b645950 Compare May 28, 2018 12:01

FRidh mentioned this pull request May 28, 2018

Python: interpreter changes and package updates #41177

Closed

11 tasks

timokau closed this May 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python-2.7.14: Fix #27177 and #25750 #39555

python-2.7.14: Fix #27177 and #25750 #39555

timokau commented Apr 26, 2018

dtzWill Apr 26, 2018

dtzWill Apr 26, 2018

timokau Apr 26, 2018

dtzWill Apr 26, 2018

timokau Apr 26, 2018

vcunat May 6, 2018

dtzWill commented Apr 26, 2018

FRidh commented Apr 26, 2018

timokau commented Apr 26, 2018

timokau commented May 6, 2018

FRidh commented May 6, 2018

vcunat commented May 6, 2018

timokau commented May 6, 2018

vcunat commented May 7, 2018

jdemeyer commented May 9, 2018

jdemeyer commented May 9, 2018

timokau commented May 28, 2018

GrahamcOfBorg commented May 28, 2018

GrahamcOfBorg commented May 28, 2018

FRidh commented May 28, 2018

GrahamcOfBorg commented May 28, 2018

timokau commented May 28, 2018

python-2.7.14: Fix #27177 and #25750 #39555

python-2.7.14: Fix #27177 and #25750 #39555

Conversation

timokau commented Apr 26, 2018

Motivation for this change

Things done

dtzWill Apr 26, 2018

Choose a reason for hiding this comment

dtzWill Apr 26, 2018

Choose a reason for hiding this comment

timokau Apr 26, 2018

Choose a reason for hiding this comment

dtzWill Apr 26, 2018

Choose a reason for hiding this comment

timokau Apr 26, 2018

Choose a reason for hiding this comment

vcunat May 6, 2018

Choose a reason for hiding this comment

dtzWill commented Apr 26, 2018

FRidh commented Apr 26, 2018

timokau commented Apr 26, 2018

timokau commented May 6, 2018

FRidh commented May 6, 2018

vcunat commented May 6, 2018

timokau commented May 6, 2018

vcunat commented May 7, 2018

jdemeyer commented May 9, 2018

jdemeyer commented May 9, 2018

timokau commented May 28, 2018

GrahamcOfBorg commented May 28, 2018

GrahamcOfBorg commented May 28, 2018

FRidh commented May 28, 2018

GrahamcOfBorg commented May 28, 2018

timokau commented May 28, 2018