Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python35, python36: fix reading large files on darwin #39627

Merged
merged 1 commit into from May 1, 2018
Merged

python35, python36: fix reading large files on darwin #39627

merged 1 commit into from May 1, 2018

Conversation

veprbl
Copy link
Member

@veprbl veprbl commented Apr 27, 2018

Motivation for this change

python3 can't read(and write?) big files on darwin:

# dd if=/dev/zero of=/tmp/test bs=1048576 count=1024 && python3 -c "open('/tmp/test', 'rb').read()"
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.00833 s, 1.1 GB/s

# dd if=/dev/zero of=/tmp/test bs=1048576 count=2048 && python3 -c "open('/tmp/test', 'rb').read()"
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.03879 s, 1.1 GB/s
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OSError: [Errno 22] Invalid argument

The bug reproduces on High Sierra and Yosemite. This is also known as macos fread bug # 6434977.

The upstream PR python/cpython#1705 is stalled on fixing tests for Travis. There is no patch for python 3.4, so I don't provide one.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@veprbl veprbl requested a review from FRidh as a code owner April 27, 2018 21:48
@GrahamcOfBorg GrahamcOfBorg added the 6.topic: darwin Running or building packages on Darwin label Apr 27, 2018
@veprbl
Copy link
Member Author

veprbl commented Apr 27, 2018 •
edited

And python2 is not affected. I checked with dtruss and it doesn't seem to call the read syscall.

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: python35, python36

Partial log (click to expand)

gzipping man pages under /nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5/share/man/
strip is /nix/store/fzj1a9khx6wwp3zlf8lkf6kv17x1g5gb-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5/lib  /nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5/bin
patching script interpreter paths in /nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5
/nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5/bin/python3.6m-config: interpreter directive changed from "/bin/sh" to "/nix/store/p98ljrcxnx6kskkb3z5nfraqyayb01rx-bash-4.4-p19/bin/sh"
/nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5/lib/python3.6/ctypes/macholib/fetch_macholib: interpreter directive changed from "/bin/sh" to "/nix/store/p98ljrcxnx6kskkb3z5nfraqyayb01rx-bash-4.4-p19/bin/sh"
/nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5/lib/python3.6/config-3.6m-x86_64-linux-gnu/makesetup: interpreter directive changed from " /bin/sh" to "/nix/store/p98ljrcxnx6kskkb3z5nfraqyayb01rx-bash-4.4-p19/bin/sh"
/nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5/lib/python3.6/config-3.6m-x86_64-linux-gnu/install-sh: interpreter directive changed from "/bin/sh" to "/nix/store/p98ljrcxnx6kskkb3z5nfraqyayb01rx-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/1c6cc0s6x6w858y4xr87m9awy2hyf9qd-python3-3.6.5...
wrong ELF type

@GrahamcOfBorg
Copy link

Failure on x86_64-darwin (full log)

Attempted: python35, python36

Partial log (click to expand)

cannot build derivation '/nix/store/k2chqf2s2546yp60hdgzk9yvrqi66ws3-flex-2.6.4.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/fsyb5xzhii16xc6fpm9b4n98jirav25k-CF-osx-10.10.5.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/24c9p4fmag96cynsg3vnhmw85vfazmgj-bootstrap_cmds-dev-tools-7.0.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/rl6xhk56ycy32zl1h3firwgx1miny1a7-python-2.7.14.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/9fv17nf84liz87fbkwn9bzv2yc3g1f8l-xnu-osx-10.11.6.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/cm8v97r2yw1an98kaccb6z0i3kxv3ahv-IOKit-osx-10.11.6.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/3gn4xz8pyvhrmwancmz2b4327rcyiq6x-configd-osx-10.8.5.drv': 8 dependencies couldn't be built
cannot build derivation '/nix/store/ffn095mzpdls04pmzcxps8q3bv7j73a2-python3-3.5.5.drv': 10 dependencies couldn't be built
cannot build derivation '/nix/store/s39nsiph63l9mv6v6qkqrlgxprdl8nw2-python3-3.6.5.drv': 10 dependencies couldn't be built
error: build of '/nix/store/ffn095mzpdls04pmzcxps8q3bv7j73a2-python3-3.5.5.drv', '/nix/store/s39nsiph63l9mv6v6qkqrlgxprdl8nw2-python3-3.6.5.drv' failed

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: python35, python36

Partial log (click to expand)

stripping (with command strip and flags -S) in /nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5/lib  /nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5/bin
patching script interpreter paths in /nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5
/nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5/lib/python3.6/config-3.6m-aarch64-linux-gnu/install-sh: interpreter directive changed from "/bin/sh" to "/nix/store/91hqfj1slg9rdr0r6aka5g54yi1wjkv3-bash-4.4-p19/bin/sh"
/nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5/lib/python3.6/config-3.6m-aarch64-linux-gnu/makesetup: interpreter directive changed from " /bin/sh" to "/nix/store/91hqfj1slg9rdr0r6aka5g54yi1wjkv3-bash-4.4-p19/bin/sh"
/nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5/lib/python3.6/ctypes/macholib/fetch_macholib: interpreter directive changed from "/bin/sh" to "/nix/store/91hqfj1slg9rdr0r6aka5g54yi1wjkv3-bash-4.4-p19/bin/sh"
/nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5/bin/python3.6m-config: interpreter directive changed from "/bin/sh" to "/nix/store/91hqfj1slg9rdr0r6aka5g54yi1wjkv3-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5...
wrong ELF type
/nix/store/5nlglw11mszvhpcjddpsg7kglv4k59iw-python3-3.5.5
/nix/store/2xqrm2aaclc4539szn7i9ncz9jkhx2lf-python3-3.6.5

@FRidh FRidh self-assigned this Apr 28, 2018
Mic92
Mic92 reviewed Apr 28, 2018
View reviewed changes
pkgs/development/interpreters/python/cpython/3.5/default.nix
] ++ optionals stdenv.isDarwin [
# Fix for https://bugs.python.org/issue24658
(fetchpatch {
url = "https://bugs.python.org/file45178/issue24658-3-3.6.diff";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this url stable? I mean, is it likely that the author will update it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand, “file45178” bit is unique id and file cannot be removed or modified. So yes.

@matthewbauer matthewbauer merged commit ca4fe58 into NixOS:staging May 1, 2018
@vcunat
Copy link
Member

vcunat commented Jul 3, 2018 •
edited

After minor update 78e0521 the patch no longer applies (staging). /cc @FRidh.

@vcunat
Copy link
Member

vcunat commented Jul 3, 2018 •
edited

Ah, I'm sorry, it's a different darwin-specific patch that doesn't apply :-/ #39576

@veprbl veprbl deleted the pr/python_issue24658 branch December 1, 2020 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Mic92 Mic92 Mic92 left review comments

@matthewbauer matthewbauer matthewbauer approved these changes

@FRidh FRidh Awaiting requested review from FRidh

Assignees

@FRidh FRidh

Labels
6.topic: darwin Running or building packages on Darwin 6.topic: python 10.rebuild-darwin: 501+ 10.rebuild-linux: 0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@veprbl @GrahamcOfBorg @vcunat @matthewbauer @Mic92 @FRidh