Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_INSTALL_TMPDIR #91272

Merged
merged 1 commit into from Jul 2, 2020
Merged

python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_INSTALL_TMPDIR #91272

merged 1 commit into from Jul 2, 2020

Conversation

misuzu
Copy link
Contributor

@misuzu misuzu commented Jun 22, 2020 •
edited

Motivation for this change

Another approach to #81441 as #90208 caused too many regressions.

With this patch pip will use NIX_PIP_INSTALL_TMPDIR as temporary directory when defined, usual code path will be used otherwise.

I've built some packages to make sure it's working:

  • pkgs.python2Packages.flask
  • pkgs.python2Packages.invoke
  • pkgs.python3Packages.flask
  • pkgs.python3Packages.invoke
  • pkgs.python3Packages.aiohttp - not reproducible (most likely unrelated)
  • pkgs.gtk-doc
  • nixos.iso_minimal.x86_64-linux - not reproducible (obvious)
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@FRidh
Copy link
Member

FRidh commented Jun 22, 2020

This way pip will always listen to the TMPDIR var, also when not used within a Nix build, and also for other operations.

Maybe following is a better place https://github.com/pypa/pip/blob/642fb07337a084252ae43c6b596dc18e434b2807/src/pip/_internal/operations/install/wheel.py#L676. Something like _temp_dir_for_testing = _temp_dir_for_testing or _os.environ.get("NIX_PIP_TMPDIR").

@misuzu
Copy link
Contributor Author

misuzu commented Jun 22, 2020

Maybe following is a better place

Yeah, less code to maintain, i like it.
I'll check it now.

@misuzu misuzu changed the title python3Packages.pip: allow setting reproducible temporary directory via TMPDIR python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_TMPDIR Jun 22, 2020
@misuzu misuzu changed the title python3Packages.pip: allow setting reproducible temporary directory via TMPDIR python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_TMPDIR Jun 22, 2020
@FRidh
Copy link
Member

FRidh commented Jun 22, 2020

even more specific name NIX_PIP_INSTALL_TMPDIR. This one really should be just internal.

@misuzu misuzu changed the title python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_TMPDIR python3Packages.pip: allow setting reproducible temporary directory via NIX_PIP_INSTALL_TMPDIR Jun 22, 2020
@misuzu
Copy link
Contributor Author

misuzu commented Jun 22, 2020

It seems to work as expected too, both python2 and python3.

@misuzu misuzu requested a review from FRidh June 26, 2020 14:46
@FRidh FRidh self-assigned this Jun 27, 2020
@FRidh FRidh merged commit aedbade into NixOS:staging Jul 2, 2020
@misuzu misuzu deleted the pip-reproducible branch April 22, 2021 21:12
SomberNight added a commit to SomberNight/electrum that referenced this pull request Aug 6, 2022
@SomberNight
mac build: force using source dist for most of our python dependencies
0c2a885 
We compile from tar.gz, instead of using pre-built binary wheels from PyPI.
(or if the dep is pure-python, use tar.gz instead of "source-only" wheel)

-----
Some unorganised things below for future reference.

```
$ dsymutil -dump-debug-map dist1/hid.cpython-39-darwin.so
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hid.o unable to open object file: No such file or directory
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hidapi/mac/hid.o unable to open object file: No such file or directory
---
triple:          'x86_64-apple-darwin'
binary-path:     'dist1/hid.cpython-39-darwin.so'
...
```

```
$ nm -pa dist1/hid.cpython-39-darwin.so
```

- https://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable
- pypa/pip#6505
- pypa/pip#7808 (comment)
- NixOS/nixpkgs#91272
- cython/cython#1576
- https://github.com/cython/cython/blob/9d2ba1611b28999663ab71657f4938b0ba92fe07/Cython/Compiler/ModuleNode.py#L913
SomberNight added a commit to SomberNight/electrum that referenced this pull request Aug 18, 2022
@SomberNight
mac build: force using source dist for most of our python dependencies
ae87148 
We compile from tar.gz, instead of using pre-built binary wheels from PyPI.
(or if the dep is pure-python, use tar.gz instead of "source-only" wheel)

-----
Some unorganised things below for future reference.

```
$ dsymutil -dump-debug-map dist1/hid.cpython-39-darwin.so
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hid.o unable to open object file: No such file or directory
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hidapi/mac/hid.o unable to open object file: No such file or directory
---
triple:          'x86_64-apple-darwin'
binary-path:     'dist1/hid.cpython-39-darwin.so'
...
```

```
$ nm -pa dist1/hid.cpython-39-darwin.so
```

- https://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable
- pypa/pip#6505
- pypa/pip#7808 (comment)
- NixOS/nixpkgs#91272
- cython/cython#1576
- https://github.com/cython/cython/blob/9d2ba1611b28999663ab71657f4938b0ba92fe07/Cython/Compiler/ModuleNode.py#L913
SomberNight added a commit to SomberNight/electrum that referenced this pull request Aug 30, 2022
@SomberNight
mac build: force using source dist for most of our python dependencies
f84516f 
We compile from tar.gz, instead of using pre-built binary wheels from PyPI.
(or if the dep is pure-python, use tar.gz instead of "source-only" wheel)

-----
Some unorganised things below for future reference.

```
$ dsymutil -dump-debug-map dist1/hid.cpython-39-darwin.so
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hid.o unable to open object file: No such file or directory
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hidapi/mac/hid.o unable to open object file: No such file or directory
---
triple:          'x86_64-apple-darwin'
binary-path:     'dist1/hid.cpython-39-darwin.so'
...
```

```
$ nm -pa dist1/hid.cpython-39-darwin.so
```

- https://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable
- pypa/pip#6505
- pypa/pip#7808 (comment)
- NixOS/nixpkgs#91272
- cython/cython#1576
- https://github.com/cython/cython/blob/9d2ba1611b28999663ab71657f4938b0ba92fe07/Cython/Compiler/ModuleNode.py#L913
SomberNight added a commit to SomberNight/electrum that referenced this pull request Sep 26, 2022
@SomberNight
mac build: force using source dist for most of our python dependencies
0abecff 
We compile from tar.gz, instead of using pre-built binary wheels from PyPI.
(or if the dep is pure-python, use tar.gz instead of "source-only" wheel)

-----
Some unorganised things below for future reference.

```
$ dsymutil -dump-debug-map dist1/hid.cpython-39-darwin.so
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hid.o unable to open object file: No such file or directory
warning: (x86_64) /private/var/folders/1n/zc14m3td0rg4nt0ftklmm7z00000gn/T/pip-install-bm88zvc1/hidapi_cd307bc31ab34252b77d11d6d7212fc5/build/temp.macosx-10.9-x86_64-3.9/hidapi/mac/hid.o unable to open object file: No such file or directory
---
triple:          'x86_64-apple-darwin'
binary-path:     'dist1/hid.cpython-39-darwin.so'
...
```

```
$ nm -pa dist1/hid.cpython-39-darwin.so
```

- https://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable
- pypa/pip#6505
- pypa/pip#7808 (comment)
- NixOS/nixpkgs#91272
- cython/cython#1576
- https://github.com/cython/cython/blob/9d2ba1611b28999663ab71657f4938b0ba92fe07/Cython/Compiler/ModuleNode.py#L913
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jonringer jonringer Awaiting requested review from jonringer

@FRidh FRidh Awaiting requested review from FRidh FRidh is a code owner

Assignees

@FRidh FRidh

Labels
6.topic: python 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@misuzu @FRidh