Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python: auto patch wheels with manylinux RPATH #78441

Closed
wants to merge 1 commit into from

Conversation

ljli
Copy link
Contributor

@ljli ljli commented Jan 24, 2020

I'm using this for some while now and it works well for me. For now please consider it as RFC because I'm not sure if there is interst in merging it at all and there might be better ways to integrate it into the existing python infrastructure.

What this does is to:

  • Patch pip to
    • Scan for ELF files after installing a wheel package
    • Patch a path with the manylinux1 libraries into the RPATH
    • Patch the interpreter to a valid path if it is an executable
  • Bundle the patched pip with python
  • Expose the python as python38Nix

That means manylinux1 compatible wheels can be used on NixOS.
E.g. the following does work with new python package:

$ python -m venv venv
$ source venv/bin/activate
$ pip install --only-binary :all: pandas
$ python -c 'import pandas'

@jonringer
Copy link
Contributor

looks like the future of manylinux will just depend on glibc for compatibility, https://www.python.org/dev/peps/pep-0600/. There may be standardization in nix around how you express required linked libraries. But I think altering the RPATH for everything is somewhat undesirable because the closure size becomes very large

I will defer to @FRidh for this

@stale
Copy link

stale bot commented Jul 22, 2020

Hello, I'm a bot and I thank you in the name of the community for your contributions.

Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do:

If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list.

If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past.

If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments.

Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 22, 2020
@FRidh
Copy link
Member

FRidh commented Jul 23, 2020

It's a nice idea but I don't think it belongs in Nixpkgs, mostly because of limitations. E.g., while you may patch binaries, wheels that have other non-Python dependencies that are not part of the manylinux standard, will still fail. Also, building from source won't work either. Thus I think this is best kept in a separate project.

@FRidh FRidh closed this Jul 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants