Figured I'd send you another one @FRidh :)

Related issue #18484

The goal is to create a Python environment that has pip and use it to install manylinux1 wheels. Then when these packages are imported within that Python environment, they should find the libraries they need.

I would say building such wheels is out of scope, it's a different problem. (Not sure if anyone would want it.)

Yeah, maybe adding this to mk-python-derivation.nix isn't the right thing. What I'm hoping to find is a good way to extend python.withPackages. What I'd like to be able to express is

myPython = python.withPackagesAndManyLinuxSupport (ps: with ps; [ pip etc ])

but currently python.withPackages doesn't offer a way to add parameters.

BTW if there were a modified version like python.withPackagesArgs that accepts parameters, it would also be convenient to pass the permitUserSite arg from the previous PR that way :). Since python.withPackages is just a shallow wrapper around the builder it seems convenient to expose more of the builder's args through it.

Anyway, thanks for the review so far! I'll look into your suggestions tomorrow.

version like python.withPackagesArgs that accepts parameters,

that's python.buildEnv

Right exactly, see my edit above haha. Curious about the idea you mentioned to not add this to the builder.

Okay @FRidh , just responded to your review.

I also implemented an idea for how manylinux1 can be selectively disabled. According to PEP 513 you signal whether it's supported by creating a module called _manylinux as part of the Python environment. I moved the place where this happens from being baked into the Python builds (in cpython/2.7/default.nix etc.) and instead happens in the builder (specifically in wrapper.nix). Thus manylinux1 support is a property of the Python environment, which I think makes more sense anyway.

@FRidh just responded to your latest two comments. Now that 19.03 is released, any chance you have time to help me push this over the finish line? :)

See https://github.com/FRidh/nixpkgs/tree/manylinux1. I split it into two commits and removed the buildEnv part. The Python package might actually need to be moved back to a normal non-Python package because....well, it's not a Python package.

The question is now, how do you want to use it? If you open e.g. a nix-shell with Python packages and this metapackage with libraries, then when pip installing a wheel it won't find those libraries during runtime. More is needed for that.

Sorry for the delay, I had to regain context on this.

I think it was wrong to remove the makeWrapperArgs stuff in my latest response to your review. (I removed it here.) As a reminder, I had added this code to mk-python-derivation.nix:

makeWrapperArgs = makeWrapperArgs ++ (lib.optionals manylinux1 [	
  "--set"	
  "LD_LIBRARY_PATH"	
  (lib.makeLibraryPath [(lib.callPackage ./manylinux1.nix {}).package])	
]);

The idea is to set LD_LIBRARY_PATH in the resulting Python environment so that it contains the manylinux1 libraries, so that things work at runtime. The way I've been using this is like this:

myWithPackages = f: let packages = f pythonPackages; in
  python.buildEnv.override {
    extraLibs = packages;
    manylinux1 = true;
  };

myPythonEnvironment = myWithPackages (ps: [ps.pip])

So, in the resulting myPythonEnvironment, I can use pip to install manylinux1 wheels.

Thus, I think the missing piece is the change to mk-python-derivation.nix. Plus the change to selectively disable manylinux1 in Python environments. What do you think? I'll clean up this PR to show what I think the final version should be.

No, nothing needs to be added to buildEnv or buildPythonPackage. If you take my branch, and add the autopatchelfhook there as setupHook, then it should work fine.

Confused, I'm looking at https://github.com/FRidh/nixpkgs/tree/manylinux1 and not seeing any autoPatchelfHook in the 2 commits there. Could you explain further?

Friendly ping @FRidh , I still need some clarification on what you mean before I can proceed.

Confused, I'm looking at https://github.com/FRidh/nixpkgs/tree/manylinux1 and not seeing any autoPatchelfHook in the 2 commits there. Could you explain further?

@thomasjm I guess FRidh wants you to add something like setupHook = ''autoPatchelfHook''; into the [...]/manylinux1/default.nix file in FRidh's branch. Then use the branch for the merge request (?)

Anyway, using the branch with the modification I was able to build some derivations I was previously not able to. Thanks to both of you.

Glad to help @Quarub. Is your use-case building manylinux1-compatible wheels? I'm wondering if you have any opinions on how the final version of this should look--for example, if we add the setupHook you mentioned and land that branch, would that be enough for you to use it?

I wanted to install a library from Pypi and ran into X is not a supported wheel on this platform and ended up here (I only saw manylinux1 wheels available to download). I'm afraid @thomasjm I'm not familiar with building wheels.

No worries @Quarub -- I'm just trying to be clear what exactly you did to make things work so I can make sure this gets packaged up in a nice way for your use-case. Any details you can provide to spell out how you made it work for you would be helpful.

Sorry @thomasjm , I had to deal with some tasks.
Now to what I did:

1. Run git clone https://github.com/FRidh/nixpkgs.git -b manylinux1
2. Add setupHook = ''autoPatchelfHook''; to /to/where/you/cloned/nixpkgs/pkgs/development/python-modules/manylinux1/default.nix (what FRidh suggested to do).
3. Run nix-shell -I nixpkgs=/to/where/you/cloned/nixpkgs in the directory with the following files:
   pack.nix
nes-py.nix
gymmario.nix
opencv4.nix
shell.nix
   It will take some time due to the rebuild needed (as shown in the labels of this request).
4. In the nix-shell run the first example of the readme from the gym-super-mario-bros repo
5. See Mario jump around randomly in a small window.

Btw. if you spot something I could improve in the expressions, please tell me. For example I just figured that I could replace <nixpkgs> in pack.nix with /to/where/you/cloned/nixpkgs so I don't need to use the -I option.

Thanks for the details @Quarub . Maybe I'm missing something obvious but I'm still a little confused by how that worked for you, since none of the files you posted seem to actually mention manylinux1/default.nix. Or maybe I just really don't understand setup hooks :P. Any clarification would be appreciated.

@FRidh , I'm thinking about closing this PR since we haven't been able to come up with a story for how to package it up. For my own purposes I can simply keep a copy of the manylinux1 file in my own code and create a manylinux1-capable environment like this:

    python.buildEnv.override {
      extraLibs = [my packages];
      makeWrapperArgs = ["--set" "LD_LIBRARY_PATH" (makeLibraryPath manylinux1.libs)];
    };

The ability to make an environment like this is all I ever wanted. It allows me to pip install a manylinux1 wheel and then successfully open a Python repl and import it. I still don't fully understand your proposed setup hook/pseudo Python package solution--it doesn't seem like a complete solution to me. So please let me know if you're interested in keeping this PR around, if not I'll close.

this is stale and not looking like it's something we want to do at this time. closing.