python: Splice packages to better support cross #104201

Ericson2314 · 2020-11-18T19:16:25Z

Motivation for this change

Fixes #104135. Also should hopefully make it easier to fix these sorts of things later.

Things done

pkgs/development/interpreters/python/default.nix

It's ugly as hell, but I suppose it is needed to codify how to make spliced package sets.

FRidh

If we don't have the current legacy, how would the interpreter + package set could look then. Like Haskell? E.g., create the package set directly, instead of via the interpreter Would that solve any issues? It should make overriding easier I think, because you no longer need to bother with self.

The same fix as for PyPy should be done for Graal.

pkgs/development/interpreters/python/cpython/2.7/default.nix

Ericson2314 · 2020-11-19T15:45:48Z

@FRidh

If we don't have the current legacy, how would the interpreter + package set could look then. Like Haskell? E.g., create the package set directly, instead of via the interpreter.

Do you mean what bigger changes might we do in addition, like making pythonXY and pythonXYPackages separately rather than having the latter just be an alias of pythonXY,pkgs, just as haskell.compiler.* and haskell.packages.* are built separately?

Would that solve any issues? It should make overriding easier I think, because you no longer need to bother with self.

I'm not sure what you mean. I do think the .override { overrides = ... } thing is not as easy to use as the overriding the makeScopes alone allow, so perhaps we could get rid of that callPackage and change the docs accordingly. But I didn't want to get into these breaking changes on my own.

The same fix as for PyPy should be done for Graal.

Graal the JVM thing? Shouldn't we do that in a separate PR?

In other places of Nixpkgs this will also need to be done eventually. Maybe its better having a single attribute that consists of the 6 pkgs{theirHost}{theirTarget} sets? That could be constructed already in stage.nix.

Yes that sounds reasonable. But I think rather land this and then try a few other makeScope invocations and then decide what to abstract. Haskell, python, and xorg (the three examples today) are more different than I'd like, so I'd like to firmly establish the pattern first.

FRidh · 2020-11-19T16:22:42Z

Do you mean what bigger changes might we do in addition, like making pythonXY and pythonXYPackages separately rather than having the latter just be an alias of pythonXY,pkgs, just as haskell.compiler.* and haskell.packages.* are built separately?

Yes. The whole overriding is often considered confusing (it is) and different across package sets. If we're making changes that are (potentially) backwards-compatible, we should try to implement what we really want.

Graal the JVM thing? Shouldn't we do that in a separate PR?

It uses the same passthru function. See pkgs/interpreters/python/default.nix.

But I think rather land this and then try a few other makeScope invocations and then decide what to abstract.

Okay. See also for Qt #102168.

FRidh · 2020-11-19T16:25:04Z

$ nix-build -A pkgsCross.raspberryPi.python3.pkgs.numpy
error: --- EvalError ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix-build
at: (74:34) in file: /home/freddy/Code/libraries/nixpkgs_2/pkgs/development/interpreters/python/cpython/default.nix

    73|     pythonPackagesHostHost = pkgsHostHost.${pythonAttr};
    74|     pythonPackagesTargetTarget = pkgsTargetTarget.${pythonAttr};
      |                                  ^
    75|   };

attribute 'python38' missing
(use '--show-trace' to show detailed location information)

pkgs/development/interpreters/python/default.nix

This makes sense as these are tools we want to run at build time.

Now non-`buildInputs` that are python packages should be resolved correctly.

Ericson2314 · 2020-11-19T17:05:20Z

If we're making changes that are (potentially) backwards-compatible, we should try to implement what we really want.

Well I think this change is pretty much backwards compatible? If we were to change things everywhere to be consistent (which I do want to do) that would be a bunch more reviewers and things. OK with doing that as a follow-up?

nix-build -A pkgsCross.raspberryPi.python3.pkgs.numpy

Oh oops, I only did the fix for that with python 2. Sorry, fixed now.

super was incorrectly possible until #104201 got merged.

NixOS/nixpkgs#104201 broke this.

Follow-up to NixOS#104201, related to NixOS#105113.

super was incorrectly possible until NixOS#104201 got merged.

The package set is an attribute of the interpreter. The function to build an environment (`buildEnv`/`withPackages`) is part of the interpreter. The interpreter is passed to itself, and needs to be updated when overridden. For cross-compilation we splice the package set, and for that the various `build/host` interpreters and sets need to be available. We select these currently through `pkgs.${pythonAttr}`. The `pythonAttr` attribute was not fixed for `pythonFull`. NixOS/rfcs#83 NixOS#104201 We need a better solution for this because this is very brittle.

The package set is an attribute of the interpreter. The function to build an environment (`buildEnv`/`withPackages`) is part of the interpreter. The interpreter is passed to itself, and needs to be updated when overridden. For cross-compilation we splice the package set, and for that the various `build/host` interpreters and sets need to be available. We select these currently through `pkgs.${pythonAttr}`. The `pythonAttr` attribute was not fixed for `pythonFull`. NixOS/rfcs#83 #104201 We need a better solution for this because this is very brittle.

wrapPython for some reason has to be not spliced because 'substitutions.pythonHost = python;' causes ``` error: output '/nix/store/li07bad6s9brxx7cf8j408631nca39ka-python3.10-chardet-5.1.0-aarch64-unknown-linux-gnu' is not allowed to refer to the following paths: /nix/store/fdqpyj613dr0v1l1lrzqhzay7sk4xg87-python3-3.10.10 ``` ericson said in NixOS#104201 (comment) that the reason they were in keep was to not splice them but before the commit which broke hook splicing (33d12e5) everything appeared to be working correctly when cross-compiling we also somehow get a correctly spliced python in nativeBuildInputs for free

as far as i can see there should be no reason to not splice the hooks wrapPython for some reason has to be not spliced because 'substitutions.pythonHost = python;' causes ``` error: output '/nix/store/li07bad6s9brxx7cf8j408631nca39ka-python3.10-chardet-5.1.0-aarch64-unknown-linux-gnu' is not allowed to refer to the following paths: /nix/store/fdqpyj613dr0v1l1lrzqhzay7sk4xg87-python3-3.10.10 ``` ericson said in NixOS#104201 (comment) that the reason they were in keep was to not splice them but before the commit which broke hook splicing (33d12e5) everything appeared to be working correctly when cross-compiling we also somehow get a correctly spliced python in nativeBuildInputs for free

ofborg bot added the 6.topic: python label Nov 18, 2020

FRidh reviewed Nov 18, 2020

View reviewed changes

pkgs/development/interpreters/python/default.nix Outdated Show resolved Hide resolved

Ericson2314 force-pushed the splice-python branch from 8a3bb2d to 8b69381 Compare November 18, 2020 21:22

lib: Create makeScopeWithSplicing

a6218c0

It's ugly as hell, but I suppose it is needed to codify how to make spliced package sets.

Ericson2314 force-pushed the splice-python branch 2 times, most recently from 0b535e8 to 51f6805 Compare November 19, 2020 05:21

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Nov 19, 2020

FRidh reviewed Nov 19, 2020

View reviewed changes

pkgs/development/interpreters/python/cpython/2.7/default.nix Show resolved Hide resolved

FRidh reviewed Nov 19, 2020

View reviewed changes

pkgs/development/interpreters/python/cpython/2.7/default.nix Show resolved Hide resolved

Ericson2314 marked this pull request as ready for review November 19, 2020 16:09

Ericson2314 requested review from edolstra, infinisil, jonringer and nbp as code owners November 19, 2020 16:09

Ericson2314 changed the title ~~WIP: python: Splice packages to better support cross~~ python: Splice packages to better support cross Nov 19, 2020

FRidh reviewed Nov 19, 2020

View reviewed changes

pkgs/development/interpreters/python/default.nix Show resolved Hide resolved

Ericson2314 added 2 commits November 19, 2020 11:58

python: Use packages from previous stage in setup-hooks

330d468

This makes sense as these are tools we want to run at build time.

python: Use makeScopeWithSplicing

b57c5d4

Now non-`buildInputs` that are python packages should be resolved correctly.

Ericson2314 force-pushed the splice-python branch from 51f6805 to b57c5d4 Compare November 19, 2020 16:58

FRidh mentioned this pull request Nov 19, 2020

cpython: fix cross-compiling extension modules #98915

Merged

10 tasks

FRidh approved these changes Nov 19, 2020

View reviewed changes

FRidh mentioned this pull request Nov 19, 2020

Python: fix cross-compilation of extension modules #53320

Closed

Ericson2314 merged commit 93b430b into NixOS:master Nov 19, 2020

Ericson2314 deleted the splice-python branch November 19, 2020 19:07

FRidh added a commit that referenced this pull request Nov 22, 2020

python.tests: use self.callPackage instead of super.callPackage

7726f81

super was incorrectly possible until #104201 got merged.

adisbladis added a commit to nix-community/poetry2nix that referenced this pull request Nov 26, 2020

Use build packages in setup hook

98e0220

NixOS/nixpkgs#104201 broke this.

FRidh mentioned this pull request Nov 27, 2020

python: Propagate packageOverrides to pythonForBuild #105113

Merged

10 tasks

FRidh mentioned this pull request Nov 27, 2020

Python: rename internal parameters related to splicing and fix splicing #105155

Merged

10 tasks

FRidh added a commit to FRidh/nixpkgs that referenced this pull request Nov 28, 2020

Python: use pythonPackagesBuildHost instead of pythonForBuild

cce2fd5

Follow-up to NixOS#104201, related to NixOS#105113.

FRidh added a commit to FRidh/nixpkgs that referenced this pull request Nov 28, 2020

python.tests: use self.callPackage instead of super.callPackage

8220b04

super was incorrectly possible until NixOS#104201 got merged.

talw mentioned this pull request Dec 9, 2020

Fix GHC bootstrap in pkgsMusl and include patch for binutils/16177 #103183

Merged

10 tasks

FRidh mentioned this pull request Feb 27, 2021

python: fix full builds by referring to the correct interpreter (fixes #114475) #114538

Merged

10 tasks

FRidh mentioned this pull request Mar 13, 2021

nix-env can't list out paths for pkgsCross.* #116207

Closed

Artturin mentioned this pull request Apr 25, 2023

python: don't use keep for hooks, so they will be spliced #228139

Draft

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python: Splice packages to better support cross #104201

python: Splice packages to better support cross #104201

Ericson2314 commented Nov 18, 2020

FRidh left a comment •

edited

Ericson2314 commented Nov 19, 2020 •

edited

FRidh commented Nov 19, 2020 •

edited

FRidh commented Nov 19, 2020

Ericson2314 commented Nov 19, 2020

python: Splice packages to better support cross #104201

python: Splice packages to better support cross #104201

Conversation

Ericson2314 commented Nov 18, 2020

Motivation for this change

Things done

FRidh left a comment • edited

Choose a reason for hiding this comment

Ericson2314 commented Nov 19, 2020 • edited

FRidh commented Nov 19, 2020 • edited

FRidh commented Nov 19, 2020

Ericson2314 commented Nov 19, 2020

FRidh left a comment •

edited

Ericson2314 commented Nov 19, 2020 •

edited

FRidh commented Nov 19, 2020 •

edited