Do you have a specific case in Nixpkgs in mind where you want to use this?

Particularly in https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/systemd-boot/systemd-boot-builder.py and in similar cases like #48378.

Is that really worth it? Many people will have another python3 on their system anyway.

Is that really worth it? Many people will have another python3 on their system anyway.

Maybe not yet? I do however think it's worth it as a step in the right direction for smaller closures.

I also see the potential in other areas such as dockerTools which I personally use a lot in CI environments that are not always having a local cache, so any savings in dependency fetching is great.

In the past "crippled" versions were removed because of the confusion they could cause.

We explicitly call it python3Minimal so it should be obvious that it's not what you'd want to use by default.

As a side note: python3Minimal clocks in at 90483408 bytes according to nix path-info, perl clocks in at 84557752.
This makes Python a contender where it previously wasn't because of closure size concerns.

I really like having a minimal python for these and other noninteractive, script-style tasks.

Having it explicitly named python3Minimal avoids people accidentially using the "wrong python version", as it was the case before #19309.

We probably want to document the existence of it once some scripts start using it, but I'd be fine with merging this as it is.

Another great use case for this would distributing minimal docker images as well. I would love to use nix to package docker images as it's much more composable than docker files.

Would you use a python without tls support in a docker container? I would quickly run into problems when communicating with external services.

I suggest disabling pkgs/withPackages/buildEnv.

Another great use case for this would distributing minimal docker images as well. I would love to use nix to package docker images as it's much more composable than docker files.

Would you use a python without tls support in a docker container? I would quickly run into problems when communicating with external services.

It really depends on your usecase, whether that version of python is doing the actual ssl communication, or if it's just a helper script.

I suggest disabling pkgs/withPackages/buildEnv.

I don't see much benefit / closure size reduction from that. Let's say, you want to use all the "minimizing aspects" of the main python, but there's some language bindings only pulling in a library already part of your closure, and you want to make use of that.

I agree with @Mic92 that removing OpenSSL is too restrictive.

Building with musl might be another way to shave off some bits, but it's not really used anywhere yet.

According to https://github.com/NixOS/rfcs/blob/0d1cc2015543039a6ffc73d821457dd9210f56d1/rfcs/0046-platform-support-tiers.md#tier-2-3, Musl-based package are on Tier 2 support, so they are infinitely-small close to be built by Hydra (Tier 2-ε is on Hydra).

cc @7c6f434c : can we suppose that Tier 2 support packages are allowed to be built by Hydra?

I don't have opinions on Hydra resource mangement, I am just trying to codify the current state (and prepare the language for evolving the description)!

What is the likely use case where Nixpkgs is used, Python3 is used but glibc is not used at all? Does Python3 path itself get smaller just because of switching to musl (not because of a finer tuning of configuration)?

@7c6f434c well yeah, you are right - we can't strip glibc that easy. But current implementation will cause package rebuilds in python3Minimal.withPackages (p: with p; [ cryptography ]) - we can rebuild Musl packages as well here.

What is the likely use case where Nixpkgs is used, Python3 is used but glibc is not used at all?

Some docker/container stuff. Personally, I don't have a glibc-free usecase.

@flokli does it makes sense to patch .withPackages in such way, that ${python3} dep is replaced with ${python3Minimal} one? In that case we can use binary cache for py packages (instead of rebuilding for python3Minimal)

@danbst extension modules will then point to python3 instead of minimal, so you don't gain anything, in fact, it takes more size

@FRidh I was talking about something like pkgs.replaceDependency or "Shipping Security Updates"-style rewrites. It is complicated, but doable.

I don't see much benefit / closure size reduction from that. Let's say, you want to use all the "minimizing aspects" of the main python, but there's some language bindings only pulling in a library already part of your closure, and you want to make use of that.

There is no size benefit. It is to prevent having to deal with tickets where users start building packages using that Python.

Would you use a python without tls support in a docker container? I would quickly run into problems when communicating with external services.

No, I wouldn't. But these changes would make it a lot easier to override and add back tls/openssl, than it would to do all the closure minification myself.

@7c6f434c in that sense, python package isn't much different in glibc and musl variants. It's size is same, and image size for python+bash would be reduced only if both are musl-based.

OTOH, glibc is 30Mb, and musl is 6Mb. So we tradeoff +6Mb excess size for composite images and -30Mb size reduction for standalone python images.

But I agree now this idea is bad. I've tried to use pkgsMusl.python3.withPackages and it turns out, that something somewhere still brings regular pkgs.python3 into closure, which is another case for investigation, not related to this PR.

So we do agree exposing the arguments in a composable way as in that PR is something useful to have in nixpkgs?

In that case, any feedback to the code itself?

alright - this PR doesn't use python3Minimal so far, so let's merge it.

Not including ssl is too restrictive as a lot of applications need internet access nowadays, has there been any talk about adding it back?

Not including ssl is too restrictive as a lot of applications need internet access nowadays, has there been any talk about adding it back?

IIRC this is used to build glibc, which needs some python in its build system and other small usecases, where we want it to be as minimal as possible. Pulling in openssl into that wouldn't work.

If you want another python, you might want to use just plain python3, or compose your own, playing with these parameters:

Not including ssl is too restrictive as a lot of applications need internet access nowadays, has there been any talk about adding it back?

The original intent was to keep the derivation as small as possible, mainly for in sandboxing use cases.
How much does it increase the closure size?
If the impact isn't too big I think it's OK.

Scratch that, see #66762 (comment).

It's additional 7MB and openssl will be in pretty much any closure anyway.

We could disable openssl when building glibc but have python3Minimal with the ssl module. WDYT?

Overriding openssl isn't trivial:

mkPoetryApplication {
  projectDir = ./.;
  python = pkgs.python3Minimal.override { openssl = pkgs.openssl; };
}

Results into

error: builder for '/nix/store/3jlwh4li4kndxs1kq99c6v1h4khyn531-python3.10-cryptography-40.0.2.drv' failed with exit code 1;
       last 10 log lines:
       > Finished executing pythonRemoveTestsDir
       > pythonCatchConflictsPhase
       > Found duplicated packages in closure for dependency 'cffi':
       >   cffi 1.15.1 (/nix/store/8r1y609lqxmqxb1f7cfy14wmqgzi5ia5-python3.10-cffi-1.15.1/lib/python3.10/site-packages)
       >   cffi 1.15.1 (/nix/store/58kx9mp3nh47zy5567si148prx35cdqr-python3.10-cffi-1.15.1/lib/python3.10/site-packages)
       > Found duplicated packages in closure for dependency 'pycparser':
       >   pycparser 2.21 (/nix/store/6d6ljx4qsgr41fvk2ajiglci5678mrc8-python3.10-pycparser-2.21/lib/python3.10/site-packages)
       >   pycparser 2.21 (/nix/store/60hgn1k5bnsnrpzbjfcax81hb3iap7hq-python3.10-pycparser-2.21/lib/python3.10/site-packages)
       >
       > Package duplicates found in closure, see above. Usually this happens if two packages depend on different version of the same dependency.
       For full logs, run 'nix log /nix/store/3jlwh4li4kndxs1kq99c6v1h4khyn531-python3.10-cryptography-40.0.2.drv'.
error: 1 dependencies of derivation '/nix/store/9qcq1md9s0k3lz5nyig1b8mvyrahfkh9-python3.10-secretstorage-3.3.3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/0ll1pj4fzkr1brql8gs477p02br5cz7y-python3.10-keyring-23.13.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/lcl6478qwihklzq60irhpkqynn3xwry4-python3.10-poetry-1.3.2.drv' failed to build

See also #169475