python3Packages.click: fix 'locale' path #41843

flokli · 2018-06-11T14:49:07Z

importing click shells out to 'locale', which currently needs to be in
PATH. Fix by setting patching locale command at runtime.

Motivation for this change

Tried dockerizing a flask application, which uses click, which failed to import due to above error

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Mic92 · 2018-06-11T14:52:23Z

pkgs/development/python-modules/click/default.nix

+  patches = [
+    (substituteAll {
+      src = ./fix-paths.patch;
+      locale = "${glibc.bin}/bin/locale";


Should be only applied on linux.
btw. @dtzWill how does musl handle locales?

I think we could probably add this to unix-tools.nix. Right now I don't think Musl comes with a locale implementation but NetBSD has a portable version.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/locale.html

so if you can use "unixtools.locale" here that would be more portable. we will eventually add more implementations - right now it is just the glibc one.

Apparently there's this: https://gitlab.com/rilian-la-te/musl-locales which I've packaged here: https://github.com/dtzWill/nixpkgs/tree/feature/musl-locales

The tool maybe kinda works--IIRC there's some environment variable to set that tells musl where to look? Honestly not sure :).

How about os x in that case? Would the netbsd version work there?

Maybe... i think apple also has its own version

Is this part of darwin.system_cmds?

so if you can use "unixtools.locale" here that would be more portable. we will eventually add more implementations - right now it is just the glibc one.

@matthewbauer unixtools.locale doesn't seem to exist currently, or am I'm missing something obvious? ;-)

The tool maybe kinda works--IIRC there's some environment variable to set that tells musl where to look? Honestly not sure :).

@dtzWill looks like that might be MUSL_LOCPATH for musl-locales, similar to LOCALE_ARCHIVE for glibc. But I'm not sure if it's the same when running applications linked against musl…

Mic92 · 2018-06-11T14:53:10Z

@GrahamcOfBorg build python3Packages.click

GrahamcOfBorg · 2018-06-11T14:53:42Z

Success on aarch64-linux (full log)

Attempted: python3Packages.click

Partial log (click to expand)

Installing collected packages: click
Successfully installed click-6.7
/build/click-6.7
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/pr74rff8332fn1i2pzdrx9jirhh0v0ca-python3.6-click-6.7
strip is /nix/store/qg2agrqkf240s656d207zqhipl0bc2id-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/pr74rff8332fn1i2pzdrx9jirhh0v0ca-python3.6-click-6.7/lib
patching script interpreter paths in /nix/store/pr74rff8332fn1i2pzdrx9jirhh0v0ca-python3.6-click-6.7
checking for references to /build in /nix/store/pr74rff8332fn1i2pzdrx9jirhh0v0ca-python3.6-click-6.7...
/nix/store/pr74rff8332fn1i2pzdrx9jirhh0v0ca-python3.6-click-6.7

GrahamcOfBorg · 2018-06-11T14:53:47Z

No attempt on x86_64-darwin (full log)

The following builds were skipped because they don't evaluate on x86_64-darwin: python3Packages.click

Partial log (click to expand)

Cannot nix-instantiate `python3Packages.click' because:
error: while evaluating the attribute 'patches' of the derivation 'python3.6-click-6.7' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/stdenv/generic/make-derivation.nix:170:11:
while evaluating the attribute 'locale' of the derivation 'fix-paths.patch' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/stdenv/generic/make-derivation.nix:170:11:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:113:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:9193:11:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:117:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/development/libraries/glibc/default.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
assertion failed at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/development/libraries/glibc/default.nix:8:1

GrahamcOfBorg · 2018-06-11T14:54:03Z

Success on x86_64-linux (full log)

Attempted: python3Packages.click

Partial log (click to expand)

Installing collected packages: click
Successfully installed click-6.7
/build/click-6.7
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/fg0qf9s0s7qq7iap83qpbv1bapdw1xj2-python3.6-click-6.7
strip is /nix/store/21ymadblbmsbb2bk4q7gl4kjasp8zmgd-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/fg0qf9s0s7qq7iap83qpbv1bapdw1xj2-python3.6-click-6.7/lib
patching script interpreter paths in /nix/store/fg0qf9s0s7qq7iap83qpbv1bapdw1xj2-python3.6-click-6.7
checking for references to /build in /nix/store/fg0qf9s0s7qq7iap83qpbv1bapdw1xj2-python3.6-click-6.7...
/nix/store/fg0qf9s0s7qq7iap83qpbv1bapdw1xj2-python3.6-click-6.7

importing click shells out to 'locale', which currently needs to be in PATH. Fix by setting patching locale command at runtime.

flokli · 2018-06-11T17:11:06Z

Changed the drv to only patch on linux and if not musl.

matthewbauer · 2018-06-11T17:51:36Z

That's okay for now. Eventually we want to provide locale everywhere though.

matthewbauer · 2018-06-11T17:52:37Z

This might fix:

flokli · 2018-06-11T18:01:35Z

@matthewbauer nope, that's an error after having found locale, which was in both bug reports in $PATH

flokli · 2018-06-11T18:01:46Z

Anyways, thanks for the merge!

matthewbauer · 2018-06-11T18:21:26Z

But why doesn't click pick up the settings in the locale binary?

flokli · 2018-06-11T19:07:51Z

click only checks "for a good unicode environment" on python3, and if codecs.lookup(locale.getpreferredencoding()).name raises an Exception.

We now replace the locale binary that's being called by click. This doesn't change behaviour of codecs.lookup(locale.getpreferredencoding()).name, though - which seems to require LC_ALL/LANG to be set.

Moredread · 2018-06-19T16:56:46Z

It seems that this breaks building for platformio, as it overrides the package for an older version of click. It fails while applying the patch

copying path '/nix/store/0fbamfgfcharvqa1md6phn7z8plcqmq1-python2.7-cffi-1.11.5-dev' from 'https://cache.nixos.org'...
unpacking sources
unpacking source archive /nix/store/v76hnxqbw0wz6lz08mznnxc46s6rjk3i-click-5.1.tar.gz
source root is click-5.1
setting SOURCE_DATE_EPOCH to timestamp 1439810654 of file click-5.1/setup.cfg
patching sources
applying patch /nix/store/41fjj9kpfz3x96l66mlhabld6gvgqyl1-fix-paths.patch
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/click/_unicodefun.py     2018-06-11 15:08:59.369358278 +0200
|+++ b/click/_unicodefun.py     2018-06-11 15:09:09.342325998 +0200
--------------------------
File to patch:
Skip this patch? [y]
Skipping patch.
1 out of 1 hunk ignored
builder for '/nix/store/k92ind9vxl601yn1jnsg8w5wxpqyl58g-python2.7-click-5.1.drv' failed with exit code 1

Edit: See PR #42254 for a fix

flokli requested a review from FRidh as a code owner June 11, 2018 14:49

Mic92 reviewed Jun 11, 2018

View reviewed changes

GrahamcOfBorg added the 6.topic: python label Jun 11, 2018

python3Packages.click: fix 'locale' path

c358e1d

importing click shells out to 'locale', which currently needs to be in PATH. Fix by setting patching locale command at runtime.

flokli force-pushed the python-click-path branch from 2f0c61e to c358e1d Compare June 11, 2018 17:10

GrahamcOfBorg added 10.rebuild-darwin: 101-500 10.rebuild-linux: 101-500 labels Jun 11, 2018

matthewbauer merged commit 64cdcb5 into NixOS:master Jun 11, 2018

flokli deleted the python-click-path branch June 11, 2018 18:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3Packages.click: fix 'locale' path #41843

python3Packages.click: fix 'locale' path #41843

flokli commented Jun 11, 2018

Mic92 Jun 11, 2018

matthewbauer Jun 11, 2018

matthewbauer Jun 11, 2018

matthewbauer Jun 11, 2018

dtzWill Jun 11, 2018

Mic92 Jun 11, 2018

matthewbauer Jun 11, 2018

Mic92 Jun 11, 2018

flokli Jun 11, 2018

Mic92 commented Jun 11, 2018

GrahamcOfBorg commented Jun 11, 2018

GrahamcOfBorg commented Jun 11, 2018

GrahamcOfBorg commented Jun 11, 2018

flokli commented Jun 11, 2018

matthewbauer commented Jun 11, 2018

matthewbauer commented Jun 11, 2018

flokli commented Jun 11, 2018

flokli commented Jun 11, 2018

matthewbauer commented Jun 11, 2018

flokli commented Jun 11, 2018

Moredread commented Jun 19, 2018 •

edited

python3Packages.click: fix 'locale' path #41843

python3Packages.click: fix 'locale' path #41843

Conversation

flokli commented Jun 11, 2018

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Jun 11, 2018

GrahamcOfBorg commented Jun 11, 2018

GrahamcOfBorg commented Jun 11, 2018

GrahamcOfBorg commented Jun 11, 2018

flokli commented Jun 11, 2018

matthewbauer commented Jun 11, 2018

matthewbauer commented Jun 11, 2018

flokli commented Jun 11, 2018

flokli commented Jun 11, 2018

matthewbauer commented Jun 11, 2018

flokli commented Jun 11, 2018

Moredread commented Jun 19, 2018 • edited

Moredread commented Jun 19, 2018 •

edited