pythonPackages.Babel: Fix use of glibcLocales, skip musl tests #75676

nh2 · 2019-12-15T02:34:44Z

The test failure:

File "/build/Babel-2.7.0/babel/messages/pofile.py", line 325, in _invalid_pofile
  print(u"WARNING: Problem on line {0}: {1}".format(lineno + 1, line))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 84: ordinal not in range(128)

occurs in either of these cases:

glibc is used and glibcLocales is not a buildInput
musl is used and tests are run

python3Packages.Babel does not have this problem, on either glibc
or musl.

I also found that the preCheck export LC_ALL="en_US.UTF-8" is not
necessary in any case.

With this commit,

python27Packages.Babel
python3Packages.Babel
pkgsMusl.python27Packages.Babel
pkgsMusl.python3Packages.Babel

all build.

Motivation for this change

Things done

Notify maintainers

No maintainers listed.

FYI @dtzWill from #74904.

nh2 · 2019-12-15T02:35:38Z

@GrahamcOfBorg build python27Packages.Babel python3Packages.Babel pkgsMusl.python27Packages.Babel pkgsMusl.python3Packages.Babel

Can somebody please ofborg-build this on OSX for me?

nh2 · 2019-12-15T03:09:02Z

Looks like Babel triggers a mass rebuild, will change to staging.

…ixOS#74904. The test failure: File "/build/Babel-2.7.0/babel/messages/pofile.py", line 325, in _invalid_pofile print(u"WARNING: Problem on line {0}: {1}".format(lineno + 1, line)) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 84: ordinal not in range(128) occurs in either of these cases: * glibc is used and `glibcLocales` is not a `buildInput` * musl is used and tests are run `python3Packages.Babel` does not have this problem, on either glibc or musl. I also found that the `preCheck` `export LC_ALL="en_US.UTF-8"` is not necessary in any case. With this commit, python27Packages.Babel python3Packages.Babel pkgsMusl.python27Packages.Babel pkgsMusl.python3Packages.Babel all build.

nh2 · 2019-12-15T03:10:57Z

@GrahamcOfBorg build python27Packages.Babel python3Packages.Babel pkgsMusl.python27Packages.Babel pkgsMusl.python3Packages.Babel

nh2 · 2019-12-22T02:10:19Z

I can't figure it out

OK I could repro it at least on one of my machines now -.-

jonringer · 2019-12-22T02:27:21Z

pkgs/development/python-modules/Babel/default.nix

+  preCheck = if isPy3k then null else ''
+    export LC_ALL=C.UTF-8
+  '';


could also be:

preCheck = lib.optionalString isPy27 '' export LC_ALL=C.UTF-8 '';

I have no opinion on that; I wanted to express that "versions < 3" get this wrong, and isPy27 seems more specific than that. But at the same time, is the only < 3 version available in nixpkgs and no 2.8 is supposed to be released, so it may as well be equivalent.

Whichever you think is better!

I don't have a strong opinion either

nh2 · 2019-12-27T01:38:02Z

I suspect the unicode error difference occurs depending on the version of nix!

With nix 2.0.4 on NixOS 18.03 in the nixpkgs checkout of this PR, I can reproduce:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe0' in position 84: ordinal not in range(128)

But if I build with nix 2.3.1 by running, on the same machine:

NIX_PATH=nixpkgs=. nix-shell -p pkgs.nix --pure --run 'NIX_PATH=nixpkgs=. nix-build -A python27Packages.Babel'

then the build of the same .drv magically succeeds (bot nix versions produce the same .drv).

@edolstra Are you aware of any changes to nix that might induce this?

nh2 · 2019-12-27T01:51:08Z

The difference is likely somewhere between nix 2.2.2 and nix 2.3:

# fails: current 19.03
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/d1dff0bcd9f8cd5cf8fca1ab8f08d55dff5c9c57.tar.gz nix-shell -p pkgs.nix --pure --run 'nix-build --version'
nix-build (Nix) 2.2.2
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/d1dff0bcd9f8cd5cf8fca1ab8f08d55dff5c9c57.tar.gz nix-shell -p pkgs.nix --pure --run 'NIX_PATH=nixpkgs=. nix-build -A python27Packages.Babel'

# works: current 19.09
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/69ed29f5f41107aca5fb674218a4f75619c22207.tar.gz nix-shell -p pkgs.nix --pure --run 'nix-build --version'
nix-build (Nix) 2.3
NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/69ed29f5f41107aca5fb674218a4f75619c22207.tar.gz nix-shell -p pkgs.nix --pure --run 'NIX_PATH=nixpkgs=. nix-build -A python27Packages.Babel

nh2 · 2019-12-27T03:13:50Z

Preliminary bisection results yield:

# possible first bad commit: [4b214e6e4566a52fa073166d8164b79627eb5601] Merge pull request #2878 from NixOS/run-in-pts
# possible first bad commit: [3cc1125595d97b4ab7369e37e4ad22f4cfecb8b2] Another attempt at getting pseudoterminals to work on macOS
# possible first bad commit: [2743bf0bb1727056ce6d6d6be4e38c3251aac1f8] Hopefully fix macOS tests
# possible first bad commit: [82ca6ef390b752faf1bf27b22daa34b29a4b00d4] Set $TERM
# possible first bad commit: [e84c26564507388f2e8b7c7f2b00eb2b57856da1] Run builds in a pseudo-terminal

Full bisection log (initial skips generated with the technique shown in https://blog.smart.ly/2015/02/03/git-bisect-debugging-with-feature-branches/, using for rev in $(git rev-list 2.2.2..2.3 --merges --first-parent); do git rev-list $rev^2 --not $rev^; done | xargs git bisect skip):

# git bisect log
git bisect start
# good: [0baf7dc25e0a5fcd25dcd9d9cb737c682d01e4e7] Bump version
git bisect good 0baf7dc25e0a5fcd25dcd9d9cb737c682d01e4e7
# bad: [22d4ea7a989d26b86fc27706dfea0abd2fb52c52] Tweak release notes
git bisect bad 22d4ea7a989d26b86fc27706dfea0abd2fb52c52
# skip: [87c604c1f04ff31a68f2fa73fe5a2ed9f6ec7ac7] Fix launchd program args
git bisect skip 87c604c1f04ff31a68f2fa73fe5a2ed9f6ec7ac7
# skip: [800fba1037ad97ff5e6d9acf9e820a30f0922f59] Use wait4path on org.nixos.nix-daemon.plist
git bisect skip 800fba1037ad97ff5e6d9acf9e820a30f0922f59
# skip: [693e68e09c9a17ca72b074bd2e575bf435647b45] Set maximum name length in Nix
git bisect skip 693e68e09c9a17ca72b074bd2e575bf435647b45
# skip: [5bdac86be2eb6d541784c4f149bfa06b59ef63a2] Reword to clarify newer generations are left alone
git bisect skip 5bdac86be2eb6d541784c4f149bfa06b59ef63a2
# skip: [1dbaf119486fecbdf57ecc2f877307030b665f69] Fix nix-env documentation for --delete-generations
git bisect skip 1dbaf119486fecbdf57ecc2f877307030b665f69
# skip: [5011a52cf3f291c22d784aff6f5bbf8b99ae186a] Just enable hack on macOS
git bisect skip 5011a52cf3f291c22d784aff6f5bbf8b99ae186a
# skip: [4a3e96281d6b9ae3dbc20e638f389677df9a649e] Handle SIGWINCH in main thread
git bisect skip 4a3e96281d6b9ae3dbc20e638f389677df9a649e
# skip: [d854e7dfd6d512c8ed687b61d7aae3358eda71cd] install-multi-user: Detect and fail lack of systemd separately
git bisect skip d854e7dfd6d512c8ed687b61d7aae3358eda71cd
# skip: [0463d5e36ffdbb60f851bd3b34756eff84151b28] Allow empty /nix directory in multi-user installer
git bisect skip 0463d5e36ffdbb60f851bd3b34756eff84151b28
# skip: [177e5742fa732856f358cfb1b22b7575e94e9bb3] Merge pull request #3056 from grahamc/operators
git bisect skip 177e5742fa732856f358cfb1b22b7575e94e9bb3
# skip: [171d784404fefa7c91e2082f92ec92c39cc1aa8e] docs: operators: Make OR and AND capitalized
git bisect skip 171d784404fefa7c91e2082f92ec92c39cc1aa8e
# skip: [92ddce4f463dc3cf2327c832d45f62d985d5e09f] operators: document exact precedenc, split up similar operators
git bisect skip 92ddce4f463dc3cf2327c832d45f62d985d5e09f
# skip: [5c06a8d3283139140e765b5f10ad7102a6a3e964] Reset tmpDirInSandbox for unsandboxed
git bisect skip 5c06a8d3283139140e765b5f10ad7102a6a3e964
# skip: [9a0855bbb6546e792848e551e79f8efc40782eeb] Don’t rely on EPERM
git bisect skip 9a0855bbb6546e792848e551e79f8efc40782eeb
# skip: [11d853462925d0b57fe956962e07edf5751fd4c3] Use sandbox fallback when cloning fails in builder
git bisect skip 11d853462925d0b57fe956962e07edf5751fd4c3
# skip: [d171090530f4a2a79efec2c385bee1a10844c706] Disable CLONE_NEWUSER when it’s unavailable
git bisect skip d171090530f4a2a79efec2c385bee1a10844c706
# skip: [d45922472434783bdc29610479a735dd236229a3] nix-daemon.service: add install section.
git bisect skip d45922472434783bdc29610479a735dd236229a3
# skip: [057af1dbd802987df7990d965e08db185820a193] docs: document the installer's use of proxy env vars
git bisect skip 057af1dbd802987df7990d965e08db185820a193
# skip: [6dab42a551a769f304c2a0e66e8771d7289502f2] installer: handle network proxy in systemd install
git bisect skip 6dab42a551a769f304c2a0e66e8771d7289502f2
# skip: [b226b5cd976ca71abb3c0861b56d5e5940430924] nix-store: fix out of sync protocol
git bisect skip b226b5cd976ca71abb3c0861b56d5e5940430924
# skip: [91b00b145f0c50e346d0250168cbcbcba7aa3b40] libutil: add SizedSource
git bisect skip 91b00b145f0c50e346d0250168cbcbcba7aa3b40
# skip: [653c40778462b5921fbbe2abffd1a2822f08ec57] Expanded documentation for .nix-defexpr
git bisect skip 653c40778462b5921fbbe2abffd1a2822f08ec57
# skip: [ee9c988a1b2e3c511b8613e698a0f9632ab1538f] Track function start and ends for flame graphs
git bisect skip ee9c988a1b2e3c511b8613e698a0f9632ab1538f
# skip: [a02457db715a80de7630d823fd8d7230a4bd451f] conf: stalled-download-timeout: make tunable
git bisect skip a02457db715a80de7630d823fd8d7230a4bd451f
# skip: [c3fefd1a6efec457395a187d15f435447dee6f3b] pathlocks: add include to fcntl.h for O_CLOEXEC
git bisect skip c3fefd1a6efec457395a187d15f435447dee6f3b
# skip: [363a2f68261af73aefe4edead9c0526030751a27] post-build-hook: docs fixup
git bisect skip 363a2f68261af73aefe4edead9c0526030751a27
# skip: [7c5596734f65b30b250ea73a423b40a4ce513fdf] Add a post-build-hook
git bisect skip 7c5596734f65b30b250ea73a423b40a4ce513fdf
# skip: [cd933b22d2041b7efc348dcc09ff255967ffc663] Add pname and version to nix-env -q --json
git bisect skip cd933b22d2041b7efc348dcc09ff255967ffc663
# skip: [ee1e3132cab9d7de14837f2b04b115e544ae1622] Disable findRuntimeRoots on darwin when running tests because lsof is slow
git bisect skip ee1e3132cab9d7de14837f2b04b115e544ae1622
# skip: [89865144c3ba0162cd37bcbe49b3095e9bab4164] Allow builtins.pathExists to check the existence of /nix/store paths
git bisect skip 89865144c3ba0162cd37bcbe49b3095e9bab4164
# skip: [03addc3b0a79cb7a58ae5651d1915af7383a220c] Use $HOME instead of $USER
git bisect skip 03addc3b0a79cb7a58ae5651d1915af7383a220c
# skip: [c82a856b36bcd008edeb9c4c981b974ae642afea] Add default for USER when unset
git bisect skip c82a856b36bcd008edeb9c4c981b974ae642afea
# skip: [9031a6838ccf10b0db0d91a89e14d71a1adf6235] Remove .github/FUNDING.yml
git bisect skip 9031a6838ccf10b0db0d91a89e14d71a1adf6235
# skip: [cf6172f05e9446b2b85dd7fa2e73cb93f56620e2] docs: document balancing cores and max-jobs
git bisect skip cf6172f05e9446b2b85dd7fa2e73cb93f56620e2
# skip: [ec0087df0a2da5c68363b3f4509d4545deed97a0] Don’t use entire /etc/nsswitch.conf file
git bisect skip ec0087df0a2da5c68363b3f4509d4545deed97a0
# skip: [648bdf153d2e4d3c6687d8e1780b0dfd0aa98cda] tarball-ttl: document
git bisect skip 648bdf153d2e4d3c6687d8e1780b0dfd0aa98cda
# skip: [82b7f0e840983879a510245903ff7c917276f65d] autoconf: Implement release tarball detection. Fixes #257.
git bisect skip 82b7f0e840983879a510245903ff7c917276f65d
# skip: [cd8bc06e8786018ddb16cea4cb10971b63d0efd2] autoconf: Add comment on use of `false`.
git bisect skip cd8bc06e8786018ddb16cea4cb10971b63d0efd2
# skip: [c3db9e6f8fd06d691be04cdd95a6bb21a400481d] autoconf: Check if --nonet works. Fixes #967 #506.
git bisect skip c3db9e6f8fd06d691be04cdd95a6bb21a400481d
# skip: [a96006d97fc87c5073f9a39db841625bdb7c401c] Get BOOST_LDFLAGS from autoconf, fix Ubuntu 16.04 build.
git bisect skip a96006d97fc87c5073f9a39db841625bdb7c401c
# skip: [d203c554faa00fec55377d6640c8fb335a611a09] Fix C++ compatibility with older editline versions.
git bisect skip d203c554faa00fec55377d6640c8fb335a611a09
# skip: [b49c3a9db516a87d38476966eff9994c13c5acf6] Makefile.config.in: Remove HAVE_READLINE.
git bisect skip b49c3a9db516a87d38476966eff9994c13c5acf6
# skip: [717e821b99797845e1bef47d862f8cb0fb69cfc9] autoconf: Allow overriding CFLAGS/CXXFLAGS from outside.
git bisect skip 717e821b99797845e1bef47d862f8cb0fb69cfc9
# skip: [20129bd83d57316cc0b69bec7abeec43011c56d5] autoconf: Fix AC_STRUCT_DIRENT_D_TYPE being used before AC_PROG_CC.
git bisect skip 20129bd83d57316cc0b69bec7abeec43011c56d5
# skip: [fe068eca00d6a2dd2ea9ee08924dc3898c37fb6c] mk: add support for passing LDFLAGS to libs and bins
git bisect skip fe068eca00d6a2dd2ea9ee08924dc3898c37fb6c
# skip: [57daa860e8ed8432937aeecdcf6b9e952b0481b1] autoconf: Fix C++17 detection not working on Ubuntu 16.04.
git bisect skip 57daa860e8ed8432937aeecdcf6b9e952b0481b1
# skip: [1f97b16b1d9d93e083a4f1436ba002e073cbe379] autoconf: Work around editline not being found on Ubuntu 16.04.
git bisect skip 1f97b16b1d9d93e083a4f1436ba002e073cbe379
# skip: [00a450026f71306bef6acc150af4aa5ffe75801c] autoconf: Detect boost, require version, set CXXFLAGS.
git bisect skip 00a450026f71306bef6acc150af4aa5ffe75801c
# skip: [96cd3d607374cb4bab27095a62a6f3a832518c5a] autoconf: Change quotes in description.
git bisect skip 96cd3d607374cb4bab27095a62a6f3a832518c5a
# skip: [68bdd83dc88ec55c6c51fa92e84e7d7d408c554a] timeout: test for error code
git bisect skip 68bdd83dc88ec55c6c51fa92e84e7d7d408c554a
# skip: [a52c331edba6e8c67469f53dda0be2903d18fa8c] build: replace 100 offset for build exit codes
git bisect skip a52c331edba6e8c67469f53dda0be2903d18fa8c
# skip: [1ac399dd115d5c86629389e0cdfefa0d654fc90a] nix-store: document exit codes
git bisect skip 1ac399dd115d5c86629389e0cdfefa0d654fc90a
# skip: [99ee3755dd12e18f7085b8319a27798b0e5b7de9] build: add tests for --check status codes
git bisect skip 99ee3755dd12e18f7085b8319a27798b0e5b7de9
# skip: [cbf84bcce7d3fddb353d3aa0f012f5cbdbb77629] build: use binary mask for build status flags
git bisect skip cbf84bcce7d3fddb353d3aa0f012f5cbdbb77629
# skip: [97baf32fbca13101f58d30bb3a601302c3d560f3] build: add exit code for hash and check mismatches
git bisect skip 97baf32fbca13101f58d30bb3a601302c3d560f3
# skip: [17d3ec3405eb114f89dc42d065afa887161f6149] checkStoreName: give more precise/verbose error information
git bisect skip 17d3ec3405eb114f89dc42d065afa887161f6149
# skip: [a3c77c15369703975f7c3e65ebb829580754bc1e] nix-store: document --add-fixed
git bisect skip a3c77c15369703975f7c3e65ebb829580754bc1e
# skip: [c8205a3413217ccf8a6a1f7e064b06a5b86c3253] builtins.fetchGit: document absolute ref support
git bisect skip c8205a3413217ccf8a6a1f7e064b06a5b86c3253
# skip: [7e35e914c1aa24957107c666c76f1d834ebae90a] fetchGit: allow fetching explicit refs
git bisect skip 7e35e914c1aa24957107c666c76f1d834ebae90a
# skip: [ec58ba38c55a3ab61fdb1da7d746251e4887ea2c] Fix `http2 = false` having no effect. Fixes #2971.
git bisect skip ec58ba38c55a3ab61fdb1da7d746251e4887ea2c
# skip: [74a65d313f5c9f4d504ea9e2caadfe0ff83776d5] Nix uses the CPP SDK, not Java
git bisect skip 74a65d313f5c9f4d504ea9e2caadfe0ff83776d5
# skip: [3cc1125595d97b4ab7369e37e4ad22f4cfecb8b2] Another attempt at getting pseudoterminals to work on macOS
git bisect skip 3cc1125595d97b4ab7369e37e4ad22f4cfecb8b2
# skip: [2743bf0bb1727056ce6d6d6be4e38c3251aac1f8] Hopefully fix macOS tests
git bisect skip 2743bf0bb1727056ce6d6d6be4e38c3251aac1f8
# skip: [82ca6ef390b752faf1bf27b22daa34b29a4b00d4] Set $TERM
git bisect skip 82ca6ef390b752faf1bf27b22daa34b29a4b00d4
# skip: [e84c26564507388f2e8b7c7f2b00eb2b57856da1] Run builds in a pseudo-terminal
git bisect skip e84c26564507388f2e8b7c7f2b00eb2b57856da1
# skip: [07d9981f34c8423e74a8dc1b4f978fe58b421aee] install-multi-user: remove unneeded settings from nix.conf
git bisect skip 07d9981f34c8423e74a8dc1b4f978fe58b421aee
# skip: [dbe4c043d75f90f30932d326d2c80fd030b1f06d] install-multi-user: reduce max-jobs from 32 to 1
git bisect skip dbe4c043d75f90f30932d326d2c80fd030b1f06d
# skip: [9e0f5f803f6cbfe9925cef69a0e58cbf5375bfaf] Daemon: warn when an untrusted user cannot override a setting
git bisect skip 9e0f5f803f6cbfe9925cef69a0e58cbf5375bfaf
# skip: [4b0d6133836113572418f383ddd14c92bfebc768] Minor typo
git bisect skip 4b0d6133836113572418f383ddd14c92bfebc768
# skip: [2d34028b1e4e46d600926d4ad0c687c4d1d58c75] Add .github/FUNDING.yml
git bisect skip 2d34028b1e4e46d600926d4ad0c687c4d1d58c75
# skip: [3b1cc8b0cbc0a2530e4671231257a497db8eede2] release-common: fix build with `config.allowAliases = false`
git bisect skip 3b1cc8b0cbc0a2530e4671231257a497db8eede2
# skip: [a8251ba2ed9e833ef3e8b20d233064aba74446db] Replace `type` with `command -v` in install script
git bisect skip a8251ba2ed9e833ef3e8b20d233064aba74446db
# skip: [b2f3a7411a509d5df5a189066870a0f02b6523a9] nix-lang: Add deep nested mixed attrs test case.
git bisect skip b2f3a7411a509d5df5a189066870a0f02b6523a9
# skip: [00584bb0915ee21fb01e9390a901161bdd988197] parser: Allow mixed nested and top-level attrpaths
git bisect skip 00584bb0915ee21fb01e9390a901161bdd988197
# skip: [10d33452e289ded93e192c7d99c1da08a52448e1] nix-lang parser: Add mixed nested attrs tests.
git bisect skip 10d33452e289ded93e192c7d99c1da08a52448e1
# skip: [b502b6682b9c73ae5760967eaf7161b3e8523e9e] doc: clarify that optional attrs in a function argument will be ignored unless specified
git bisect skip b502b6682b9c73ae5760967eaf7161b3e8523e9e
# skip: [6ade7ec022c836b7d1f9bd06be45e2c07835ec8c] progress-bar: hide expected if expected is 0 (unknown)
git bisect skip 6ade7ec022c836b7d1f9bd06be45e2c07835ec8c
# skip: [92f461e4f46b595e3712b41c30b8403905dd8673] Don’t set NIX_REMOTE=daemon in daemon profile
git bisect skip 92f461e4f46b595e3712b41c30b8403905dd8673
# skip: [7c20ee448fa924d898bcebf84bd0a7caf368a656] Sync NIX_PROFILES between single-user and multi-user modes
git bisect skip 7c20ee448fa924d898bcebf84bd0a7caf368a656
# skip: [66b8a62101cb1dfe2e368346cf99efd32e9328ae] nix: Add --print-build-logs flag
git bisect skip 66b8a62101cb1dfe2e368346cf99efd32e9328ae
# skip: [ce02fc74b2db35e45906865c8a3ce2e98871eeb8] build: make needsHashRewrite a method
git bisect skip ce02fc74b2db35e45906865c8a3ce2e98871eeb8
# skip: [d75bdb5793e5ebf9e480f5a0012d141347725801] build: add test for sandboxed --check
git bisect skip d75bdb5793e5ebf9e480f5a0012d141347725801
# skip: [ff6867ab94cbe9ddcb4ba18d68a4a2dcb79b865d] build: move needsHashRewrite initialization to startBuilder
git bisect skip ff6867ab94cbe9ddcb4ba18d68a4a2dcb79b865d
# skip: [f1b8e9efe77014655f059b44afa05c38990dc4aa] runProgram: Uncomment chdir support
git bisect skip f1b8e9efe77014655f059b44afa05c38990dc4aa
# skip: [73b797c207e1c7a0fd9059d2cf1e3479502f8f1b] handleDiffHook: stop passing allowVfork
git bisect skip 73b797c207e1c7a0fd9059d2cf1e3479502f8f1b
# skip: [a5efe617862484ab7dd234a495d315e7b08aa519] Clarify where output from the diff hook goes.
git bisect skip a5efe617862484ab7dd234a495d315e7b08aa519
# skip: [b4a05edbfe49f87555fd284dfb0d6c56ed43217d] runProgram: support gid, uid, chdir
git bisect skip b4a05edbfe49f87555fd284dfb0d6c56ed43217d
# skip: [dde8eeb39ae9fb73011462c74e5fa6405e432147] chdir, setgroups
git bisect skip dde8eeb39ae9fb73011462c74e5fa6405e432147
# skip: [6df61db0600ca73ccd51e3e5bec5312a04e99da1] diff hook: execute as the build user, and pass the temp dir
git bisect skip 6df61db0600ca73ccd51e3e5bec5312a04e99da1
# skip: [c78686e411e0a14cff51836fe6c35d7584171df3] build: run diff-hook under --check and document diff-hook
git bisect skip c78686e411e0a14cff51836fe6c35d7584171df3
# skip: [82f054d7d5fc8f9de45afa7107557644d1514c98] installer: update macOS version check to 10.12.2
git bisect skip 82f054d7d5fc8f9de45afa7107557644d1514c98
# skip: [288f93cec0efeba5a767c09e4d58203c67704a6f] manual: "Nix Package collection" -> "Nixpkgs package collection".
git bisect skip 288f93cec0efeba5a767c09e4d58203c67704a6f
# skip: [fcd766097636943ff11b84747d11a4e52d3d8e38] repl: Restore CTRL-C behaviour
git bisect skip fcd766097636943ff11b84747d11a4e52d3d8e38
# skip: [cbc7d9a4124343c7ba68b695b5e9b3c8188fb267] findRootsNoTemp: fixes comment about findRuntimeRoots
git bisect skip cbc7d9a4124343c7ba68b695b5e9b3c8188fb267
# skip: [3f192ac80ca421888c668896b63485486e1397ae] Add builtins.hashFile
git bisect skip 3f192ac80ca421888c668896b63485486e1397ae
# skip: [b9567aa8b6f935522d1037377e0e5205a7529fd8] install script: don't abort when "nix-channel --update" fails
git bisect skip b9567aa8b6f935522d1037377e0e5205a7529fd8
# skip: [caa76c369a6d77eb3ccaa14bd8144581c8cd50a7] docs: Mention `--max-jobs 0` to build remotely only
git bisect skip caa76c369a6d77eb3ccaa14bd8144581c8cd50a7
# skip: [b614e0e53da1f825e198996f88935a1d1a89725f] build: only skip hash rewriting for sandboxing on linux
git bisect skip b614e0e53da1f825e198996f88935a1d1a89725f
# skip: [75ec68f93ac98956552c06a7fdf4dc8da6841a65] manual: mention the "channel:" shorthand for NIX_PATH
git bisect skip 75ec68f93ac98956552c06a7fdf4dc8da6841a65
# skip: [514b3c7f8345cfcbbe166981214497ed9d93ae18] Add isPath primop
git bisect skip 514b3c7f8345cfcbbe166981214497ed9d93ae18
# skip: [d7a7a029ff9c1fc6c5a22b91718e28c4c1062d9f] scripts: remove default 'sandbox = false' from multi-user installer
git bisect skip d7a7a029ff9c1fc6c5a22b91718e28c4c1062d9f
# skip: [2aa89daab3a8351a03f6d57d3e1f737cccd59850] eval: improve type description for primops and applied primops
git bisect skip 2aa89daab3a8351a03f6d57d3e1f737cccd59850
# skip: [6f093073b6ff38fb2c4e22331e73930baa1b30e7] manual: include builtins.* for globally available builtins
git bisect skip 6f093073b6ff38fb2c4e22331e73930baa1b30e7
# skip: [684c7fff80b575da8055015c6b87693fd1d858ca] Link to `builders-use-substitutes` in chapter on distributed builds
git bisect skip 684c7fff80b575da8055015c6b87693fd1d858ca
# skip: [81a23fa7e2eb4d58d5edf2a145af5c8bbe4c331e] Add a 5 seconds of timeouts to connect to S3.
git bisect skip 81a23fa7e2eb4d58d5edf2a145af5c8bbe4c331e
# skip: [34fade478a84d822646a8daa334014134c0ec59e] docs: fix typo in AllowDirectReads
git bisect skip 34fade478a84d822646a8daa334014134c0ec59e
# skip: [fff8db205c393c53e42326f274b322eb70259250] canBuildLocally: check for features
git bisect skip fff8db205c393c53e42326f274b322eb70259250
# skip: [8ac1130cc2f0d717fa49adf18465ecf5613d295c] nix-daemon: add variable to disable fork safety
git bisect skip 8ac1130cc2f0d717fa49adf18465ecf5613d295c
# skip: [06d633598727763c54b4b049dbc213106474d10c] fix indentation
git bisect skip 06d633598727763c54b4b049dbc213106474d10c
# skip: [ac200c3678c0f569cd962d8bbd22bb74b223d748] Apply param values on S3Helper initialization
git bisect skip ac200c3678c0f569cd962d8bbd22bb74b223d748
# skip: [56c18c67d98078dbed1d05ac68663cc52d2cb543] Extend S3 URL parsing with parameters extraction
git bisect skip 56c18c67d98078dbed1d05ac68663cc52d2cb543
# skip: [07f992a74b64f4376d5b415d0042babc924772f3] Extract and expose splitUriAndParams function
git bisect skip 07f992a74b64f4376d5b415d0042babc924772f3
# skip: [b402148d8f584144b80fe159e76e553a1792ecb7] remove noop uses of nix-store --init
git bisect skip b402148d8f584144b80fe159e76e553a1792ecb7
# skip: [5f1891b795c4c4daac5336d1ec60c94f521ede1d] nix-store: make --dump-db take a list of paths to dump
git bisect skip 5f1891b795c4c4daac5336d1ec60c94f521ede1d
# skip: [7ce1fae59f6cb4341f61a48a5216cb252e4eca54] Support --disable-shared flag.
git bisect skip 7ce1fae59f6cb4341f61a48a5216cb252e4eca54
# skip: [b30be6b450f872f8be6dc8afa28f4b030fa8d1d1] Add builtins.appendContext.
git bisect skip b30be6b450f872f8be6dc8afa28f4b030fa8d1d1
# skip: [1d757292d0cb78beec32fcdfe15c2944a4bc4a95] Add builtins.getContext.
git bisect skip 1d757292d0cb78beec32fcdfe15c2944a4bc4a95
# skip: [087be7281a02a6d0e548ae56b910771837023d21] Treat plain derivation paths in context as normal paths.
git bisect skip 087be7281a02a6d0e548ae56b910771837023d21
# skip: [15efd5437377aa7054bca753abff5451396bfdca] manual: "sandbox" option default is "true" on Linux
git bisect skip 15efd5437377aa7054bca753abff5451396bfdca
# skip: [2ae5624b2f32e3c84c57d164f2c7fef205b458cc] Update instructions for Haskell script using nix-shell interpreter
git bisect skip 2ae5624b2f32e3c84c57d164f2c7fef205b458cc
# good: [bbaf865655cff51f30be254f94c22407e368cb96] Add 2.2 release notes
git bisect good bbaf865655cff51f30be254f94c22407e368cb96
# bad: [f27e53f77e39d1247c47b48acb63af3d24ca74bf] Cleanup
git bisect bad f27e53f77e39d1247c47b48acb63af3d24ca74bf
# good: [abdedcdb387628aaa4d9dc362b7e588bbc62baae] bump cpptoml to v0.1.1
git bisect good abdedcdb387628aaa4d9dc362b7e588bbc62baae
# bad: [b5ae85f0882f4857f550ed68b23af260af5000a0] Merge pull request #2882 from grahamc/docs/1115-tarball-ttl
git bisect bad b5ae85f0882f4857f550ed68b23af260af5000a0
# bad: [5600b070a766b83200a68d3632793917cf19a550] Add "warning" verbosity level
git bisect bad 5600b070a766b83200a68d3632793917cf19a550
# bad: [7b9c68766d513260d5262d5782b46384834cdb33] Add '--no-net' convenience flag
git bisect bad 7b9c68766d513260d5262d5782b46384834cdb33
# good: [b693029ca076abf8a4084a053a122ffcdf4d61b2] Style fix
git bisect good b693029ca076abf8a4084a053a122ffcdf4d61b2
# bad: [4b214e6e4566a52fa073166d8164b79627eb5601] Merge pull request #2878 from NixOS/run-in-pts
git bisect bad 4b214e6e4566a52fa073166d8164b79627eb5601
# good: [38a4d38bc33d9943bdee4de8c77f5161bcc3fbb0] Merge pull request #2746 from bjornfor/install-multi-user-defaults
git bisect good 38a4d38bc33d9943bdee4de8c77f5161bcc3fbb0
# only skipped commits left to test
# possible first bad commit: [4b214e6e4566a52fa073166d8164b79627eb5601] Merge pull request #2878 from NixOS/run-in-pts
# possible first bad commit: [3cc1125595d97b4ab7369e37e4ad22f4cfecb8b2] Another attempt at getting pseudoterminals to work on macOS
# possible first bad commit: [2743bf0bb1727056ce6d6d6be4e38c3251aac1f8] Hopefully fix macOS tests
# possible first bad commit: [82ca6ef390b752faf1bf27b22daa34b29a4b00d4] Set $TERM
# possible first bad commit: [e84c26564507388f2e8b7c7f2b00eb2b57856da1] Run builds in a pseudo-terminal

Run builds in a pseudo-terminal sounds possibly relevant.

nh2 · 2019-12-27T14:48:15Z

I have checked manually and found that this nix commit NixOS/nix@e84c265 is the one that makes the Python test failure go away:

commit e84c26564507388f2e8b7c7f2b00eb2b57856da1
Author: Eelco Dolstra <edolstra@gmail.com>
Date:   Fri May 17 22:29:15 2019 +0200

    Run builds in a pseudo-terminal
    
    This allows many programs (e.g. gcc, clang, cmake) to print colorized
    log output (assuming $TERM is set to a value like "xterm").
    
    There are other ways to get colors, in particular setting
    CLICOLOR_FORCE, but they're less widely supported and can break
    programs that parse tool output.

@grahamc confirmed on IRC that ofborg runs nix-2.3pre6631_e58a7144 which is slightly too old to have the above commit, which is why ofborg fails here.

What I'm not sure on is: Is this now a nix bug, or a Python / Babel bug? Should the test work even when not in a pseudo-terminal? Should I set that locale?

grahamc · 2019-12-27T14:51:30Z

TBH I think the bug is squarely in Babel's side: builds shouldn't depend upon a build-time PTY and it is useful for builds to continue working in older versions of Nix. That said, ofborg needs a system upgrade anyway, which may happen today (or next month...).

nh2 · 2019-12-27T17:17:10Z

builds shouldn't depend upon a build-time PTY

I buy this argument. What should we do to ensure that though, given that Nix 2.3, and soon ofborg, may hide these problems?

nh2 · 2019-12-27T17:59:30Z

looks like export LC_ALL="en_US.UTF-8" is still needed for python27:

@jonringer I've confirmed now that adding any LC_ALL does not help if an older version of Nix is used:

Even before this PR, on current master (commit 5aae6ee), does the build fail with NIX_PATH=nixpkgs=. nix-build -A python27Packages.Babel --check if e.g. Nix 2.0.4 is used.

-    export LC_ALL="C.UTF-8"
+    export LC_ALL=C.UTF-8
The former fails (with the usual error), the latter succeeds.

Why is that? I thought these should be equivalent in bash.

I have most likely cleared this up as well now: I used remote --builders with 2 machines, one of them having Nix 2.3 and the other one Nix 2.0.4 on them. The time the removal of quotes made a difference, it probably happened to pick the 2.0.4 machine

jonringer · 2020-01-11T22:26:47Z

this is now passing on master

jonringer · 2020-01-11T22:27:05Z

going to close if no one has any reservations

FRidh · 2020-01-12T08:46:44Z

I pushed 296b2ab which (partially) fixed this.

jonringer · 2020-01-12T21:36:13Z

also, conflicts

nh2 · 2020-01-27T23:39:36Z

TBH I think the bug is squarely in Babel's side: builds shouldn't depend upon a build-time PTY and it is useful for builds to continue working in older versions of Nix.

I have figured out what the precise difference is.

From https://stackoverflow.com/questions/17430168/python-encoding-issue-when-using-linux/17430280#17430280:

When redirecting the output, sys.stdout is not connected to a terminal and Python cannot determine the output encoding. When not directing the output, Python can detect that sys.stdout is a TTY and will use the codec configured for that TTY when printing unicode.

To verify it, I printed In the Babel test suite (here):

        import sys, locale, os
        print(('sys.stdout.encoding', sys.stdout.encoding))
        print(('sys.stdout.isatty()', sys.stdout.isatty()))
        print(('locale.getpreferredencoding()', locale.getpreferredencoding()))
        print(('sys.getfilesystemencoding()', sys.getfilesystemencoding()))
        print(('os.environ["PYTHONIOENCODING"]', os.environ.get("PYTHONIOENCODING")))

and got:

# nix 2.2.2 with Python 2.7

('sys.stdout.encoding', None)
('sys.stdout.isatty()', False)
('locale.getpreferredencoding()', 'UTF-8')
('sys.getfilesystemencoding()', 'UTF-8')
('os.environ["PYTHONIOENCODING"]', None)

# nix 2.3.2 with Python 2.7

('sys.stdout.encoding', 'UTF-8')
('sys.stdout.isatty()', True)
('locale.getpreferredencoding()', 'UTF-8')
('sys.getfilesystemencoding()', 'UTF-8')
('os.environ["PYTHONIOENCODING"]', None)

# nix 2.2.2 with Python 3.7

('sys.stdout.encoding', 'UTF-8')
('sys.stdout.isatty()', False)
('locale.getpreferredencoding()', 'UTF-8')
('sys.getfilesystemencoding()', 'utf-8')
('os.environ["PYTHONIOENCODING"]', None)

# nix 2.3.2 with Python 3.7

('sys.stdout.encoding', 'UTF-8')
('sys.stdout.isatty()', True)
('locale.getpreferredencoding()', 'UTF-8')
('sys.getfilesystemencoding()', 'utf-8')
('os.environ["PYTHONIOENCODING"]', None)

As you can see:

The nix 2.2 -> 2.3 output, as expected, switches isatty() to True.
Python 3 defaults sys.stdout.encoding to 'UTF-8' if it's not a TTY, and Python 2 defaults it to None.

When on Python 2.7 with Nix 2.2 this line from the tests containing the str type contents

            Pour toute question, veuillez communiquer avec Fulano  à nadie@blah.com

is run through pofile.read_po(), the à in the line becomes (repr(invalid_po) reveals it`):

Pour toute question, veuillez communiquer avec Fulano  \\xc3\\xa0 nadie@blah.com\\n

and that cannot be print()ed when sys.stdout.encoding is None.

The issue can be reproduced on Ubuntu 18.04 by running python setup.py test | cat, where the cat makes sure that there is no TTY.

(I used Babel commit 4fa749b91.)

So it's a Babel ~~test runner~~ pofile.py bug.

nh2 · 2020-01-27T23:51:58Z

So it's a Babel pofile.py bug.

@grahamc I've PR'd a fix to Babel: python-babel/babel#691

nh2 · 2020-01-28T02:29:17Z

I've also added a comment explaining the above in #78644.

I have confirmed that musl makes no difference in this, only the nix version is relevant.

I may still want to make a PR to apply the upstream patch I submitted to fix the test, because I use some remote builders that use Nix < 2.2.

dtzWill · 2020-01-28T05:10:15Z

❤️ 🚀 Thanks for the great work chasing this down!!

(cherry picked from commit 401649d)

nh2 requested review from FRidh and jonringer as code owners December 15, 2019 02:34

nh2 requested a review from dtzWill December 15, 2019 02:34

nh2 mentioned this pull request Dec 15, 2019

pkgsMusl.python27Packages.Babel tests fail with unicode error #74904

Closed

ofborg bot added 6.topic: python 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Dec 15, 2019

nh2 force-pushed the issue-74904-python27-Babel-drop-glibcLocales-fix-musl branch from 525c135 to 9cfb480 Compare December 15, 2019 03:10

nh2 requested review from alyssais, Ericson2314, ttuegel and zimbatm as code owners December 15, 2019 03:10

nh2 changed the base branch from master to staging December 15, 2019 03:10

ofborg bot added 6.topic: golang 6.topic: nixos 6.topic: qt/kde 6.topic: ruby 6.topic: vim 8.has: changelog 8.has: documentation 8.has: module (update) 10.rebuild-darwin-stdenv 10.rebuild-linux-stdenv 8.has: clean-up labels Dec 15, 2019

jonringer reviewed Dec 22, 2019

View reviewed changes

nh2 mentioned this pull request Dec 27, 2019

Document situations where not pinning the nix version changes outputs nh2/static-haskell-nix#72

Open

nh2 mentioned this pull request Dec 27, 2019

Run builds in a pseudo-terminal NixOS/nix#2878

Merged

FRidh self-assigned this Dec 30, 2019

nh2 mentioned this pull request Jan 27, 2020

Fix unicode printing error on Python 2 without TTY. python-babel/babel#691

Merged

nh2 added a commit to nh2/nixpkgs that referenced this pull request Jan 27, 2020

pythonPackages.Babel: Add comment about NixOS#75676

19ff097

nh2 mentioned this pull request Jan 27, 2020

pythonPackages.Babel: Add comment #78644

Merged

1 task

nh2 added a commit to nh2/nixpkgs that referenced this pull request Jan 30, 2020

pythonPackages.Babel: Fix build with Nix < 2.3. Fixes NixOS#75676

ff3fb9f

nh2 mentioned this pull request Jan 30, 2020

pythonPackages.Babel: Fix build with Nix < 2.3 #78808

Merged

10 tasks

FRidh closed this in 401649d Feb 5, 2020

Staging automation moved this from Needs review to Done Feb 5, 2020

dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Feb 6, 2020

pythonPackages.Babel: Fix build with Nix < 2.3. Fixes NixOS#75676

47cb163

(cherry picked from commit 401649d)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pythonPackages.Babel: Fix use of glibcLocales, skip musl tests #75676

pythonPackages.Babel: Fix use of glibcLocales, skip musl tests #75676

nh2 commented Dec 15, 2019

nh2 commented Dec 15, 2019

nh2 commented Dec 15, 2019

nh2 commented Dec 15, 2019

nh2 commented Dec 22, 2019

jonringer Dec 22, 2019

nh2 Dec 24, 2019

jonringer Jan 11, 2020

nh2 commented Dec 27, 2019

nh2 commented Dec 27, 2019

nh2 commented Dec 27, 2019

nh2 commented Dec 27, 2019

grahamc commented Dec 27, 2019

nh2 commented Dec 27, 2019

nh2 commented Dec 27, 2019

jonringer commented Jan 11, 2020

jonringer commented Jan 11, 2020 •

edited

FRidh commented Jan 12, 2020

jonringer commented Jan 12, 2020

nh2 commented Jan 27, 2020 •

edited

nh2 commented Jan 27, 2020

nh2 commented Jan 28, 2020

dtzWill commented Jan 28, 2020

pythonPackages.Babel: Fix use of glibcLocales, skip musl tests #75676

pythonPackages.Babel: Fix use of glibcLocales, skip musl tests #75676

Conversation

nh2 commented Dec 15, 2019

Motivation for this change

Things done

Notify maintainers

nh2 commented Dec 15, 2019

nh2 commented Dec 15, 2019

nh2 commented Dec 15, 2019

nh2 commented Dec 22, 2019

jonringer Dec 22, 2019

Choose a reason for hiding this comment

nh2 Dec 24, 2019

Choose a reason for hiding this comment

jonringer Jan 11, 2020

Choose a reason for hiding this comment

nh2 commented Dec 27, 2019

nh2 commented Dec 27, 2019

nh2 commented Dec 27, 2019

nh2 commented Dec 27, 2019

grahamc commented Dec 27, 2019

nh2 commented Dec 27, 2019

nh2 commented Dec 27, 2019

jonringer commented Jan 11, 2020

jonringer commented Jan 11, 2020 • edited

FRidh commented Jan 12, 2020

jonringer commented Jan 12, 2020

nh2 commented Jan 27, 2020 • edited

nh2 commented Jan 27, 2020

nh2 commented Jan 28, 2020

dtzWill commented Jan 28, 2020

jonringer commented Jan 11, 2020 •

edited

nh2 commented Jan 27, 2020 •

edited