glibc: add patch to use utf-8 when LANG is unset #61202

matthewbauer · 2019-05-09T17:48:42Z

This allows the default builder to use utf-8 encoding. Previously,
when LANG and LC_ALL was left unset, glibc would default to “C”. This
is not good in many.

As a result of this, I think we can get rid of a lot of uses of LC_ALL
in Nixpkgs, where utf-8 was needed. These can be found by grepping
for:

(?:LC_ALL|LANG) *= *["']?(?:en_US|C)\.UTF-8["']?

A lot of times the ones using en_US.UTF-8 also add glibcLocales.

/cc @vaibhavsagar

matthewbauer · 2019-05-09T17:58:06Z

For reference, this is what musl already does:

https://wiki.musl-libc.org/functional-differences-from-glibc.html#Default-locale

Mic92 · 2019-05-09T18:52:33Z

@matthewbauer I would love to see this. However we should also take macOS into account. If we start to use UTF-8 by default on linux we also should on macOS or otherwise we break accidentally packages for macOS.

I don't have access to macOS however I roughly described in #54485 (comment) what needs to be done in order to get C.UTF-8 support.

matthewbauer · 2019-05-09T20:12:56Z

@matthewbauer I would love to see this. However we should also take macOS into account. If we start to use UTF-8 by default on linux we also should on macOS or otherwise we break accidentally packages for macOS.

I don't have access to macOS however I roughly described in #54485 (comment) what needs to be done in order to get C.UTF-8 support.

Have you ever seen this problem appear on macOS? It's possible it happens but I can't recall it happening. Usually these hacks are glibc specific.

peti

How will glibc provide that UTF-8 locale without referencing the glibcLocale files? My understanding is that those files are necessary at run-time, but glibc does not know their location.

matthewbauer · 2019-05-09T20:19:46Z

Some possible things that may work:

we can use PATH_LOCALE to put custom locales in the stdenv
create a C.UTF-8, using the same method done in https://reviews.freebsd.org/D17833
just set LC_CTYPE=UTF-8 everywhere on macOS

matthewbauer · 2019-05-09T20:28:37Z

How will glibc provide that UTF-8 locale without referencing the glibcLocale files? My understanding is that those files are necessary at run-time, but glibc does not know their location.

I don't think it's necessary for C.UTF-8. It's definitely necessary for en_US.UTF-8, but that is due to the US localizations. UTF-8 and C should be baked in to glibc.

Mic92 · 2019-05-09T21:01:36Z

@matthewbauer you don't want to set a invalid locale on macOS or things start to fail.

Mic92 · 2019-05-09T21:07:17Z

pkgs/stdenv/darwin/default.nix

@@ -37,6 +37,7 @@ in rec {
    export CMAKE_OSX_ARCHITECTURES=x86_64
    # Workaround for https://openradar.appspot.com/22671534 on 10.11.
    export gl_cv_func_getcwd_abort_bug=no
+    export LC_CTYPE=UTF-8


It would be still great if we could set LANG on macOS as well. Reason is, that $LC_CTYPE has a different semantics/priority compared to $LANG. LANG has a lower priority then all LC_* variables, so if an existing package already sets LC_ALL or similar to some value, it would be preferred.

It doesn't seem to work correctly:

$ env -i LANG=UTF-8 locale LANG="UTF-8" LC_COLLATE="C" LC_CTYPE="C" LC_MESSAGES="C" LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_ALL=

vs.

$ env -i LANG=en_US.UTF-8 locale LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL=

vs.

$ env -i LANG=en_US.UTF-8 locale LANG= LC_COLLATE="C" LC_CTYPE="UTF-8" LC_MESSAGES="C" LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_ALL=

This is the same thing described in the FreeBSD proposal:

Introduce C.UTF-8 locale, using the same common CTYPE map as other UTF-8 locales ... and having all other components use C locale

UTF-8 is only valid for LC_CTYPE, which makes sense. This is a BSDism it looks like though, so documentation on it would be helpful. My thinking on this is that if a package really needs a non-UTF-8 encoding in 2019, they should specifically request it through setting LC_CTYPE or LC_ALL. Also just to show that LC_ALL does take precedence:

$ env -i LC_ALL=en_US.UTF-8 LC_CTYPE=UTF-8 locale LANG= LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL="en_US.UTF-8"

Yeah, I meant by having a C.UTF-8 locale generated on macOS instead of using LC_CTYPE=UTF-8.

Mic92 · 2019-05-09T21:09:00Z

How will glibc provide that UTF-8 locale without referencing the glibcLocale files? My understanding is that those files are necessary at run-time, but glibc does not know their location.

The code of glibc does first look into the archive in its own installation prefix before falling back to $LOCALE_ARCHIVE, which is quite handy for our use case. In nixpkgs we put C.UTF-8 into $out/lib/locale/locale-archive of the glibc package.

matthewbauer · 2019-05-10T02:44:50Z

I should verify that c65124a3ce0b59dcf71e4ad9c72f6616f67b7079 works correctly for all of the changed files. It's possible that somewhere the locale vars are set and they specifically want one of the glibcLocales included. This might make sense for certain kinds of tests, but also seems like kind of bad practice.

edolstra · 2019-05-10T08:31:02Z

I don't know, it seems scary to patch Glibc in this way. Does upstream have an opinion about this?

matthewbauer · 2019-05-10T14:34:04Z

I don't know, it seems scary to patch Glibc in this way. Does upstream have an opinion about this?

@edolstra Upstream ticket and discussions:

It looks like it's part of the roadmap, just never been done? I can't find any distros that have done the switch for the default, though many provide C.UTF-8 in their glibc package (as we already do). It seems reasonable to me to enable UTF-8 by default, though.

vcunat · 2019-05-15T18:40:54Z

Why do these even use the huge full glibcLocales and not one with only this single language? 🤷‍♀️

matthewbauer · 2019-05-15T22:11:17Z

Yeah unfortunately C.UTF-8 only works on Glibc/Musl. So we need to to either use something that works on macOS like en_US.UTF-8 or else apply a patch like this.

This allows the default builder to use utf-8 encoding. Previously, when LANG and LC_ALL was left unset, glibc would default to “C”. This is not good in many. As a result of this, I think we can get rid of a lot of uses of LC_ALL in Nixpkgs, where utf-8 was needed. These can be found by grepping for: (?:LC_ALL|LANG) *= *["']?(?:en_US|C)\.UTF-8["']?

A couple of places set LANG=“en_US.UTF-8” to get unicode working in the Nix builder. This is very very bad! Not everyone who uses Nix lives in the U.S. and is an English speaker. Instead we should use the “C.UTF-8”, which is an UTF-8 capable version of the unlocalized C locale. Please use this one! The previous commit makes Glibc fall back to this locale when LANG is unset. This is in line with the Musl behavior. As a result, the Nix builder is made UTF-8 capable without any LOCALE_ARCHIVE or LANG hacks.

Unfortunately, the default macOS locale does not have UTF-8 support. You need to set this for correct behavior.

These break due to default locale not being what gnulib expects.

Mic92 · 2019-07-26T16:07:35Z

pkgs/development/libraries/glibc/common.nix

@@ -92,6 +92,11 @@ stdenv.mkDerivation ({
        url = "https://salsa.debian.org/glibc-team/glibc/raw/49767c9f7de4828220b691b29de0baf60d8a54ec/debian/patches/localedata/locale-C.diff";
        sha256 = "0irj60hs2i91ilwg5w7sqrxb695c93xg0ik7yhhq9irprd7fidn4";
      })
+      (fetchurl {
+        url = "https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=874160;filename=0001-Default-to-C.UTF-8-on-setlocale-.-if-no-env-vars-are.patch;msg=5";
+        name = "0001-Default-to-C.UTF-8-on-setlocale-.-if-no-env-vars-are.patch";


So debian switched to C.UTF-8 now?

No this is just a proposed patch

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=874160

lovesegfault · 2020-02-28T22:49:09Z

@matthewbauer This needs a rebase :)

Mic92 · 2020-03-04T17:06:38Z

I got a osx vm now and could actually test the patch I proposed here: #54485 (comment)

Mic92 · 2020-05-26T06:59:54Z

Alacritty also switched to LC_CTYPE=UTF-8 by default on macOS: alacritty/alacritty#3608 Maybe we just do the same.

Mic92 · 2020-05-26T07:03:42Z

Instead of patching glibc to default to C.UTF-8 we could start instead with a less intrusive version and just that LANG=C.UTF-8 in stdenv.

matthewbauer · 2020-05-26T14:32:13Z

Closing for now - glibc probably shouldn't be patched like this & probably should just set LANG=C.UTF-8 in stdenv.

matthewbauer requested a review from basvandijk as a code owner May 9, 2019 18:13

matthewbauer requested review from fpletz, FRidh, vcunat, peti and Mic92 May 9, 2019 18:13

ofborg bot added the 10.rebuild-linux-stdenv label May 9, 2019

ofborg bot requested a review from edolstra May 9, 2019 18:31

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ 6.topic: erlang 6.topic: haskell labels May 9, 2019

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ and removed 10.rebuild-darwin: 11-100 labels May 9, 2019

peti reviewed May 9, 2019

View reviewed changes

Mic92 reviewed May 9, 2019

View reviewed changes

ofborg bot added the 6.topic: stdenv Standard environment label May 9, 2019

ofborg bot added the 10.rebuild-darwin-stdenv label May 9, 2019

matthewbauer changed the base branch from master to staging May 18, 2019 19:58

FRidh added this to the 19.09 milestone May 28, 2019

cw789 mentioned this pull request Jun 23, 2019

elixir_1_9: init at 1.9.0 #63606

Merged

10 tasks

matthewbauer mentioned this pull request Jul 10, 2019

ghc does not support Unicode, reports locale encoding ASCII. #64603

Closed

matthewbauer added 3 commits July 10, 2019 17:01

darwin-stdenv: set LC_CTYPE to UTF-8

0ebb6d2

Unfortunately, the default macOS locale does not have UTF-8 support. You need to set this for correct behavior.

matthewbauer force-pushed the glibc-default-to-utf8 branch from 977d329 to 0ebb6d2 Compare July 10, 2019 21:03

matthewbauer added 2 commits July 16, 2019 16:44

gnulib: skip localename tests

a5999f6

These break due to default locale not being what gnulib expects.

nixos: add release notes for glibc default locale

9b218d1

matthewbauer force-pushed the glibc-default-to-utf8 branch from 2b3829b to 9b218d1 Compare July 17, 2019 19:51

ofborg bot added 6.topic: nixos 8.has: changelog 8.has: documentation 2.status: merge conflict labels Jul 17, 2019

Merge branch 'staging' into glibc-default-to-utf8

6b828a2

ofborg bot removed the 2.status: merge conflict label Jul 26, 2019

Mic92 reviewed Jul 26, 2019

View reviewed changes

matthewbauer modified the milestones: 19.09, 20.03 Sep 4, 2019

matthewbauer removed this from the 20.03 milestone Nov 7, 2019

matthewbauer mentioned this pull request Apr 29, 2020

Change the order of locale archive search path #85823

Open

matthewbauer closed this May 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

glibc: add patch to use utf-8 when LANG is unset #61202

glibc: add patch to use utf-8 when LANG is unset #61202

matthewbauer commented May 9, 2019 •

edited

matthewbauer commented May 9, 2019

Mic92 commented May 9, 2019 •

edited

matthewbauer commented May 9, 2019

peti left a comment

matthewbauer commented May 9, 2019

matthewbauer commented May 9, 2019

Mic92 commented May 9, 2019

Mic92 May 9, 2019

matthewbauer May 10, 2019 •

edited

Mic92 May 10, 2019 •

edited

Mic92 commented May 9, 2019 •

edited

matthewbauer commented May 10, 2019

edolstra commented May 10, 2019

matthewbauer commented May 10, 2019

vcunat commented May 15, 2019 •

edited

matthewbauer commented May 15, 2019

Mic92 Jul 26, 2019

matthewbauer Jul 26, 2019

matthewbauer Jul 26, 2019

lovesegfault commented Feb 28, 2020

Mic92 commented Mar 4, 2020

Mic92 commented May 26, 2020 •

edited

Mic92 commented May 26, 2020

matthewbauer commented May 26, 2020

glibc: add patch to use utf-8 when LANG is unset #61202

glibc: add patch to use utf-8 when LANG is unset #61202

Conversation

matthewbauer commented May 9, 2019 • edited

matthewbauer commented May 9, 2019

Mic92 commented May 9, 2019 • edited

matthewbauer commented May 9, 2019

peti left a comment

Choose a reason for hiding this comment

matthewbauer commented May 9, 2019

matthewbauer commented May 9, 2019

Mic92 commented May 9, 2019

Mic92 May 9, 2019

Choose a reason for hiding this comment

matthewbauer May 10, 2019 • edited

Choose a reason for hiding this comment

Mic92 May 10, 2019 • edited

Choose a reason for hiding this comment

Mic92 commented May 9, 2019 • edited

matthewbauer commented May 10, 2019

edolstra commented May 10, 2019

matthewbauer commented May 10, 2019

vcunat commented May 15, 2019 • edited

matthewbauer commented May 15, 2019

Mic92 Jul 26, 2019

Choose a reason for hiding this comment

matthewbauer Jul 26, 2019

Choose a reason for hiding this comment

matthewbauer Jul 26, 2019

Choose a reason for hiding this comment

lovesegfault commented Feb 28, 2020

Mic92 commented Mar 4, 2020

Mic92 commented May 26, 2020 • edited

Mic92 commented May 26, 2020

matthewbauer commented May 26, 2020

matthewbauer commented May 9, 2019 •

edited

Mic92 commented May 9, 2019 •

edited

matthewbauer May 10, 2019 •

edited

Mic92 May 10, 2019 •

edited

Mic92 commented May 9, 2019 •

edited

vcunat commented May 15, 2019 •

edited

Mic92 commented May 26, 2020 •

edited