FreeType 2.7.1 and Fontconfig defaults #23819
Conversation
The Infinality bytecode interpreter is removed in favor of the new v40 TrueType interpreter. In the past, the Infinality interpreter provided support for ClearType-style hinting instructions while the default interpreter (then v35) provided support only for original TrueType-style instructions. The v40 interpreter corrects this deficiency, so the Infinality interpreter is no longer necessary. To understand why the Infinality interpreter is no longer necessary, we should understand how ClearType differs from TrueType and how the v40 interpreter works. The following is a summary of information available on the FreeType website [1] mixed with my own editorializing. TrueType instructions use horizontal and vertical hints to improve glyph rendering. Before TrueType, fonts were only vertically hinted; horizontal hints improved rendering by snapping stems to pixel boundaries. Horizontal hinting is a risk because it can significantly distort glyph shapes and kerning. Extensive testing at different resolutions is needed to perfect the TrueType hints. Microsoft invested significant effort to do this with its "Core fonts for the Web" project, but few other typefaces have seen this level of attention. With the advent of subpixel rendering, the effective horizontal resolution of most displays increased significantly. ClearType eschews horizontal hinting in favor of horizontal supersampling. Most fonts are designed for the Microsoft bytecode interpreter, which implements a compatibility mode with TrueType-style (horizontal and vertical) instructions. However, applying the full horizontal hints to subpixel-rendered fonts leads to color fringes and inconsistent stem widths. The Infinality interpreter implements several techniques to mitigate these problems, going so far as to embed font- and glyph-specific hacks in the interpreter. On the other hand, the v40 interpreter ignores the horizontal hinting instructions so that glyphs render as they are intended to on the Microsoft interpreter. Without the horizontal hints, the problems of glyph and kerning distortion, color fringes, and inconsistent stem widths--the problems the Infinality interpreter was created to solve--simply don't occur in the first place. There are also security concerns which motivate removing the Infinality patches. Although there is an updated version of the Infinality interpreter for FreeType 2.7, the lack of a consistent upstream maintainer is a security concern. The interpreter is a Turing-complete virtual machine which has had security vulnerabilities in the past. While the default interpreter is used in billions of devices and is maintained by an active developer, the Infinality interpreter is neither scrutinized nor maintained. We will probably never know if there are defects in the Infinality interpreter, and if they were discovered they would likely never be fixed. I do not think that is an acceptable situtation for a core library like FreeType. Dropping the Infinality patches means that font rendering will be less customizable. I think this is an acceptable trade-off. The Infinality interpreter made many compromises to mitigate the problems with horizontal hinting; the main purpose of customization is to tailor these compromises to the user's preferences. The new interpreter does not have to make these compromises because it renders fonts as their designers intended, so this level of customization is not necessary. The Infinality-associated patches are also removed from cairo. These patches only set the default rendering options in case they aren't set though Fontconfig. On NixOS, the rendering options are always set in Fontconfig, so these patches never actually did anything for us! The Fontconfig test suite is patched to account for a quirk in the way PCF fonts are named. The fontconfig option `hintstyle` is no longer configurable in NixOS. This option selects the TrueType interpreter; the v40 interpreter is `hintslight` and the older v35 interpreter is `hintmedium` or `hintfull` (which have actually always been the same thing). The setting may still be changed through the `localConf` option or by creating a user Fontconfig file. Users with HiDPI displays should probably disable hinting and antialiasing: at best they have no visible effect. The fontconfig-ultimate settings are still available in NixOS, but they are no longer the default. They still work, but their main purpose is to set rendering quirks which are no longer necessary and may actually be detrimental (e.g. setting `hintfull` for some fonts). Also, the vast array of font substitutions provided is not an appropriate default; the default setting should be to give the user the font they asked for. [1]. https://www.freetype.org/freetype2/docs/subpixel-hinting.html
|
I, for one, welcome our new penultimate overlords! More seriously, nice to have freetype-ultimate cleaned up with something from upstream, we've been even dragging along 3rd party patches to make it build lately. From your overview it's not immediately obvious what is |
Also provides a NixOS module.
|
For the Changelog, I said,
Should I be more detailed? I thought for the Changelog, a high-level summary would be sufficient. |
|
I haven't tested your changes but high-level things now seem okay to me. Thanks! |
|
I have tested this myself enough to be sure it's not obviously broken, but I think it will not see other testing until Hydra builds it, so in we go! |
|
My rendered fonts has become thinner and... worse (subjectively). I'll try to find out differences later but maybe you have any immediate ideas? The fonts themselves has not changed, so this probably is something about hinting/antialiasing. I use default fontconfig settings sans DPI, difference is most visible with Source Code Pro (in Emacs). |
|
It might depend on the particular screen, too. I've seen some screenshot of Apple-rendered fonts and it was ugly on my screen (visible color fringes etc.) but it reportedly looked perfect on the original device... |
|
The screen is the same (a laptop screen) so it shouldn't make a difference... |
|
@abbradar: I meant that it's possible that it now appears ugly on your screen but not on "ours". |
|
Indeed. I'll just try to figure out differences in |
|
I must say that at least new nixpkgs (updated freetype) + old nixos (old |
|
@abbradar: oh, wait. Selecting "full hinting" is likely to be the culprit. Better use "slight hinting", AFAIK. EDIT: I don't have a clue why I had "full" on this system, but it looked OK only with infinality. |
|
Seems it's already like that:
|
|
Hmm, killing the "full hinting" global default fixed all visual issues I could see here, but maybe that's unrelated to what you're seeing. |
@abbradar I too use Source Code Pro in Emacs. The old For debugging, you actually want to run |
|
The problem was actually in two places:
It seems much better now, thank you all! |
|
🎉 I guess we should add those two points to the release notes for future/17.09? |
|
Maybe think of some generic mechanism to find out how the font is named now? So that we can recommend "use EDIT: this doesn't seem possible without pulling the whole |
|
I think it's enough to suggest the user check their hinting settings if rendering seems off and use |
|
My fonts NixOS config
~~~~
{
font.fontconfig.enable = true;
# plus a long list of fonts
# is all there is
}
~~~~
While I like that after this or the previous font patchset now I don't
need to add `autohint=true` to random configs, on today's master with
xftfont unifont:size=12:antialias=false
in conky "C", "X", "Y", and "x" glyphs look really ugly (aka "fontconfig
ugly broken letters", for SEO, or "some strokes turn into dotted lines
or look as if display had some random pixels stuck" for humans).
See for yourself with
conky -c ./file.rc
with the following file.rc:
~~~~
background no
out_to_console no
out_to_stderr no
alignment top_left
gap_x 20
gap_y 20
minimum_size 400 400
maximum_width 400
double_buffer yes
own_window yes
own_window_class Conky
own_window_type override
own_window_transparent yes
use_xft yes
xftfont unifont:size=12:antialias=false
default_color white
TEXT
ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
~~~~
The diff of
FC_DEBUG=1 fc-match 'unifont:size=12:antialias=false'
outputs between configurations gives
~~~~
--- older
+++ newer
@@ -1,6 +1,6 @@
FC_DEBUG=1
-Match Pattern has 25 elts (size 32)
- family: "unifont"(s) [censored for privacy]
+Match Pattern has 27 elts (size 32)
+ family: "unifont"(s) [censored for privacy]
familylang: "en"(s) "en-us"(w)
stylelang: "en"(s) "en-us"(w)
fullnamelang: "en"(s) "en-us"(w)
@@ -9,18 +9,20 @@
width: 100(i)(s)
size: 12(f)(s)
pixelsize: 12.5(f)(s)
- antialias: False(s)
- hintstyle: 1(i)(w)
- hinting: True(s)
+ antialias: False(s) True(w) True(w)
+ hintstyle: 1(i)(w) 1(i)(w) 1(i)(w)
+ hinting: True(w) True(w)
verticallayout: False(s)
- autohint: False(s)
+ autohint: True(w) False(w)
globaladvance: True(s)
dpi: 75(f)(s)
+ rgba: 1(i)(w) 1(i)(w)
scale: 1(f)(s)
lang: "en"(w)
fontversion: 2147483647(i)(s)
embeddedbitmap: True(s)
decorative: False(s)
+ lcdfilter: 1(i)(w) 1(i)(w)
namelang: "en"(s)
prgname: "fc-match"(s)
symbol: False(s)
@@ -39,7 +41,7 @@
width: 100(i)(w)
spacing: 90(i)(w)
foundry: "PfEd"(w)
- file: "/nix/store/[censored]-unifont-9.0.06/share/fonts/truetype/unifont.ttf"(w)
+ file: "/nix/store/[censored]-unifont-9.0.06/share/fonts/truetype/unifont.ttf"(w)
index: 0(i)(w)
outline: True(w)
scalable: True(w)
~~~~
Meanwhile
(set-default-font "unifont:size=16:antialias=false")
in Emacs, while still looking pretty ugly, doesn't produce any glitches
(`size=12` is way too small to see).
`FreeMono` font looks fine with the current default config, but its too
wide for my conky.
Adding `autohint=false` to that conky line with `unifont` doesn't change
a thing. Removing `antialias=false` from there helps, but then my brain
starts interpreting the resulting fuzzyness of antialiased `unifont` as
me losing a contact lens or something. Which is disabling, since I use
conky to display a lot of important stuff (e.g. org-agenda) and look at
it frequently. I tried to tinker with the rest of the options that
changed to no avail.
Also, generally, changes like
- autohint: False(s)
+ autohint: True(w) False(w)
look suspicious to me. Not the "False -> True" part, but the
multiplication part. I use the super-vanilla one-line default config,
why does the default config produce some overrides? (I assume those
sequences are overrides.)
In the worst case, I can live with ugly glyphs in conky, but this feels
like a regression to me.
|
Could you please pastebin the full before and after? The diff is missing important context.
They are not overrides, more like a default and a fallback. When there are multiple entries in a field, Fontconfig takes the first one it can satisfy. In this case, that is perfectly fine if it occurs in the pattern debugging, but without the omitted context I can't tell if that's line is from a pattern or from a match.
It certainly is a regression. I didn't look hard at bitmap font rendering because, well, they're supposed to be bitmaps! I'm also a little confused why there's a |
|
@oxij Using |
|
Thomas Tuegel <notifications@github.com> writes:
Could you please pastebin the full before and after? The diff is missing important context.
@oxij Using `unifont:size=12:antialias=false:hinting=false` fixes the
problem for me, could you confirm that it works for you?
It does! Thanks!
If this solves the problem for you, I will add a rule to our Fontconfig settings for Unifont.
Feels a bit ad-hocish, but better than nothing, I guess.
|
Motivation for this change
Besides updating FreeType to the latest version 2.7.1, a number of changes are made to the patches we carry on FreeType and Cairo which I believe will reduce maintenance burden and improve security. To account for the new TrueType interpreter in FreeType 2.7, the defaults provided by fontconfig-ultimate (tuned for the Infinality interpreter) have been replaced by my own derivative thereof, fontconfig-penultimate. Along the way, I discovered that fontconfig-ultimate wasn't doing the right thing in several cases (probably because it isn't properly maintained). The new defaults are significantly simpler.
The commit message has most of the relevant details:
The Infinality bytecode interpreter is removed in favor of the new v40 TrueType
interpreter. In the past, the Infinality interpreter provided support for
ClearType-style hinting instructions while the default interpreter (then v35)
provided support only for original TrueType-style instructions. The v40
interpreter corrects this deficiency, so the Infinality interpreter is no longer
necessary.
To understand why the Infinality interpreter is no longer necessary, we should
understand how ClearType differs from TrueType and how the v40 interpreter
works. The following is a summary of information available on the FreeType
website [1] mixed with my own editorializing.
TrueType instructions use horizontal and vertical hints to improve glyph
rendering. Before TrueType, fonts were only vertically hinted; horizontal hints
improved rendering by snapping stems to pixel boundaries. Horizontal hinting is
a risk because it can significantly distort glyph shapes and kerning. Extensive
testing at different resolutions is needed to perfect the TrueType
hints. Microsoft invested significant effort to do this with its "Core fonts for
the Web" project, but few other typefaces have seen this level of attention.
With the advent of subpixel rendering, the effective horizontal resolution of
most displays increased significantly. ClearType eschews horizontal hinting in
favor of horizontal supersampling. Most fonts are designed for the Microsoft
bytecode interpreter, which implements a compatibility mode with
TrueType-style (horizontal and vertical) instructions. However, applying the
full horizontal hints to subpixel-rendered fonts leads to color fringes and
inconsistent stem widths. The Infinality interpreter implements several
techniques to mitigate these problems, going so far as to embed font- and
glyph-specific hacks in the interpreter. On the other hand, the v40 interpreter
ignores the horizontal hinting instructions so that glyphs render as they are
intended to on the Microsoft interpreter. Without the horizontal hints, the
problems of glyph and kerning distortion, color fringes, and inconsistent stem
widths--the problems the Infinality interpreter was created to solve--simply
don't occur in the first place.
There are also security concerns which motivate removing the Infinality patches.
Although there is an updated version of the Infinality interpreter for FreeType
2.7, the lack of a consistent upstream maintainer is a security concern. The
interpreter is a Turing-complete virtual machine which has had security
vulnerabilities in the past. While the default interpreter is used in billions
of devices and is maintained by an active developer, the Infinality interpreter
is neither scrutinized nor maintained. We will probably never know if there are
defects in the Infinality interpreter, and if they were discovered they would
likely never be fixed. I do not think that is an acceptable situtation for a
core library like FreeType.
Dropping the Infinality patches means that font rendering will be less
customizable. I think this is an acceptable trade-off. The Infinality
interpreter made many compromises to mitigate the problems with horizontal
hinting; the main purpose of customization is to tailor these compromises to the
user's preferences. The new interpreter does not have to make these compromises
because it renders fonts as their designers intended, so this level of
customization is not necessary.
The Infinality-associated patches are also removed from cairo. These patches
only set the default rendering options in case they aren't set though
Fontconfig. On NixOS, the rendering options are always set in Fontconfig, so
these patches never actually did anything for us!
The Fontconfig test suite is patched to account for a quirk in the way PCF fonts
are named.
The fontconfig option
hintstyleis no longer configurable in NixOS. Thisoption selects the TrueType interpreter; the v40 interpreter is
hintslightandthe older v35 interpreter is
hintmediumorhintfull(which have actuallyalways been the same thing). The setting may still be changed through the
localConfoption or by creating a user Fontconfig file.Users with HiDPI displays should probably disable hinting and antialiasing: at
best they have no visible effect.
The fontconfig-ultimate settings are still available in NixOS, but they are no
longer the default. They still work, but their main purpose is to set rendering
quirks which are no longer necessary and may actually be
detrimental (e.g. setting
hintfullfor some fonts). Also, the vast array offont substitutions provided is not an appropriate default; the default setting
should be to give the user the font they asked for.
[1]. https://www.freetype.org/freetype2/docs/subpixel-hinting.html
Things done
(nix.useSandbox on NixOS,
or option
build-use-sandboxinnix.confon non-NixOS)
nix-shell -p nox --run "nox-review wip"./result/bin/)I expect these changes to be controversial, so I will not merge immediately to allow for discussion.