Oswald not rendering in Firefox and Chromium on Linux

[jean-dao]

It has been a few weeks now that the Oswald font does not render anymore neither in Firefox nor Chromium on Linux (Debian 9.5). It works fine on Windows (both with Firefox and Chrome).

A related Stackoverflow post exists, but it seems no issue has been opened yet.

As described in the Stackoverflow post, if you go to https://fonts.google.com/specimen/Oswald, the text shows up briefly then disappears.

Looking in the Network -> Font page in the Chromium inspector, 5 Oswald fonts are queried (https://fonts.gstatic.com/s/oswald/v23/...) and return 200. All but one show an actual font in the preview tab. https://fonts.gstatic.com/s/oswald/v23/TK3iWkUHHAIjg752GT8G.woff2 shows nothing in the preview, although the status is 200. The 4 other Oswald fonts seem to show a default font in the preview, not the actual Oswald font. Clearing Chromium cache doesn't change anything.

In Chrome on Windows, I get the exact same 5 fonts queries, and the one missing on Linux shows up on Windows, showing the actual Oswald font in the preview tab.

[m4rc1e]

Thanks for reporting this.

Oswald works for me on Debian 9.9, FireFox 60.7.2esr

I'll check Chrome once I have it installed. I'll then repeat the checks on Debian 9.5

What version of Firefox and Chrome did you test on?

[jean-dao]

I tested on Firefox 67.0.4 and Chromium 71.0.3578.80.

[m4rc1e]

Ok, I've had some time to investigate this.

TL;DR
Debian uses an older version of FreeType which doesn't fully support variable fonts. We've hinted our variable fonts using an experimental version of ttfautohint which relies on a newer version of FreeType. This means that Debian's FreeType may not work correctly with our autohinted variable fonts.

---

Around a month ago, we upgraded the Oswald family. We converted all six fonts into a single variable font. We then autohinted the font using an experimental version of ttfautohint. We thoroughly tested the new font on OS X, Win, Android, IOS and Ubuntu. There were no significant visual regressions so we went ahead and released it.

Debian Stretch, 9x (current stable release) uses freetype 2.6.3 by default. This is a fairly old release and it doesn't support variable fonts. Variable font support was added in 2.8.3. If I view Oswald in Google Fonts in FireFox 67, I get the following:

This is clearly incorrect.

If I then upgrade my libfreetype to 2.9.1 and recheck in FireFox 67, everything is correct.

Chrome is a different story. Even with the upgraded system libfreetype, it produced incorrect output. This leads me to believe that Chrome may have its own sandboxed version of FreeType which may be unique for each Linux distro. @drott is this statement correct? (apologies if this is a ridiculous question)

Strangely, I can get the family to render correctly on Chrome with freetype 2.6.3, if I run it with the following flags:

FREETYPE_PROPERTIES="truetype:interpreter-version=35"  google-chrome

Unhinted variable fonts don't suffer with this issue. They seem to render just fine with freetype 2.6.3.

I suspect the issue may be that we've hinted the fonts with our experimental ttfautohint which has functions that don't work correctly in older versions of FreeType. Or quite simply, our experimental ttfautohint is faulty and shouldn't be used.

@garretrieger Idk if user agents are able to tell us which distro of Linux a user is using. Perhaps it may be best to disable variable fonts for Linux, until most major distros start using FreeType 2.8.1+? An alternative solution is to release our variable fonts unhinted. However, this may upset users of other platforms. Ideally we should invest some time into improving ttfautohint-vf and testing this issue further?

Sorry for the long speculative post. This was a lot more research/testing than I anticipated. If I was to invest more time, I'd try and determine whether the fault lies fully with the autohinter or older versions of FreeType. It's quite interesting because the fonts render just fine in Ubuntu 18 since it uses a newer FreeType. However, this autohinter is definitely a cause for concern

[drott]

Chrome is a different story. Even with the upgraded system libfreetype, it produced incorrect output. This leads me to believe that Chrome may have its own sandboxed version of FreeType which may be unique for each Linux distro. @drott is this statement correct? (apologies if this is a ridiculous question)

Chrome uses its own bundled FreeType and HarfBuzz, which is always very very new for the Canary versions, and pretty new in the stable Chrome version. If you tell me which Chrome version you tested with, I can tell you which FreeType commit was used in that. So the system FreeType version in Chrome will not matter for Chrome's rendering. But specifiying a FREETYPE_PROPERTIES="truetype:interpreter-version=35" environment variable I believe does affect Chrome rendering. However, specifying that will downgrade the TrueType interpreter that FreeType uses inside Chromium - indicating perhaps really that something is not quite right with how the font is hinted?

[m4rc1e]

@drott thank you for this information!

If you tell me which Chrome version you tested with, I can tell you which FreeType commit was used

I'm using Chrome 75.0.3770.100 (Official Build) (64 bit) on Debian 9.9.

perhaps really that something is not quite right with how the font is hinted?

I just made a new test by hinting the variable font using the standard ttfautohint. I get the same bad results in Chrome.

It's reassuring to know that our custom ttfautohint has the same faults as the standard version. Both autohinters may be broken.

Strangely, I don't get any rendering issues using Chrome 75 on Ubuntu 18.0. Ubuntu 18.0 has FreeType 2.8.1.

I'm now wondering if Debian's font settings are causing this issue. I may have to keep going down this rabbit hole.

[drott]

I believe we do take FontConfig settings into account. So if you can test by assigning the different FontConfig hinting settings, hintslight, hintmedium, hintfull etc, those may cause differences, but they should only do that for local fonts, not for web fonts.

[m4rc1e]

@drott A user reported today that it's broken in Win Chrome 76+, googlefonts/OswaldFont#16. I take it I can find specific FreeType version info from the main repo by viewing /third_party/freetype /README.chromium? so for version 76, I can go to https://chromium.googlesource.com/chromium/src/+/refs/tags/76.0.3809.55/third_party/freetype/README.chromium

@garretrieger Since this issue has arisen twice on different platforms, it may be best to roll back to static fonts for the time being. Comfortaa should also be rolled back. The other VFs we've released are unhinted so they should be fine. I will dedicate more time to work out exactly what the issue is today.

[bungeman]

I took a look at this font ( https://fonts.gstatic.com/s/oswald/v23/TK3iWkUHHAIjg752GT8G.woff2 , converted to ttf) in ftmulti (part of the FreeType demos) at current tip of tree FreeType which allows easily adjusting all the parameters. I can reproduce when using the v40 interpreter and asking for hinting. With this setup, as the size changes the advances do not change (they seem to be stuck at size ~16). Note that this does not happen with the v35 interpreter (so setting FREETYPE_PROPERTIES="truetype:interpreter-version=35" should affect this in Chrome), nor with hinting off. Also at size 201 the advances become normal again (200 seems to be a special number in the 'prep' table). I'll try to take a look at what exactly is actually happening in the hinting.

[m4rc1e]

@bungeman thank you for looking into this. I get the same results as you using ftmulti. I previously inspected the font using ftview and ftstring and they looked ok. I guess these tools may lack variable font support.

Also at size 201 the advances become normal again (200 seems to be a special number in the 'prep' table). I'll try to take a look at what exactly is actually happening in the hinting.

TTFautohint has the following arg:

  -G, --hinting-limit=N      switch off hinting above this PPEM value
                             (default: 200); value 0 means no limit

This is probably the culprit.

Here's a link to the custom ttfautohint I built, https://groups.google.com/d/msg/googlefonts-discuss/WJX1lrzcwVs/SIzaEvntAgAJ. It comes from a GSOC project to add variable font support to ttfautohint, https://gitlab.com/ewaldhew/tah-vf/commits/gvar-wip. It's still WIP, however, I tested it on OS X, Win and Ubuntu and it looked great. I foolishly believed this was enough. Interestingly, I get the same broken results when I use the normal ttfautohint which can be downloaded here, https://www.freetype.org/ttfautohint/#download. I guess all version of ttfautohint may have the same issue.

[bungeman]

I can get this to happen in ftview by opening this font at a size ~40, setting the hinting engine to the v40 engine, selecting the 'light AA' hinting method, and then choosing any other non-monochrome hinting method. Something like "ftview 40 TK3iWkUHHAIjg752GT8G.ttf" then 'C' (to select 'light AA' then 'B' (to select 'normal AA'). If I then force the engine back to v40 then everything seems to go back to working.

Interestingly, I think this is just because 'light AA' is forcing autohinting on. Doing "ftview 40 TK3iWkUHHAIjg752GT8G.ttf", then 'f', then 'f' (to toggle force autohinting on and off) seems to reproduce the issue as well. If one hits 'H' to toggle the hinting engine to v35 and then back to v40 everything draws fine again. Toggling other settings does not seem to affect the state of the issue. Toggling 'forced autohint' when using the v35 engine does not cause this issue nor fix it. In fact, if I change ftcommon.h to make 'autohint' on by default, I can run ftview and just hit 'f' and reproduce. However, this doesn't seem to explain anything with regard to what is happening in ftmulti, since it never seems to set 'force autohint'.

[bungeman]

I've opened https://savannah.nongnu.org/bugs/index.php?56601 . This is an issue in FreeType when using HVAR/VVAR with a hinted glyph with the v40 minimal hinter. In this case the phantom points may not be scaled correctly, leaving the advance to be reported in funits instead of 26.6 fixed point pixels.

[m4rc1e]

Thank you so much @bungeman for the thorough investigation! This has saved me a significant amount of time. It has really brightened my day.

I'm guessing you are based in MTV? if so, I'd definitely like to thank you in person when I'm next over.

[bungeman]

To clarify, this FreeType issue affects FreeType versions 2.9 to current 2.10.1 (a patch exists and hopefully a fix will be landed soon). Note that there may be a difference in versions of FreeType used by Chrome vs Chromium. Chrome is built by Google and ships with its own FreeType which is kept as close to FreeType master as practical. Chromium is built by the distribution and can be built to link against the system FreeType. (I think this is what leads to some early confusion on this thread, since the original report seems to be against Chromium, but then later investigation seems to be using Chrome.)

Unfortunately, this means Ubuntu disco, Debian buster, Fedora 29 and 30, and possibly others are affected. Chrome will update fairly soon with a fix, but other apps (including Chromium) may continue to have issues on these platforms with these sorts of fonts unless they cherry-pick back a patch.

[sterndata]

I can confirm the problem exists on Fedora 30.

[drott]

The fix landed in https://bugs.chromium.org/p/chromium/issues/detail?id=981443#c7 thanks to @bungeman who fixed this in FreeType. Please retest in a Canary, Canary 77.0.3852.0 will probably have this fix (not before).

[bungeman]

This is now fixed in Chrome Canary 77.0.3853.0 and later, and should be fixed in Chrome 76.0.3809.68 and later. The FreeType change of interest is https://git.savannah.gnu.org/cgit/freetype/freetype2.git/commit/?id=12e4307dc7b48c9a4a4fc3ac6c32220874ab18ec . Unfortunately, I doubt I will have time to file issues against the distribution packages, so if someone here is motivated enough to do so, it may be interesting to bring this to the attention of the distributions and request they apply this patch to FreeType 2.9 through to (and including) 2.10.1. I think updating the distributions may be necessary to fix this issue in Chromium, Firefox, and possibly others.

Note that this FreeType change is known to fix the advance width issue as seen in the screen captures on this issue. However, this does not fix the issue in the related stack overflow report. That issue appears to have been introduced with FreeType 052f6c5649f7a46e599742eb6f47fa91d090f77e in VER-2-6-2 and fixed with FreeType 4e659d7eadb2cb17c28f1c23ac77cebd7b7e3e33 in VER-2-7.

[bungeman]

The Debian 9 issue from the stack overflow post (which seems to be the issue in the original report here) is tracked at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=929982 . This has been assigned to the FreeType package with the above information about when the issue was introduced and fixed in FreeType.

[bungeman]

Created a request at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932303 for debian to apply the fix for the FreeType 2.9-2.10.1 issue to their FreeType package.

[m4rc1e]

Thank you so much Ben!

@nyshadhr9 Is it possible for us to serve static fonts for the affected distros?

[bungeman]

Both debian and Ubuntu now have FreeType 2.9.1-4 for their newest packages, which contain the fix for this FreeType issue. The fix for the FreeType 2.6.3 issue is pending in debian.

[davelab6]

@bungeman when can this issue be closed? :)

[bungeman]

Other than hounding distros to pick up the fixes, I suppose there isn't much left to track here. It looks like Ubuntu disco still has the issue, but every other current release looks ok. On debian, sid and bullseye should have the fix, but buster and stretch still have the issue (they may get fixed in the future as the changes roll through). Chrome 76 and later should now be fixed on all platforms. So I suppose this issue should be closed when either enough platforms have the fix that you're ok with it or you figure out how to detect the issue and route around it.