Some of the extended Unicode characters are rendered with incorrect size and placement

[vblazhkun]

Environment

Windows build number: 10.0.18363.0
Windows Terminal version (if applicable): 1.1.1671.0

Any other software?

- WSL Ubuntu 20.04 LTS

- The font is PragmataPro Mono Liga v.0.828.

Steps to reproduce

Some of the extended Unicode characters are rendered with incorrect size and placement (i.e. the lower-left corner of the cell). As an example the char ✽ (0xE29CBD).

Expected behavior

Actual behavior

[DHowett]

According to Unicode 13.0, U+273D HEAVY TEARDROP-SPOKED ASTERISK is ambiguous width, non-emoji, non-emoji-presentation -- that means that it is narrow, and will always be narrow, in compliant terminals. If your font specifies that it is larger than one monospaced cell in width, it is time to fix your font. 😄

[DHowett]

(as an example, this behaviour is not recapitulated in the font I have set right now:

)

[vblazhkun]

According to Unicode 13.0, U+273D HEAVY TEARDROP-SPOKED ASTERISK is ambiguous width, non-emoji, non-emoji-presentation -- that means that it is narrow, and will always be narrow, in compliant terminals. If your font specifies that it is larger than one monospaced cell in width, it is time to fix your font. 😄

It occupies exactly one cell (i.e. it is narrow). But, the Notepad and other terminals render it correctly (within a single cell as demonstrated on the initial picture) opposing to the WT.

[vblazhkun]

@DHowett Actually, even the earlier releases of the WT demonstrated the correct rendering behavior. I believe it is a bug.

[vblazhkun]

A quick snip from ConEmu:

[DHowett]

fair enough

[lhecker]

This appears to be fixed with AtlasEngine:

Please let me know if this issue still appears for you in Windows Terminal Preview 1.16.

[vblazhkun]

This appears to be fixed with AtlasEngine:

Please let me know if this issue still appears for you in Windows Terminal Preview 1.16.

The original issue is fixed however there is a new one. The given character U+273D HEAVY TEARDROP-SPOKED ASTERISK is always rendered with white foreground color despite the WT color scheme or surrounding ANSI escape sequences (such is actually visible on your screenshot as well). I will open a new issue describing that.

[lhecker]

Thanks! Coincidentally I was just talking about this exact issue elsewhere like 5 minutes ago. 😅
It is caused by this code:

terminal/src/renderer/atlas/AtlasEngine.cpp

Lines 1506 to 1530 in 71fc4b1

	// Do fonts exist *in practice* which contain both colored and uncolored glyphs? I'm pretty sure...
	// However doing it properly means using either of:
	// * IDWriteFactory2::TranslateColorGlyphRun
	// * IDWriteFactory4::TranslateColorGlyphRun
	// * IDWriteFontFace4::GetGlyphImageData
	//
	// For the first two I wonder how one is supposed to restitch the 27 required parameters from a
	// partial (!) text range returned by GetGlyphs(). Our caller breaks the GetGlyphs() result up
	// into runs of characters up until the first canBreakShapingAfter == true after all.
	// There's no documentation for it and I'm sure I'd mess it up.
	// For very obvious reasons I didn't want to write this code.
	//
	// IDWriteFontFace4::GetGlyphImageData seems like the best approach and DirectWrite uses the
	// same code that GetGlyphImageData uses to implement TranslateColorGlyphRun (well partially).
	// However, it's a heavy operation and parses the font file on disk on every call (TranslateColorGlyphRun doesn't).
	// For less obvious reasons I didn't want to write this code either.
	//
	// So this is a job for future me/someone.
	// Bonus points for doing it without impacting performance.
	auto flags = CellFlags::None;
	if (fontFace)
	{
	const auto fontFace2 = wil::try_com_query<IDWriteFontFace2>(fontFace);
	WI_SetFlagIf(flags, CellFlags::ColoredGlyph, fontFace2 && fontFace2->IsColorFont());
	}

Thank you for opening another issue! I'll try to fix it... somehow. 🥲

[carpet92]

In the previous version of terminal the icons definitely were larger (more proportional) than now:

amro.omp.json theme for oh-my-posh

WT 1.16.2523.0

[lhecker]

@Zx-EvM That problem is specific to Nerd Fonts. Other kinds of Emojis didn't work in the previous version either. I'm working on a fix over here: #6864 (comment)

[vblazhkun]

This appears to be fixed with AtlasEngine:

Please let me know if this issue still appears for you in Windows Terminal Preview 1.16.

It seems, there is a regression with the wide characters rendering (WTP Version: 1.22.3232.0). Right side of the double-width character collides with the next one. Attaching an example (tested with both Grapheme clusters and wcwidth):

[DHowett]

Indeed we have determined that -- for proper support of all languages -- glyphs that extend outside the boundaries of their cells must be preserved and not shrunk.

That's a very nice font you're using. The folks who made it might be willing to add a right-sized glyph for that character. :)

[vblazhkun]

Thank you for the prompt response! I am not really sure what you mean by "making right-sized glyph" as the given character occupies 2 cells exactly and fits their boundaries:

Also, Word renders it correctly:

[DHowett]

According to Unicode 13.0, U+273D HEAVY TEARDROP-SPOKED ASTERISK is ambiguous width, non-emoji, non-emoji-presentation -- that means that it is narrow, and will always be narrow, in compliant terminals. If your font specifies that it is larger than one monospaced cell in width, it is time to fix your font. 😄

What I mean is this. The glyph is effectively required in all terminal emulators to physically occupy one cell, even if the rendering shows it as taking more than one cell.

We've learned a lot since this issue was originally created, but the result is the same.

The glyph can be larger than a single cell (and if the font says it must be, it must be, for the aforementioned language reasons) but the text in the cells next to it also have to go in their own cells too.

A terminal is strictly grid-based.
If the fancy star occupies cell 0,0 and spills over into cell 0,1 it can't just push the character from cell 0,1 over to the right. Those cells are used for exact cursor positioning, and applications running inside a terminal can target them specifically.

A word processor is designed for proportional text, and has no requirements about cells - so Word and Notepad and every other non-grid-based editor will just nudge the characters over based on the physical rendered size of the glyph.

We just don't have that luxury, because CLI applications cannot handle that.

[vblazhkun]

I see. Would it be possible to implement some customizable (per font) mapping table assigning ambiguous-width characters to desired cell widths, so the terminal handles the positioning correctly? That is: force ✽ -- 2 cells wide, ❄ -- 3 cells wide, etc. without altering the actual rendering. This way the terminal knows the actual character width and does not skew the output thinking the cell is not occupied.

As the characters are ambiguous-width per the Unicode specification it is not fixable with the font itself as another application might not be cell-based and will render the output in an ugly fashion.

There is an option to use some escape sequences for such purpose as an alternative.

I have just found that mintty tries implementing something similar: ambiguous-width-setting.

UPD: Some more information on how mlterm handles that: issuecomment-1684850852.