[css-text-3] Discarding Line Breaks Adjacent to Ambiguous Characters

[fantasai]

The discussion in #337 has veered off in a wide variety of directions, but @hax originally filed the issue to bring up the question of "ambiguous" characters, i.e. those which are commonly used both within and outside Chinese and Japanese context:

https://drafts.csswg.org/css-text-3/#line-break-transform

Otherwise, if the East Asian Width property [UAX11] of both the character before and after the line feed is F, W, or H (not A), and neither side is Hangul, then the segment break is removed.

As this rule, common use cases of quotation marks in Chinese

简体中文的
“引号”
两边不应该有空格。

will have unexpected spaces, because quotation marks are A.

Ideally, we should consider the language information of the context. If the context is East Asian language, A should be treat as W. Even in the unknown language context, if any side of the line feed is A and other side is F, W or H, the segment break should also be removed.

We decided to switch to a Unicode Block listing instead of relying on the East Asian Width property (in particular due to some backwards-incompatible changes on Unicode's side). The current draft does not have a concept of ambiguous characters: all characters are strong "discard" or "don't discard", with discarding behavior requiring both sides of the line break to be "discard".

We might want to consider classifying some characters as "ambiguous", particularly symbols and maybe also the few common punctuation marks used in Chinese (double quotes, specifically). These could defer to the character on the other side, and if both are ambiguous, default to "don't discard".

Do we want to do this? If so, should it be language-dependent or universal?

[macnmm]

My recent thoughts in adapting our J-specific layout to be more universal, I am finding it easier to support several aspects of conflicting conventions by thinking of them as language-specific and introducing the idea of a mode, often at the paragraph level. Embox-based line heights and leading versus ascent-descent; space character widths between Korean words versus in English; how to treat justification between Latin and CJK for JISx4051 versus for English; and these ambiguous Unicode characters, where the font (default glyph) used or the paragraph convention preferred, causes a conflict in desired behavior. There are two dimensions of this conflict -- I am not convinced that we only need to look at the code points around these characters to derive desired behavior. I suspect that in one paragraph mode there is one default for these characters in context of their surroundings, but that in another mode the answer is different, even with the same surrounding characters.
Even though shaping can be done for every language in the same way in the line, the way that line is arranged with spacing adjustments and vertical placement for leading or gridding could change if one decides the containing paragraph should follow Japanese conventions overall versus some other language's conventions. I think they need to be independent. However I could be wrong in that the need I see could be an artifact of limited technology giving rise to the Japan-specific convention for mixed text that differs from English conventions for mixed text.

[kojiishi]

Most fonts I know have Latin double quotes by default, so I think the space is desired. It becomes fullwidth only when author sets fwid OpenType feature. Maybe Chinese fonts are different, but I'm hesitate to build rules into the spec based on what current major fonts have.

[fantasai]

After discussion with @kojiishi, particularly about cases where the quotation mark is used adjacent to spaces intentionally, I agree that we should not make double quotes Ambiguous here.

Question still stands about whether we should do this with Symbols generally, such as Emoji and and characters in the Enclosed Ideographics block.

[r12a]

Which aspect of ambiguity are we talking about here?
a. a space may be added where not wanted, eg. 支持W3C实现“\尽展
b. a space may be removed when it should stay, eg. 空格字符“ \”不可见

Whichever applies, i suspect that my comments at #4992 (comment) about using common-sense or internationalised apps for line breaking may apply.

But Chinese & Japanese use a quite a lot of characters that are not in the discard blocks, not just quote marks. I've actually been trying to make lists of what characters are used by what language, and fwiw currently i have the following from non-discard blocks:

Chinese
Basic Latin | 21 | !​"​#​%​&​(​)​*​-​.​/​:​;​?​@​[​\​]​_​{​}
General Punctuation | 25 | ‐​‑​–​—​―​‖​‘​’​“​”​†​‡​‥​…​‧​‰​′​″​‵​※​‾‼⁇⁈⁉
Latin-1 Supplement | 2 | §​·

Japanese
Basic Latin | 20 | !​"​#​%​&​(​)​*​-​.​/​:​;​?​@​[​\​]​{​}
General Punctuation | 20 | ‐​—​―​‖​‘​’​“​”​†​‡​‥​…​‰​′​″​※‼⁇⁈⁉
Latin-1 Supplement | 2 | §​¶

[r12a]

fwiw, just added the following to my lists above: ‼⁇⁈⁉ (sorry about the emojification)

[kidayasuo]

@fantasai I know it is a CSS novice question but may I? Even with English, there are cases where you would not want a space inserted between two characters. Examples include between quotation marks or parentheses and its contents, around EM DASH depending on the style, before a colon, etc. Does the rule has an expectation that authors would not fold a line at certain places, i.e. where they do not want a space inserted?

[kidayasuo]

a more interesting case in English is a semi compound word connected by a hyphen, e.g. "well-known". As they look two separate word one might insert a line break after the hyphen, exactly like hyphenated words. If the Segment Break Translation Rule inserts a space after the hyphen it would be different from the naive user's expectation.

What I am trying to understand is what we can naturally expect from the segment break transformation rules.