Consider Canonicalization of language tags in :lang() selector maching

[frivoal]

Cantonese may be described with any of the following language tags:

• zh-HK (most common, legacy)
• zh-yue (also legacy)
• yue (current most appropriate value)

Even though these refer to the same language, :lang(yue) would only match the last one.

RFC 5646 section 4.5 together with the IANA Language Subtag Registry define canonicalization and mappings that would allow to match the last two (by extlang form). RFC4647 section 3.2 says we should use this.

The last paragraph of that same section 3.2 also says we may also want to consider mappings like zh-HK to zh-yue, and that too could seem appropriate, but as far as I know there’s no equivalent to IANA Language Subtag registry for such mappings.

While it is possible for authors to deal with this themself by using :lang(yue, zh-yue, zh-HK), to the extend we can automate this, they shouldn't have to deal with this.

(I'm taking Cantonese as an example, but the same can be said about other languages)

[r12a]

Section 4.5 talks about canonicalising items that are marked in the registry as deprecated (eg. grandfathered tags) or that contain extlang subtags (eg. zh-yue -> yue). These are things that can be determined automatically by using the information in the registry. But it doesn't include zh-HK.

Section 3.2 mentions zh-CN as possible equivalent to zh-Hans, but as i read this as a configuration option offered to the user, not an assumption that they meant the same thing in all contexts (which is what canonicalisation means) or that the author intended one while writing the other. And I think assuming that zh-HK means yue is more of a stretch than the zh-CN/zh-Hans assumption.

I think we need to be careful about making inappropriate assumptions on the behalf of the content author. zh-HK may be used to mean Mandarin chinese, but with traditional script, or possibly written text that includes the few additional characters that are used in Hong Kong - in fact, i thought that the predominant legacy usage for zh-HK arose from an early workaround before zh-Hant existed - not for yue. Of course, it could also be used for Cantonese. But it's possible that it's even used for Hakka or Minnan Chinese as spoken in Hong Kong. It depends, really, on what the author needed and intended. It may also depend on whether it is used for a spoken or written phrase.

So i think it makes sense to assume equivalence for language tags that can be automatically paired using information in the registry, but not for things like zh-HK.