[css-highlight-api] Figure out how highlights are exposed to the accessibility tree

[frivoal]

@atanassov recently reminded us that figure out how highlights are exposed to the accessibility tree. This had been know for a while, but we didn't have an issue I could find, so here's one.

[frivoal]

To sighted users, the highlights are only visible through the way they are styled. There is no other way to know what they're for. Maybe the same should be true of the highlights? Getting the highlight itself described through some text sounds like it might be more convenient, but I'm not sure where we'd pull that description from, how to deal with multiple languages, or the potential discrepancies between the descriptive text and what the highlight is actually used for.

[astearns]

And whatever gets decided here should likely match what is decided for #6417

[frivoal]

How are things like these exposed to the accessibility tree today?

• ::spelling-error
• ::grammar-error
• ::target-text

We don't necessarily have to do the same, but it'd be good to start with an understanding of how these are handled.

[ffiori]

#4601 seems to have some related remarks.

[dlibby-]

UserAgent detected spelling and grammar errors (i.e. ::spelling-error and ::grammar-error) in Chromium do have a mapping to various AT APIs. These are serialized as part of the accessibility tree and as far as I can tell, this is not part of any of the Aria-related standards. For example on UIA on Windows spelling errors are exposed as a text range with an AnnotationType_SpellingError. Not sure how this would fit into the spec, but perhaps is a path forward (following the UIA example, there is an AnnotationType_Highlight).

For completeness, spelling and grammar errors that are computed by the webpage can be annotated with aria-invalid. The corresponding concept for highlights would probably be the mark role (see https://w3c.github.io/aria/). However, these require additional DOM nodes, which is counter to the purpose of the HighlightAPI.

[dandclark]

I agree with @dlibby- that for the question of how to expose the highlight ranges to ATs we should be able to follow the lead of how ::spelling-error and ::grammar-error are exposed in Chromium, at least in part.

For expressing the semantics of the highlight ranges, the most flexible option seems to be to add a title string attribute to Highlight, which would be exposed to ATs. However, as pointed out in #4601 (comment), these strings wouldn't be localized. This approach also depends on the author to provide a string with the correct level of descriptiveness.

An alternative I'd like to consider is adding an enum to Highlight that captures the common use cases we expect for this API. These would be exposed as appropriate in different accessibility platforms, e.g. in UIA the most closely matching AnnotationType would be used.

enum HighlightType {
 "highlight",
 "spellingerror", // Exposed to ATs in the same way as ::spelling-error
 "grammarerror", // Exposed to ATs in the same way as ::grammar-error
 "textmatch", // e.g. for custom find-on-page results
// ...and any others for the common use cases we can think of.
};

[Exposed=Window]
interface Highlight {
  constructor(AbstractRange... initialRanges);
  setlike<AbstractRange>;
  attribute long priority;
  attribute HighlightType type; // Exposed to ATs. Default is "highlight".
};

The benefit of this is that we're not relying on author-specified strings, so it's easier for developers and it solves the issue of localization. An enumerated set of types could also be easier to expose to ATs.

The disadvantage is that use cases we don't anticipate won't have any good choices so authors will have to fall back to the generic "highlight". I don't think we'd be locked into the initial set of HighlightTypes though; we could probably add more if additional common uses cases arise.

[michael-n-cooper]

Tagging for a11y-tracker, since the dup that was closed was so tagged.

[dandclark]

Another thing I'll add here is that an advantage of an enum over a string is that when OSs have some first-class representation for any of the use cases in the enum, user agents could plug into them directly. For example, custom spell check implementations using Highlight API could be represented to accessibility tools in the exact same way as ranges marked by the native spell-checker. In UIA this might AnnotationType_SpellingError, or NSAccessibilityMarkedMisspelledTextAttribute on Mac. This would be a more consistent experience for AT users and would save the author the trouble of localizing a "spelling error" string.

It's also feasible that in the future ATs could become more powerful and offer first-class representations of more of these use cases, and it would be great if authors can provide metadata that provides a better experience in those cases.

If needed, in an L2 of the spec a string attribute could be added to Highlight for use cases that fall outside of those we've expressed in the enum. But for use cases that become common and have support in ATs across OSs, it seems more powerful to enable these as additions to the enum. Allowing arbitrary strings could miss opportunities to line up with things that OSs recognize.

[dandclark]

There's been some delay on agenda space opening up to discuss this during CSSWG calls, so I wanted try again to see if we could make some progress here in the issue.

Considering the enum approach considered here and looking more at the current capabilities of UIA's Annotation Type Identifiers and Mac's Attributed String Keys, I'd like to propose starting with an enum with just these:

enum HighlightType {
 // Exposed with AnnotationType_Highlight for UIA, accessibilityCustomText for Mac
 "highlight",
 // Exposed with AnnotationType_SpellingError for UIA, NSAccessibilityMisspelledTextAttribute for Mac
 "spellingerror",
 // Exposed with AnnotationType_GrammarError for UIA, NSAccessibilityMisspelledTextAttribute for Mac
 "grammarerror"
 // More can be added as use cases emerge and accessibility platforms add more text semantic types
};

[Exposed=Window]
interface Highlight {
  constructor(AbstractRange... initialRanges);
  setlike<AbstractRange>;
  attribute long priority;
  attribute HighlightType type; // Exposed to ATs. Default is "highlight".
};

This approach lets custom spellcheck achieve the same experience with accessibility tools as native spellcheck for UIA and Mac today, and leaves room to add more entries to the enum as use cases an support emerge. A custom string could be added in an L2 if the need for that level of flexibility emerges.

@frivoal @sanketj @dlibby- @lilles, any thoughts on this approach?