[cssom-view] {element,elements,nodes}FromPoint but without restricting to the viewport clip?

[emilio]

Current spec is: https://drafts.csswg.org/cssom-view/#extensions-to-the-document-interface

There was this very old request from an Apple engineer here: https://lists.w3.org/Archives/Public/www-style/2012Oct/0683.html

This week I was asked for some help with performance of an add-on, which was doing something particularly expensive to see if an element was visible. I suggested that, given a lot of the bottleneck was walking the DOM since it's an extension and it has security wrappers, using elementFromPoint-like APIs should speed up stuff, and indeed it did quite a lot. See mozilla/fathom#91. But that misses the element if it's outside of the viewport.

We have similar non-web-exposed APIs that avoid this clip for various Firefox features, and apparently the best work-around this problem if you're an author resides in using window.scrollTo and such. See for example: https://stackoverflow.com/questions/19715620/javascript-get-element-at-point-outside-viewport

The original thread suggests new APIs altogether. I think it'd be better to add some dictionary argument to the existing APIs, something like:

dictionary FromPointOptions {
  boolean ignoreViewportClip = false;
}

Or such?

While working on this, I could also try to do a bit of spec maintenance, since there's multiple stuff we've resolved on that hasn't been incorporated to the spec. If only, moving these to DocumentOrShadowRoot, the node/nodesFromPoint APIs discussed in here.

This also fits nicely for the "include display: contents parents" option that people discussed in #556.

Does an API like this make sense to y'all? :)

/cc @tabatkins @zcorpan @smfr

[emilio]

Note that there are further issues with these APIs that we should also put in a spec, like #1378.

[tabatkins]

Without further context, I'm fine with the idea. It does mean you have to run full layout outside of the viewport, which is something we're trying to allow skipping in some circumstances (a sufficiently contain'd element, for example).

Can you explain how this helps the cited use-case, tho? I don't quite understand how eFP() working outside the viewport would help them determine whether an element is visible.

[emilio]

Basically, what that page is doing (AIUI) is scanning the DOM for potentially interesting elements, and needs to determine whether they're "visible". getElementsFromPoint is a nice way to ensure that the element is displayed and potentially-visible.

Another use-case (which I read yesterday but forgot the source) is checking whether two things intersect at an arbitrary point. elementsFromPoint() at that point works, but only if inside the viewport.

[tabatkins]

For the first, presumably returning null if something's not in the viewport is actually a good behavior to tell if something is displayed? It's not ideal, as part of the element might be in the viewport, just not the point you specified, but that's what IntersectionObserver is for, right?

For your second, that's a v good use-case for this.

[chrishtr]

Just like Firefox, Chromium has non-web-exposed use-cases for skipping the clip. Our use cases are:

• Visually positioning carets and selection ranges
• Detecting colors underneath elements in DevTools
• IntersectionObserver occlusion testing

I'd like to understand better the required web-exposed use cases before we agree to this API, because as Tab mentioned there are some performance downsides, and this API could end up being a new footgun in practice.

In particular, for the use case you mentioned @emilio of measuring on-screen-ness: is IntersectionObserver awkward for that use-case?

Regarding detecting whether two elements intersect at an arbitrary point, or overlap that point: I'm not sure what that's for?

[emilio]

Here's the intersection code that I thought of: https://stackoverflow.com/questions/9011668/get-element-at-point-in-entire-page-even-if-its-not-visible

Upon re-reading it I think what that code is trying to do would be better done by something like flex or grid, but I haven't thought out too much how, it seems it's sort of like the masonry layout stuff.

Anyhow... The usecase that prompted me to propose it was some investigation in mozilla/fathom#91. But I've seen similar code elsewhere in sites like Facebook and such (see https://bugzilla.mozilla.org/show_bug.cgi?id=1381071#c1 for example).

The facebook code is:

function isVisible(element: HTMLElement): boolean {
  // Look upward in the DOM until we either reach the document or
  // find something hidden.
  let elem = element;
  while (
    elem &&
    elem !== document &&
    Style.get(elem, 'visibility') != 'hidden' &&
    Style.get(elem, 'display') != 'none') {
    elem = elem.parentNode;
  }

  // If we reached the document, the element is visible.
  return elem === document;
}

The code in the Mozilla add-on was similar too:

export function isVisible(fnodeOrElement) {
    const element = toDomElement(fnodeOrElement);
    for (const ancestor of ancestors(element)) {
        const style = getComputedStyle(ancestor);
        if (style.visibility === 'hidden' ||
            style.display === 'none' ||
            style.opacity === '0' ||
            style.width === '0' ||
            style.height === '0') {
            return false;
        }
    }
    return true;
}

(I skipped some branch which was doing some viewport stuff because it's not clear what it's trying to do, I think it's trying to check elements scrolled out of view towards the top, or something).

The question that code is trying to answer is whether a given node is "visible", for some definition of visible. Both functions are bogus in multiple ways:

• They don't work with anything that looks like shadow DOM in any way.
• They assume visibility works differently than how it works (visibility: visible inside visibility: hidden is definitely visible).
• The style.width and style.height checks in the add-on code is wrong (should be 0px instead), they're meaningless is overflow is visible, etc...
• They don't handle other ways to make an element "hidden".

The add-on use-case is even worse because walking the DOM from an add-on is slower due to security wrappers / proxies.

In the add-on use-case at least, the complaint against elementsFromPoint() was that it returned false for out-of-viewport elements. But using it actually improved performance compared to all the DOM walking and layout querying (mostly because of the reduced number of DOM calls, actually).

It seems to me like a good improvement over those two functions but, that being said, chances are that a more thought-out proposal may want a bit more flexibility. For example, people may want to ignore all scroller clips, or maybe only those with user-scrollable overflow (that is, don't ignore overflow: hidden clips but ignore overflow: scroll clips)... In any case ignoring the viewport clip is a pretty good improvement over the situation now.

The Facebook code doesn't need to deal with all of the web, but libraries / frameworks and extensions do, this could allow them to write actually correct code :)

[chrishtr]

I 100% agree that developers should not write their own code to detect visibility, as they will inevitably get it wrong, and the performance will be much worse.

Why can't they use IntersectionObserver for those though? (The "v1" spec doesn't check for visibility, but the "v2" one does.)

[emilio]

Ah, I was not familiar at all with IOv2. It seems to me that it'd miss a bunch of hard cases. It seems that IOv2 has a very concrete use-case that may not be the use-case everybody wants or needs.

Also it's unclear to me if the "completely unoccluded" part of the visibility algorithm includes the viewport or not. I assume per your description it does not include the viewport clip, right?

If so I guess it does make sense to use IOv2 for some of these cases. Probably there are use-cases that still want or need a fast and sync answer for this, maybe...

Also, IntersectionObserver only allows to observe elements, so this probably won't fly for the Shadow DOM / display: contents use cases that prompted the introduction of nodesFromPoint in #556

[biancadanforth]

Why can't they use IntersectionObserver for those though? (The "v1" spec doesn't check for visibility, but the "v2" one does.)

I wanted to respond here as one of the motivating use cases for Emilio bringing this issue up for discussion. I did try using the Intersection Observer API (I believe v2 based on your description). I saw about a 20% performance improvement compared to the 80% improvement using elementsFromPoint described in the same thread.

My use case differs in a couple of key ways from what seems to be the optimal use case for IntersectionObserver:

1. We want to check visibility for every single element on the page, not just a handful of say, ads. My sample page had upwards of 6000 elements, which means 6000 observers.
2. We only want to check once, not monitor changes over time. It seems like overkill to walk the DOM and attach an observer to each element only to remove it after the first event.