[cssom-view] Add segments property to visual viewport (#9237) #9285

darktears · 2023-08-31T19:07:58Z

[cssom-view] Add segments property to visual viewport (#9237)

This property exposes in JS what is already exposed to CSS via the env() viewport segment variables. TAG review here.

bokand

Not an expert on segments but the text lgtm generally.

bokand · 2023-09-11T19:32:35Z

cssom-view-1/Overview.bs

+1. If the <a>visual viewport</a>'s <a for=visualviewport>associated document</a> is not <a>fully active</a>, return 0.
+2. Returns null if called from within an iframe context and abort these steps.
+3. Returns null if there is only a single viewport segment and abort these steps.
+4. Otherwise, return the <a>visual viewport</a>'s segments array.


Is there a definition for segments array we could link to?

I'm not sure what you mean but line 1861 defines what the array is. If you mean by "linking" like jumping into the spec I think I can't use <dfn attribute for=VisualViewport>segments</dfn> again without getting a warning.

Ah I see. You could link to it with but that seems rather self-referential, I was more thinking about some spec text that explains how these DOMRects are defined but I'm not sure that exists today.

We could link this?

That defines DOMRect interface; I mean: what values do we put to the x, y, width, height for each rect in segments? I know we say they "represent the dimensions of each existing viewport segment" but that's a bit hand wavy. i.e. what is a "viewport segment"? Is it affected by pinch-zoom? Overlay keyboards? etc.

Perhaps we could link "viewport segment" to https://drafts.csswg.org/css-env-1/#viewport-segments? I think it'd explain that this is the same thing as the env variables and if we want to make their definition more precise in the future (w.r.t my questions above) we could then do so in one place for both visualViewport.segments and the env variables.

darktears · 2023-09-14T15:38:48Z

@bokand what do you think about the latest change?

bokand · 2023-09-14T15:49:14Z

lgtm!

darktears · 2023-09-14T15:50:14Z

Feel free to merge. I'm not sure if additional reviews are required.

bokand · 2023-09-14T15:52:56Z

Hmm, I'm not an editor so not sure if that's ok - perhaps @smfr or @emilio could take a look?

darktears · 2023-09-14T15:54:12Z

Yes, I think we should wait their feedback :)

darktears · 2023-10-27T14:53:24Z

@smfr @emilio any comments?

smfr · 2023-10-28T18:41:17Z

I have a few thoughts, mainly about potential for user fingerprinting based on how they generally use a foldable/multi-screen device.

First, on a device that's foldable but with no hinge gap (i.e. continuous foldable screen), does the number of segments change when the screen goes between some folded state and unfolded flat? At what fold angle does this occur? If so, this might be used to identify users based on their habits of using the device fully unfolded or partially folded. This may also be used to estimate whether the user is paying full attention to the screen (fully unfolded) or is using it in a more glanceable fashion (partially folded), which may provide a signal to malicious content about when it can put things on the screen. Combined with device orientation data, this could be used to obscure activity on the less visible screen segment.

Second, there are potentially two additional means by which screen segment data could leak per-user fingerprintable data. If an OS allows the user to position a window freely in a way that spans screens, the segments API is way to leak the window position (the segment geometry would change based on window position). This is akin to desktop browsers exposing window.screenX/screenY which can be very user-specific. In addition, even with non-draggable windows but modifiable browser UI including toolbars, the segments would leak information about the UI/toolbar geometry.

darktears · 2023-10-30T12:51:29Z

The number of segments changes when the screen goes between the folded state and unfolded flat indeed. That's the essence of the API in the first place which is the same goal than the CSS MQs added in MQ Level 5. Here we're just basically providing the same functionality to JavaScript developers where CSS isn't a good fit (e.g. Canvas based content).

I think we need to take a step back here. User is expected to pay full attention when the device is in a folded state. The entire idea of the form factor is to provide bigger screen estate for content when the device is half folded and the keyboard detached. Even on smaller foldable form factors like smartphones typically content author will display a split ux. Have a look here to see various pictures of devices in action. There is also plenty of pictures online showcasing the Samsung foldables with their split UX ideas.

Regarding the potential fingerprinting, on Chromium browser segments are only returned when the browser window is in fullscreen or maximized, this is an implementation choice which we can decide to standardize or not. We did not hear from content authors the need to get segments when the window isn't maximized and floating around. As a clarification the browser window is not resized when changing state (e.g. folded to unfolded) but could be if the keyboard is docked for example.

I want to emphasize that the segments are defined in client coordinates (so adjusted to zoom factor and so forth) therefore relative to the viewport. I'm not sure how much information they would "leak" provided that say top segment + bottom segment will be equivalent to the inner height/width of the viewport.

Could you use these APIs to fingerprint that it's a folded device? Yes sure but again today you can do that by checking the window dimensions because they have not so common sizes (including for example Android where the dimensions are off for a tablet and a typical phone). Can you use these APIs to fingerprint a specific foldable model? Likely no because many of them share the same panel dimensions/sizes (phone or not) but again even if one had an exotic panel, they add nothing more than what I could already derive from the viewport size.

Thanks for your comments. I'll be happy to clarify more if you need so.

emilio · 2023-10-30T19:29:25Z

cssom-view-1/Overview.bs

@@ -1806,6 +1806,8 @@ interface VisualViewport : EventTarget {

  readonly attribute double scale;

+  readonly attribute FrozenArray&lt;DOMRect> segments;


I don't think you need the <?

I don't think a FrozenArray can be used tho, right? A FrozenArray is supposed to be immutable (https://webidl.spec.whatwg.org/#idl-frozen-array), but this is mutable right?

I think it should just be a sequence?

I don't think you need the <?

I get an error otherwise.

I don't think a FrozenArray can be used tho, right? A FrozenArray is supposed to be immutable (https://webidl.spec.whatwg.org/#idl-frozen-array), but this is mutable right?

It's frozen because it's not changing, just like any other property of visualviewport. You're expected to listen for some events to query the new state (e.g. resize). If just the posture is changed then Device Posture API has an event listener specifically for that. Both APIs are complimentary.

It's frozen because it's not changing, just like any other property of visualviewport. You're expected to listen for some events to query the new state (e.g. resize). If just the posture is changed then Device Posture API has an event listener specifically for that. Both APIs are complimentary.

Well but that means that they do change (when the event fires) right? or what am I missing?

Here is a concrete example: initial page load and the device is used flat (or a laptop posture with the keyboard docked) the segments property will be 0 meaning an array of 0 length.

If I change the posture to folded (or detach the keyboard) then the events will fire and the segments property will have an array of 2 segments representing the two sides beside the fold/hinge.

Just like the other properties of the visual viewport segments they represent the given state at the time window.visualViewport was queried.

Right, I get that, the point is that my understanding is that FrozenArrays aren't supposed to change, ever, not even in reactions to an event. @domenic is that right?

FrozenArray properties can change. The contents of the array cannot, but a new array can start being returned.

Note that it is invalid for attributes to be sequences; that is part of why FrozenArray exists.

emilio · 2023-10-30T19:37:58Z

cssom-view-1/Overview.bs

+
+Note: Based on the data returned for each viewport segment, developers will be able to infer the number of hinges available as well as the hinge orientation.
+The browser window can be moved/resized such that the number of segments and/or their dimensions change.
+In these cases, resize events will fire at which point authors should re-query this property, as the returned FrozenArray is just a snapshot of the current state.


It'd be good to note here that these are returned in un-zoomed CSS pixels. It's a bit weird that these are part of the VisualViewport API even though they don't have anything to do with it right? In particular, they don't change when zooming and panning, right?

Or do those environment variables change when zooming and panning? That seems unfortunate if so.

They don't change when zooming or panning. But unless I don't understand your point, I'm not sure why they have "anything to do" with the viewport? For example, on a symmetrical folding device which is in a folded state, they're helping developers to implement say a split UX/dual pane view in their app/websites which is basically dividing the viewport in half logically and they need the segments to do so. But again, maybe I misinterpreted your comment.

@emilio : Take a look here. You can see in the pictures what the API is trying to achieve in JS, similar to what the CSS API is doing.

fantasai

Adding new features (or substantially changing existing ones) requires CSSWG approval, not just editor approval... it doesn't seem like #9237 has that?

darktears · 2023-11-27T13:54:21Z

Adding new features (or substantially changing existing ones) requires CSSWG approval, not just editor approval... it doesn't seem like #9237 has that?

I believe it doesn't have approval (at least I didn't request but I'm not sure what MSFT did in the past). The CSS equivalent (MQs) is already in https://drafts.csswg.org/css-env-1/#viewport-segments and MQ Level 5 (https://www.w3.org/TR/mediaqueries-5/#mf-horizontal-viewport-segments). It's essentially the exact same feature, just at a JS level. Anyway, how do I request such approval?

aykutbulut · 2024-03-27T15:49:09Z

Privacy section in the explainer indicates that segments will return null when called from within an iframe and explains how this prevents using the API as a fingerprinting vector from cross origin iframes.

A concern I have is accessing the segments through CSS media queries. Having CSS API in iframes might enable this type of fingerprinting. Is disabling CSS media queries for the segments feasible?

darktears · 2024-03-27T16:10:00Z

Privacy section in the explainer indicates that segments will return null when called from within an iframe and explains how this prevents using the API as a fingerprinting vector from cross origin iframes.

A concern I have is accessing the segments through CSS media queries. Having CSS API in iframes might enable this type of fingerprinting. Is disabling CSS media queries for the segments feasible?

Do we have precedence in doing that? I couldn't find anything in CSS specifications where certain CSS features are restricted to iframes, especially in the MQ/CSS env context but the domain is vast so I could have been missing something.

But again, the real question is whether the fingerprinting is really that big of an issue if segments are just subdivisions of the total viewport size. Foldables already have exotic viewport size when unfolded so they can be identified already but as more devices are in the market, targeting a specific device is becoming harder as they share panel dimensions so segments definition as well.

You could use this to detect how the device is being used (folded vs not) but the Device Posture API already reveal that. One could argue that in some way it is like the pointer MQ which is dynamic and exposing whether a device with detachable keyboard has it connected or not, also revealing how the device is being used.

Yes, it does add a bit of entropy but nothing that couldn't be derived from the viewport size today (or even combining with other APIs) but I'm open to discussing this further.

aykutbulut · 2024-04-03T23:01:02Z

I am not aware of such a precedence. I agree with your arguments.

dbaron · 2024-06-11T12:50:43Z

cssom-view-1/Overview.bs

@@ -1981,6 +1996,9 @@ Changes {#changes}
 This section documents some of the changes between publications of this specification. This section is not exhaustive. Bug fixes and editorial changes are
 generally not listed.

+<h3 id='changes-from-2022-07-07' class=no-num>Changes From 14 September 2023</h3>


The ID here should match the title.

(It's also not clear to me why there are so many changes sections in this spec. I'd normally expect one changes section for each new publication to https://www.w3.org/TR/cssom-view-1/ .)

darktears · 2024-06-11T20:15:55Z

Close this PR per CSS WG resolution where we want the segments property to belong on a new field in window.

darktears requested a review from bokand September 11, 2023 16:34

bokand approved these changes Sep 11, 2023

View reviewed changes

[cssom-view] Add segments property to visual viewport (w3c#9237)

b675ccb

darktears force-pushed the viewport-segments branch from e366c47 to b675ccb Compare September 14, 2023 15:38

darktears requested review from smfr and emilio September 14, 2023 15:54

darktears mentioned this pull request Sep 14, 2023

Viewport Segment API mozilla/standards-positions#883

Open

emilio reviewed Oct 30, 2023

View reviewed changes

fantasai reviewed Nov 20, 2023

View reviewed changes

fantasai marked this pull request as draft November 20, 2023 22:02

darktears mentioned this pull request Mar 6, 2024

Viewport Segments CSS and JS API WebKit/standards-positions#327

Open

dbaron reviewed Jun 11, 2024

View reviewed changes

css-meeting-bot mentioned this pull request Jun 11, 2024

[css-viewport-1] Bring the segments property into viewport #9237

Closed

darktears closed this Jun 11, 2024

		@@ -1806,6 +1806,8 @@ interface VisualViewport : EventTarget {

		readonly attribute double scale;

		readonly attribute FrozenArray<DOMRect> segments;

[cssom-view] Add segments property to visual viewport (#9237) #9285

[cssom-view] Add segments property to visual viewport (#9237) #9285

Uh oh!

Conversation

darktears commented Aug 31, 2023

Uh oh!

bokand left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darktears Sep 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darktears commented Sep 14, 2023

Uh oh!

bokand commented Sep 14, 2023

Uh oh!

darktears commented Sep 14, 2023

Uh oh!

bokand commented Sep 14, 2023

Uh oh!

darktears commented Sep 14, 2023

Uh oh!

darktears commented Oct 27, 2023

Uh oh!

smfr commented Oct 28, 2023

Uh oh!

darktears commented Oct 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fantasai left a comment

Choose a reason for hiding this comment

Uh oh!

darktears commented Nov 27, 2023

Uh oh!

aykutbulut commented Mar 27, 2024

Uh oh!

darktears commented Mar 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aykutbulut commented Apr 3, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

darktears commented Jun 11, 2024

Uh oh!

darktears Sep 12, 2023 •

edited

Loading

darktears commented Oct 30, 2023 •

edited

Loading

darktears commented Mar 27, 2024 •

edited

Loading