[css-color-hdr] How should multiple layered elements with different dynamic-range-limit values behave?

[weinig]

I am trying to wrap my head around how dynamic-range-limit is intended to work (an introduction to the property, not just an introduction to the concepts, would be helpful I think), and I am trying to work out what the intended behavior of multiple elements, all occupying the same space, one on top of another, with different dynamic-range-limit values should be?

If the goal of the property is to rein in overzealous dynamic ranges, choosing the lowest might make sense, but that would contradict the example given:

body { dynamic-range-limit: standard; }
div.photogrid img { dynamic-range-limit: constrained-high } 

which indicates that the highest should be used? Or maybe it is indicating that z-order is important?

(Ultimately, this will need to be mapped to some platform concept, so if there Windows/macOS/etc APIs you are expecting these to be built on / are taking inspiration from, I would be interested to read up on them).

[svgeesus]

(an introduction to the property, not just an introduction to the concepts, would be helpful I think)

Agreed, but pending an improvement in the specthere is an explainer which may help?

[weinig]

That sheds light on the inspiration (looks like UIImage and NSImage from apple's platform have quite similar APIs), but unfortunately I can't find any information on how those work. e.g. are they just hints? if you have two UIImages overlapping each other with different preferredImageDynamicRange values, which one takes precedence?

I think the spec needs to clarify what exactly dynamic-range-limit does and what effects it has.

I can't even really tell if the implication of having dynamic-range-limit: high be the default means that a CSS color value like color(srgb 10 10 10) should now be super bright.

[annevk]

cc @ccameron-chromium

[ccameron-chromium]

When HDR content is composited, it can be thought of* as being all put into a single buffer. This single buffer has the maximum available headroom, which is the headroom of the screen.

• On macOS, this is NSScreen's maximumExtendedDynamicRangeColorComponentValue
• On Android, this is Display.getHdrSdrRatio
• On Windows this is the ratio of DISPLAYCONFIG_SDR_WHITE_LEVEL::SDRWhiteLevel to DXGI_OUTPUT_DESC::MaxLuminance

Let's call the headroom of the screen H.

Each piece of HDR content is tone mapped into this single buffer independently.

• This is a very important concept. Each image, each video, etc, is treated separately.
• Any content that has a maximum pixel value** greater than H is tone mapped so that its maximum value is H.

The dynamic-range-limit allows each individual element, on its own, to say "use something other than H for the tone mapping step".

• If you say standard, the value is 1.
• If you say constrained-high, it's up to the user agent. Chromium uses min(2, H).
• If you say high, then you use H (which is the default behavior today).

So, to the question:

If you have two UIImages overlapping each other with different preferredImageDynamicRange values, which one takes precedence?

Neither. Each image is tone mapped separately. That's what happens already. Consider the following example (assuming dynamic-range-limit doesn't exist):

• I have Image A with maximum pixel value** 8
• I have image B with maximum pixel value** 5
• My display has headroom H=6

In this case, Image A has a tone mapping performed that maps its maximum (8) to 6, while Image B has a totally different tone mapping (in particular, it has no tone mapping applied, because 5<6).

Now assume that Image A has high and Image B has standard. As before, Image A will be tone mapped so that it has a maximum value of 6, while Image B will now be tone mapped so that it has a maximum value of 1.

Once the images have been tone mapped, they're just regular pixel values (that can be overlapping, covering each other, blending, etc).

I can't even really tell if the implication of having dynamic-range-limit: high be the default means that a CSS color value like color(srgb 10 10 10) should now be super bright.

For the moment, I'm only applying this to HDR images, videos, and canvases. For colors, Chromium has unrelated bugs. With respect to how to express this, my opinion on this is as follows: "colors should be like HDR image pixels". HDR image pixels must specify something to the effect of "I am (0.5, 0.9, 0.7) when there is headroom 1 and (6, 9, 8) when there is headroom 10, and what you see will be interpolated between those two based on the display's headroom".

[*] Hardware overlays, etc, will make it so that content will not always end up in a single buffer, but that's a separate detail.
[**] Pixel value in linear space

[weinig]

Thank you @ccameron-chromium, super helpful.

So, for (non-image/video) color values then, do you imagine we will need new css syntax to express these different values at different headrooms?

[ccameron-chromium]

So, for (non-image/video) color values then, do you imagine we will need new css syntax to express these different values at different headrooms?

I think that's the direction to go. It allows colors to "opt in" to being HDR ("is this color HDR?" is a question with messy borders), and it also allows for the design of an HDR site without having to know the display's exact headroom (which I understand to be a very important constraint).

[weinig]

Makes sense.

I think having this processing model spelled out in the spec would be a huge improvement.

I'd also be interested to understand how (if at all) this plays into gamut mapping.

Also, what is the benefit in having constrained-high be user agent defined?

[ccameron-chromium]

I'd also be interested to understand how (if at all) this plays into gamut mapping.

My sense is that gamut mapping should be done to the user's gamut at the effective headroom. So if you have P3 primaries, and the effective headroom is 4, then gamut mapping is done to [0,4]^3 in P3 linear space.

Also, what is the benefit in having constrained-high be user agent defined?

We wanted to allow some flexibility in the implementations to account for "other stuff" (e.g, ambient brightness, etc) in that.

[weinig]

@svgeesus I am happy to close this, unless you want to keep it open to track to adding more information about the processing model to the spec. Just let me know.

[svgeesus]

unless you want to keep it open to track to adding more information about the processing model to the spec

I would prefer to keep it open, so that we can add more to the spec rather than just having it documented in this issue :)

[css-meeting-bot]

The CSS Working Group just discussed [css-color-hdr] How should multiple layered elements with different `dynamic-range-limit` values behave?, and agreed to the following:

• RESOLVED: modify spec text as discussed, to try and flesh out if there are blocking compositor issues in implementations; possibly add an appendix of test cases and blocking issues with platform limitations

The full IRC log of that discussion

<jcraig> weinig: for multiple layered elements, need more added to clarifying the model underlying what this property actually does
<jcraig> uses tone map example
<jcraig> s/map example/map compositing example to explain compositor controlling whether this is additional headroom/
<jcraig> ChrisL: we have a brief section on compositing... would be good to get better text
<jcraig> weinig: understanding what implementors would like to apply here would be useful
<jcraig> and the constraints of existing compositors
<jcraig> Said: the dynamic range limit is cascading, correct?
<jcraig> weinig: inherited but overridden by an explicit defining on higher specificity
<jcraig> simon: yes, agreed at F2F that these are not multipliers
<jcraig> ChrisL: should be documented, possibly in an appendix...
<jcraig> easy to write something that looks great from a color theory perspective but may not be implementable
<jcraig> PROPOSAL: modify spec text as discussed, to try and flesh out if there are blocking compositor issues in implementations; possibly add an appendix of test cases and blocking issues
<jcraig> weinig: WPT is lacking here because the color space can be modified
<jcraig> ChrisL: and WPT is limited to sRGB
<jcraig> s/PROPOSAL: modify spec text as discussed, to try and flesh out if there are blocking compositor issues in implementations; possibly add an appendix of test cases and blocking issues/PROPOSAL: modify spec text as discussed, to try and flesh out if there are blocking compositor issues in implementations; possibly add an appendix of test cases and blocking issues with platform limitations/
<jcraig> RESOLVED: modify spec text as discussed, to try and flesh out if there are blocking compositor issues in implementations; possibly add an appendix of test cases and blocking issues with platform limitations
<jcraig> weinig: I will take that issue

[svgeesus]

@weinig do I recall correctly that you planned to make a PR related to this issue?

[weinig]

@svgeesus, yep, I am working on a PR to add a set of definitions that dynamic-range-limit can be implemented on top of. This is very much a draft:

Webpage HDR Rendering Pipeline

The following describes a simplified description of the rendering/display pipeline to be used by web browsers displaying SDR and HDR content.

The system is comprised of the following components:

• The output device.

  • Has a color space value.
    (e.g. NSScreen.colorSpace in Apple's ecosystem.)
  • Has a current headroom value. This value is equal to 1.0 when only SDR content can be shown and is greater than 1.0 when HDR content can be shown.
    (e.g. NSScreen.maximumExtendedDynamicRangeColorComponentValue in Apple's ecosystem.)

  ACTIONS:
  - Can be sent a message requesting additional headroom.
  (e.g. setting wantsExtendedDynamicRangeContent on one of your CALayers in Apple's ecosystem.)

  A webpage can

• The webpage.

  • Has a backing store with a color space equal to that of the output device.

  ACTIONS:
  - Can be sent a message when current headroom has changed.
  (e.g. listening for NSApplicationDidChangeScreenParametersNotification in Apple's ecosystem.)

• The content.

  • Examples include:
    SDR images
    HDR images
    SDR video
    HDR video
    SDR canvas
    HDR canvas
    SDR color values [color(srgb 0.4 0 1.0)]
    HDR color values [color(srgb 3 5 2)]

  • Has a backing store with a color space (potentially different from the webpage and output device)

  • Has a dynamic-range-limit value that places limits on how much of the current headroom can be used during gamut / tone mapping.

  • Has an optional metadata that can augment how gamut / tone mapping should behave.

  ACTIONS:
  - Can be drawn to the webpage backing store by:
  1. Converting each pixel in its backing store to match the color space of the webpage
  2. If necessary (see pipeline descriptions below), applying gamut / tone mapping to each converted pixel using the values of current headroom, dynamic-range-limit and the optional metadata.
  3. Compositing (using operations defined elsewhere) the final pixels into the webpage backing store.

Pre-HDR webpage pipeline

In the pre-HDR webpage pipeline (that is, the world where output devices might have supported HDR, but webpages did not), the following was expected to take place:

For an `output device` with a `current headroom` value equal to 1.0:
    - When drawing SDR `content` to the `webpage`, no `gamut / tone mapping` is applied.
    - When drawing HDR `content` to the `webpage`, `gamut / tone mapping` is applied, bringing all pixel values to be less than or equal to 1.0 (i.e. `current headroom`) in each channel.

For an `output device` with a `current headroom` value greater than 1.0:
    - When drawing SDR `content` to the `webpage`, no `gamut / tone mapping` is applied.
    - When drawing HDR `content` to the `webpage`, `gamut / tone mapping` is applied with an OVERRIDDEN `current headroom` set to 1.0, bringing all pixel values to be less than or equal to (i.e. the OVERRIDDEN `current headroom`) in each channel.

    Examples
        - ADD EXAMPLE FOR HDR IMAGE HERE
        - For a hypothetical `output device` using `srgb` as its `color space`, a color like `color(srgb 3 5 2)`, would be mapped to a color channel value less than or equal to 1.0. If the mapping was a pure clip, this would be `color(srgb 1 1 1)`.

HDR webpage pipeline

In the HDR webpage pipeline, the following is now expected to take place:

 For an `output device` with a `current headroom` value equal to 1.0:
    - When drawing SDR `content` to the `webpage`, no `gamut / tone mapping`. (No change from pre-HDR pipeline!)
    - When drawing HDR `content` to the `webpage`, `gamut / tone mapping` is applied, bringing all pixel values to be less than or equal to 1.0 (i.e. `current headroom`) in each channel. (No change from pre-HDR pipeline!)

        NOTE: In parallel to this, the `webpage` should message the `output device` requesting `additional headroom`. If the `output device` can accommodate the request, it will update `current headroom` to be greater than 1.0 and message the `webpage` informing it the `current headroom` has changed, at which point the branch below will be followed.

For an `output device` with a `current headroom` value greater than 1.0:
    - When drawing SDR `content` to the `webpage`, no `gamut / tone mapping`. (No change from pre-HDR pipeline).
    - When drawing HDR `content` to the `webpage`, `gamut / tone mapping` is applied, bringing all pixel values to be less than or equal to `current headroom` in each channel. 

        NOTE: The only change between the two pipelines is that in the HDR world, HDR content uses `current headroom` as its upper bound for `gamut / tone mapping`, instead of the OVERRIDDEN value of 1.0.

NOTE: Due to gamut / tone mapping taking current headroom as one of its inputs, webpages will need to potentially repaint whenever current headroom changes.

Optimization Warnings (non-normative)

The pipeline described above is a simplification of the real world, where browsers often use multiple buffers with different backing store configurations, layers, and have the operating system composite and color match on their behalf allowing for certain optimizations (such as hardware color correction). These optimizations should remain usable, but implementors must be careful to ensure that all content drawn into one of these layers require the same corrections to applied by the system.

For example, for a hypothetical output device with a current headroom value of 2.0, a webpage that wants to draw a solid SDR background with a single HDR image on top of it, could use one layer for the background, and one layer for the contents of the image, and have the operating system perform the gamut / tone mapping itself. However, if image has say, a solid SDR border around it, the webpage can't draw the SDR border into the layer used by the image, as the operating system would also apply the gamut / tone mapping to it.

(This all assumes layers have gamut / tone mapping applied by the system for all its pixels. If future systems allow per-pixel metadata, this warning no longer applies.)