[css-values-4] Specify that calculation trees explicitly work in double precision

[tabatkins]

In this twitter thread by @starsandrobots, a CSS hack is shared that, in Chrome, causes values to snap to multiples of 1024px. This is because Chrome evaluates calc()s in f64 precision, and the hack uses very large values that overflow the safe-integer range of a double by several bits.

In general CSS precision is intentionally unspecified (tho not explicitly unspecified; I should fix that...), to allow impls freedom to use their desired size of LayoutUnit, pack values into optimized structs, etc., so numeric precision/width can vary across impls and across properties in a single impl. But it seems likely that, within a calc(), impls will use a single consistent precision (since otherwise it would require multiple separate implementations of calculation trees and all the math functions for each numeric type).

Currently this precision also varies across impls (the hack doesn't work in Firefox, since they use f32s in calculation trees), but it seems likely that we can standardize this value across impls, and as people get used to using more complex calc()s, it's likely they'll come to depend on particular precisions, accidentally or purposefully. Since the Typed OM exposes numbers in f64s (JS's native numeric type), I think we should specify that calculation trees do their work in that precision too.

Thoughts? @emilio @heycam @domenic @smfr @bfgeek @xiaochengh

[smfr]

WebKit stores values internally as doubles, so I have no problem specifying double precision for calc values.

[emilio]

I don't understand why making calc() special here. If I understand correctly, Blink and WebKit store every other non-calc() length as float.

I actually see no occurrence of double in WebKit's CalculationValue.cpp, @smfr what am I missing?

I think the difference here is that the WebKit CSS parser does operate on double and they then truncate to float rather than Gecko's, which uses float everywhere... But maybe I am missing something here.

[smfr]

I actually see no occurrence of double in WebKit's CalculationValue.cpp, @smfr what am I missing?

The values are stored via CSSPrimitiveValue, which has 'double num' in its union.

[emilio]

Sure, but that means that the actual math for calc() happens in float precision right?

[SelenIT]

The popularity of the StackOverflow answer referenced in the twitter tread linked above also appears to demonstrate the need in the functionality similar to Math.floor() (to get the "step function" behavior).

[Loirooriol]

@SelenIT See #2513

[tabatkins]

Yeah, floor/mod/round are all reasonable requests for additional functions.

("mod" especially was proposed as part of the original calc() proposal over a decade ago; it was left out at the time because it created the "unexpected zero division not detectable at parse-time" problem that we can now handle.)

[smfr]

Sure, but that means that the actual math for calc() happens in float precision right?

You're right, Length stores values in floats. So WebKit will do parse-time simplification in double precision, but will use floats later at computed value time.

[css-meeting-bot]

The CSS Working Group just discussed Double precision math in calc(), and agreed to the following:

• RESOLVED: Math precision in CSS is currently kept undefined

The full IRC log of that discussion

<fremy> Topic: Double precision math in calc()
<RossenTheReal> github: https://github.com//issues/4551
<TabAtkins> https://twitter.com/starsandrobots/status/1199757377286754309
<myles> s/-infinity sounds odd/does - have to be followed by a digit in order to be classified as a negation? can we actually do -infinity without a parser change?/
<fremy> TabAtkins: somebody mentioned that you can use int precision in Blink to implement mod()
<tantek_> -1nfinity
<fremy> TabAtkins: this is ridiculous, and I think it shouldn't work
<fremy> dbaron: what is the consistency difference?
<myles> q+
<fremy> emilio: Firefox uses floats for everything, but Blink and Webkit use floats only when parsing and simplifying, but inside the math expression in use-value-time, it's float
<TabAtkins> s/use floats only/use doubles only/
<fremy> TabAtkins: In general, I agree that precision should be kept undefined if we can
<fremy> TabAtkins: but TypedOM will allow input any double, so I guess we already have that requirement
<fremy> emilio: not really, right now we store as float inside the cssom, typedom is truncated
<fremy> myles: also, we could also be able to used fixedpoint
<fremy> TabAtkins: but not inside a calc() right?
<fremy> myles: not right now, but I would like to keep the option
<astearns> ack myles
<fremy> myles: also, order of operations would need to be specified for the double-precision to make sense as a requirement
<fremy> myles: and I'd like not to have to do this
<astearns> ack dbaron
<fremy> TabAtkins: based on the concensus in the room, it seems we want to reject this proposal
<fremy> dbaron: note that we had compat issues in serialization as dbl vs floats
<chris> q+ to mention an upcoming float/double serialization issue
<fremy> dbaron: specified values would be in float, but computed values were doubles, and that caused issues
<fremy> dbaron: maybe these compat issues are not relevant in the current discussion, but it's interesting to note that if we do, we have 7 random digits at the end
<fremy> chris: we also will have similar issues with colors
<fremy> TabAtkins: we could specify that we also serialize in float space
<fremy> TabAtkins: but myles also wanted to keep fixedpoint, so maybe not
<chris> because color() lab() lch() all use float
<fremy> TabAtkins: but it's a tangent
<dbaron> (as I was talking through it I think I realized the issues I was remembering were about the problem that once you truncate something from double to float, you should never use double-based serialization code to serialize it because you'll then get 6 digits or so tacked on to the end)
<fremy> TabAtkins: is there any objection to keep precision undefined for math?
<fremy> RossenTheReal: any objection?
<fremy> (no objection)
<chris> ack c
<Zakim> chris, you wanted to mention an upcoming float/double serialization issue
<fremy> RESOLVED: Math precision in CSS is currently kept undefined
<fremy> iank_: (Chrome could also investigate changing the parser to use floats, but it's not high in our priority list)
<fremy> iank_: (right now, it depends on the class where we store it, sometimes a float, sometimes a double)