[css-typed-om] Rename CSSTokenStream to something that doesn't talk about tokens

[tabatkins]

It's not really exposing tokens a la Syntax, so we should avoid that word.

[FremyCompany]

What is the difference from the tokens from Syntax exactly?
It might help in finding a better name for it.

[tabatkins]

The difference is that it's not exposing tokens, at all. It's just a sequence of alternating strings and variable references.

[FremyCompany]

Uh. That's terrible design. Why not expose tokens?

[tabatkins]

Because tokens are huge overkill for this usecase? What do you think is "terrible" about it? An uninterpreted property is usually exposed as just a string; this is that, but with the vars cut out for easier reference.

[FremyCompany]

The whole point of the Typed OM api is to transform a string into usable values, so I don't like we end up reporting strings again. I mean, the parser already know what things are numbers, functions calls, etc... It is sad we have to give up and return strings.

That being said, I might have made an assumption in regards to how it was being used...

To clear this out, let's look at this exemple. The details are terribly wrong but for the sake of it let's just accept the general flow:

var strValue = getComputedStyle(element).width; //"3px"
var value = CSSValue.parse(strValue);
value.px += 1;
element.style.width = value;

Now are you telling me that in this case, I have to call CSSValue.parse recursively to get my string parsed as it should?

var strValue = getComputedStyle(element).backgroundSize; //"3px var(--height)"
var value = CSSValue.parse(strValue);
value[0].px += 1; // nope, var was used so that was not parsed, and it is a string, or a string wrapped in some value
element.style.backgroundSize = value;

Or do we limit the string-only contagion to the area that is not parsable otherwise? So CSSTokenStream would be CSSFalbackValueList and contains either a list of string or CSSVariableValue?

In this case "background: url(...) rgba(0,0,0,var(--alpha)) 3px 3px" would be

[ CSSUrlValue { ... }, CSSVariableValue { fragments: "rgba(0,0,0,", CSSVariableRef { name: "--alpha: }, ")" }, CSSLength, CSSLength ]

That still does not look very good to me, because we know that rgba is a function, and zero is a number, we could expose this in a better way. But at least it isn't as bad as I thought it was.

[FremyCompany]

What I have in mind as the good solution isn't tokens, it is more the resulting AST of parsing the non-conformant part as CSS. Blocks, functions, those things would be recognized and you could manipulate these this way.

[tabatkins]

Variables disappear by computed-value time. This is about specified values, when a property using a variable is literally impossible to parse into something reasonable.

And for your example, it'll be more like:

["url(...) rgba(0,0,0,", CSSVarRef{name: "--alpha"}, ") 3px 3px"]

While we can theoretically parse some parts of that more thoroughly in some cases, we can't do it in general (since we often can't tell what part of the property the surrounding fragments actually are), so we'll bail entirely. You get meaningful values when the variables resolve and we can do a real parse.

[FremyCompany]

So, yeah, I am not happy with that behavior then. It is incorrect to say you cannot know if things are well-formatted or not, because if they were not, that would be rejected by the parsing rules for css properties.

For instance rgba(0,0,0,var(--alpha) would be rejected here, so it is inaccurate to say we have no idea where we start being unable to parse. https://jsfiddle.net/1j3wg0d0/

[tabatkins]

No, that wouldn't be rejected. Why would it be? In your jsfiddle, the second line is indeed rejected after substitution, because it's trying to set 'color' to rgba(0,0,0,0; }, which obviously is not a valid value for 'color' to have.

We do indeed tokenize before vars are substituted, but we can't interpret them any further. We can't even, say, interpret "2px" as a length; in some cases it'll be something else.

[FremyCompany]

That's simply not true. If you look at the generated CSS OM you will see that the color declaration is being ignored as invalid at parse time.

$0.ownerDocument.styleSheets[1].rules[0].cssText
"* { --alpha1: 1; color: rgba(0,0,0,var(--alpha1)); }"
$0.ownerDocument.styleSheets[1].rules[1].cssText
"* { --alpha0: 0; }"

This can be shown by looking at https://jsfiddle.net/1j3wg0d0/2/ (where I replace alpha1 by 0.5 instead).

Also see https://jsfiddle.net/1j3wg0d0/1/ for further proof that two wrongs cannot make a right in this case.

[FremyCompany]

Regarding 2px being able to be interpreted as something else than a length, I don't see how this is possible, but I remain open the possibility. Could you provide an example of possible ambiguity between two valid interpretations of the same set of css tokens? Except some identifiers tokens might be valid keywords in some context, but in this case it is fine to leave them as just being identifiers, you cannot change an identifier but replace it by something else anyway.

[tabatkins]

The canonically ambiguous example is colors. red could a color (and thus returned as a CSSColor or whatever), or any other keyword in any context. #f00 could be a color (ditto), or be a hash.

I'm not super interested in trying to specify a best-effort "parse some of the values if you can" thing; it'll be (a) confusing for authors as some things get parsed and others don't, often for obscure grammar reasons (how many authors know about nav-up/etc?), and (b) future-hostile, as things become ambiguous or unambiguous as we change the language.

(I thought 2px was ambiguous, but it's probably not; I'm continually forgetting exactly what the value space of font-family is. (It's just <string> | <custom-ident>+.))

[FremyCompany]

My issue with this is that it makes the whole API rather useless. If you receive unparsed strings back when you parse some values, then ultimately it means you will need to come up with your own parser to deal with those, like you have to do for every style today already. If I have to ship my own parser to handle certain values, this entire API is not really useful anymore: why not handle all values through my own parser in this case?

I really think we should do better. I would be okay with something intermediate that only supports the following things:

• values are at least broken apart by AST roots: 5px url(...) center would become three values featuring their own cssText: 5px, url(...) and center. We can do this without loss of generality based on whitespace and block processing in css-syntax.
• values like numbers and numbers+units get parsed properly based on their unit if it is known, as {value:number,unit:string} if not, and we make the promise css will never overload this class of values, which I think makes sense going forward
• blocks and functions can be introspected (blocks as { openingChar, content:CSSValue[], closingChar } and functions as { functionName, arguments:CSSValue[] }), content is parsed the same way as described here

Everything else remains a dummy value that can only be analyzed through cssText. That would be sufficient to solve the efficiency problem when you need to update the numeric value of something and needs to resort to string manipulation to do so now. As far as I know is the aim of this api.

Now, if things need to go further, because things have been broken apart, authors can decide to subparse the various cssText blocks based on additional knowledge, like "#f00" or "red" to colors by using CSSValue.parse("color",...) but they are not forced to if what they wanted to change was the background-position part of the background (which uses a variable).

[FremyCompany]

In this scenario, var(--x, y) can be nested properly in a block or function, so we do not need the weird String | Variable thing. We just have CSSUnparsedValue {cssText:"red"}, CSSUnparsedBlockValue {openingChar:'{'...}, CSSUnparsedFunctionValue {functionName:'url'...} and CSSUnparsedVariableValue {variableName:'primary-color'...}.

[shans]

The CSSTokenStream object represents values which are literally not parsed yet. As in, no engine has parsed them. When they become parsed they stop being CSSTokenStream objects. While engines are free to tokenize up-front, it is not a goal of the typed OM to expose internal vendor-specific implementation details.

Your example above doesn't demonstrate that the values are parsed. Instead it demonstrates that the value has been rejected because of a parentheses mismatch. Look at any implementation of custom properties (or the specification for that matter) and you'll see that there is a parse step after substitution of custom property values.

Also, I take strong exception to your rhetoric:

• "Uh. That's terrible design."
• "My issue with this is that it makes the whole API rather useless."

The first statement is an unwarranted and unhelpful statement of personal opinion. The second, at best, is immediately demonstrably untrue. Please, either participate in a civil manner or do not participate.