[css-syntax-3] Consume an ident-like token algorithm differs for function tokens that start with `url`

[romainmenke]

https://www.w3.org/TR/css-syntax-3/#consume-ident-like-token

1. If string’s value is an ASCII case-insensitive match for "url", and the next input code point is U+0028 LEFT PARENTHESIS ((), consume it
2. While the next two input code points are whitespace, consume the next input code point.
3. If the next one or two input code points are U+0022 QUOTATION MARK ("), U+0027 APOSTROPHE ('), or whitespace followed by U+0022 QUOTATION MARK (") or U+0027 APOSTROPHE ('), then create a <function-token> with its value set to string and return it.

This algorithm indicates that whitespace characters are consumed and "attributed" to the function token.

For example two spaces before "foo" :

url(  "foo")

Following this algorithm it tokenizes as :

['function-token', 'url( ', 0, 4, { value: 'url' }],
['whitespace-token', ' ', 5, 5, undefined],
['string-token', '"foo"', 6, 10, { value: 'foo' }],
[')-token', ')', 11, 11, undefined],
['EOF-token', '', -1, -1, undefined],

While any other function name tokenizes as :

['function-token', 'not-url(', 0, 7, { value: 'not-url' }],
['whitespace-token', '  ', 8, 9, undefined],
['string-token', '"foo"', 10, 14, { value: 'foo' }],
[')-token', ')', 15, 15, undefined],
['EOF-token', '', -1, -1, undefined],

The raw values of the function tokens are :

• 'url( ' (notice the trailing space)
• 'not-url('

The raw values of the whitespace tokens are :

• ' '
• ' '

Sorry for the technical notation, when I tried to write it differently it became more ambiguous

---

First of all I am unsure if I read the specification correctly.
I don't know if I am tokenizing these correctly or if I have a bug.

I don't think it affects parsing because the meaningful values are not affected by consuming whitespace in this way.

This might be relevant in developer tooling where it is important to tokenize and serialize without mutations.

[tabatkins]

This notion of characters "attributed" to a token doesn't exist in the spec, and is not considered in any way while writing the spec. If you're producing some tooling that cares about that, you're going beyond what's specced, and are on your own. ^_^

That said, the reason urls and non-urls parse differently in this way is to retain the limited character lookahead (CSS only requires 3 chars of lookahead to tokenize). When consuming a url(), I don't know whether to consume it as a normal function or in the special url-token way until I see the whether the first argument is a string or not, but there can be an arbitrary amount of whitespace between the ( and the character that tells me that. Since the amount of whitespace is intentionally not preserved by the parser as specced, only the presence of it, I can eagerly consume all but the last space, and only need to look two characters ahead to know, eventually, which branch to take.

By doing this, if I encounter a string, then I can just return the function token immediately and there's still a single unconsumed whitespace character, which ensures that a whitespace token is correctly emitted, the same as if I'd been able to predict immediately that it would be a normal function. Exactly when the N-1 preceding whitespace characters are consumed changes between the spec and the hypothetical correct early guess, but that's not observable in the token stream as produced by the specced parser.

[romainmenke]

If you're producing some tooling that cares about that, you're going beyond what's specced, and are on your own.

Correct :) Still doing my best not to invent too many things.

the reason urls and non-urls parse differently in this way is to retain the limited character lookahead

In that case it makes sense for us to do more lookahead and "attribute" all the whitespace characters to the whitespace token.

As you said this does not affect the actual token stream and wont affect parsing into component values.

Thank you for these insights 🙇