[selectors] Why would this particular grammar production not be "optimized"?

[amn]

Looking at the <attribute-selector> production in the Selectors 4 (draft) grammar:

<attribute-selector> = '[' <wq-name> ']' |
    '[' <wq-name> <attr-matcher> [ <string-token> | <ident-token> ] <attr-modifier>? ']'

I wonder, out of mere curiosity, why not express the production with the following alternative:

<attribute-selector> = '[' <wq-name> [ <attr-matcher> [ <string-token> | <ident-token> ] <attr-modifier>? ]? ']'

Is there some significance to specifically defining the production with the former rule, or was it happenstance? I understand that swapping a grammar rule even with [what appears to be] an equivalent rule, has implications of its own, but like I said I'm just curious if this incidence (and possible others like it) is by-design? It's possible it is written the way it is written for better expressing to the reader the fact they're two different attribute selector syntaxes? It didn't have that effect on me, I must say, on the contrary.

[cdoublev]

I think spec authors write production rules to be human-friendly, and they welcome rewrites, as long as they preserve that aspect.

For example, this could have been written with:

<attribute-selector> = '[' <attr-name> <attr-value-comparison>? ']'
<attr-name> = <wq-name>
<attr-value-comparison>? = <attr-matcher> <attr-value> <attr-modifier>?
<attr-value> = <string-token> | <ident-token>

From my experience, some production rules are even worse for the performance of a CSS parser consuming them:

<legacy-rgb-syntax> = rgb( <percentage>#{3} , <alpha-value>? ) |
                      rgb( <number>#{3} , <alpha-value>? )

If it were written with rgb( [... | ...], <alpha-value>? ), a parser could abort parsing the declaration value if it finds a function whose name is rgb but the result from parsing its value is a failure: the declaration value is guaranteed to be invalid. In property values like background, in which <color> has many sibling productions, this is very unfortunate.

[amn]

If the specification is intended to be readable, which I think is a worthy goal (meant to be read by humans, after all, including the grammar), then perhaps it could in the very least be useful to define "equivalence" criteria between rules expressed with the grammar and alternatives, e.g. like those you suggest above which admit/refuse the same set of input (don't necessarily parse equivalently, since the concrete syntax tree is different, but are otherwise "equivalent"). That way, if an implementation chooses to rewrite a grammar production for the sake of optimization, taking into account some peculiarities of the implementation that benefit from or outright demand alternative production expressions (an "optimized" variant), at least the implementation can rightfully claim adherence to the specification.

As it is today, since the specification doesn't say whether the grammar is to be followed exactly, or how much leeway there is [allowed] in interpreting and implementing it, questions like my own above are valid, I guess.

While your suggestion(s) are valid in more general context, if the grammar is to be taken literally and no less strictly, then the kind of "optimizations" we're debating, are in violation of the spec. And I, for one, would like to know how much leeway there is for an implementer. Grammar is quite decisive for a parser -- like you yourself said, it makes for a tangible difference for a parser how a production is composed. Certain kinds of parsers, for example, are unable to handle left- or right- recursion in the grammar (which then must be "optimized" [for the parser]), and although current grammar does not contain any productions exhibiting such propert(ies), if rewriting grammar is "not permitted", there's even less leeway for an entire class of parsers that are already grammar-bound.

[tabatkins]

The grammar has to be followed exactly in the sense that you must get exactly the same observable end result as what the spec says. If you can do so by rewriting grammars to something else and handling them in a different way, but at the end the observable result is the same, great! Whatever you do internally is not our concern; what matters is what behavior we can observe via tests.

We often write our grammars in ways intended to be somewhat more human-readable. In the attr selector case, for example, an attr without a comparison ([foo]) looks different from one with a comparison ([foo="bar"], while all the comparison ones are only trivially different. It definitely could have been written all in one production, and you'll definitely find examples of that in other grammars; the exact choice depends on what the grammar writer felt was more readable at the time.

If you ever wonder if you can handle something differently from how it's written in a spec, just ask yourself: does what I'm doing result in different behavior that the user of the page could observe (including with script)? If so, it's not valid; but if your difference is unobservable, feel free to do whatever.

[amn]

Thank you for chipping in (again), appreciate your elaboration.

I think one still needs to establish what "exactly" exactly means in this context, or in other words what constitutes "sameness" and "equivalence" between two specification-compliant parse trees (since there's specified a grammar, after all).

The [Selectors] specification does not explicitly define parsing procedures, there's only the grammar and other bits in the Syntax and Values & Units specifications to go by, for parsing of selectors, that is.

As such, one could in principle have two parsers but where one builds one type of structure containing the same tokens as another parser with another type of structure. I assume when you mention "exact", you're admitting that two such parse trees are equivalent, ref. "observable result", despite having different internal structure(s) i.e. implemented differently?

For instance, I have a parser generator that generates a parser that, parsing "foo" as a selector list expression (selector_list in the grammar), returns, due to being auto-generated recursive descent parser, an array of arrays that somewhere contain "foo" in the form of an <ident-token> object, say [ [ ident_token("foo") ] ] (expressed with JavaScript syntax). Another parser may, parsing the same input as the same grammar production, simply return ident_token("foo") (again, JavaScript syntax).

Are both parse results permissible by the spec.? The way I understand your last comment, they are, but wanting to write a specification-compliant implementation, I want to be sure, and unfortunately the spec. itself does not make it clear for my part. Maybe I expect more of it than what it set out to define in the first place, that is entirely possible.

Anyway, a grammar is not a parser, admittedly, so if I were more experienced in writing parsers, I could, perhaps, simply assume that the spec allows any parser which somehow vends the same sequence of tokens (as another parser) for same input when parsing the same production.

In the implementation I am writing I have a "flatten" procedure that takes a parser product (as I call it, aka a branch of a parse tree) and vends all contained tokens, hence "flattening" the product.

I would imagine then, sameness and equivalence and exactness can be defined as "if the result of parsing input in accordance to a given production constitutes the same ordered set of tokens as another result, the results are equivalent/same in this context". What would you say to that?