8000 [css-syntax] Add a note about tokenizer lookahead. · w3c/csswg-drafts@6a2f5de · GitHub
Skip to content

Commit 6a2f5de

Browse files
tabatkinstabatkins
committed
[css-syntax] Add a note about tokenizer lookahead.
1 parent c9a5edd commit 6a2f5de

2 files changed

Lines changed: 11 additions & 0 deletions

File tree

‎css-syntax/Overview.html‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -682,6 +682,11 @@ <h2 id=tokenization><span class=secno>4. </span> Tokenization</h2>
682682
tokens with the "id" type are valid <a
683683
href="http://www.w3.org/TR/selectors/#id-selectors">ID selectors</a>.
684684

685+
<p class=note> As a technical note, the tokenizer defined here requires
686+
only three characters of look-ahead. The tokens it produces are designed
687+
to allow Selectors to be parsed with one token of look-ahead, and
688+
additional tokens may be added in the future to maintain this invariant.
689+
685690
<h3 id=token-diagrams><span class=secno>4.1. </span> Token Railroad
686691
Diagrams</h3>
687692

‎css-syntax/Overview.src.html‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -481,6 +481,12 @@ <h2>
481481
The type flag of hash tokens is used in the Selectors syntax [[SELECT]].
482482
Only hash tokens with the "id" type are valid <a href="http://www.w3.org/TR/selectors/#id-selectors">ID selectors</a>.
483483

484+
<p class='note'>
485+
As a technical note,
486+
the tokenizer defined here requires only three characters of look-ahead.
487+
The tokens it produces are designed to allow Selectors to be parsed with one token of look-ahead,
488+
and additional tokens may be added in the future to maintain this invariant.
489+
484490

485491
<h3 id='token-diagrams'>
486492
Token Railroad Diagrams</h3>

0 commit comments

Comments
 (0)