Skip to content

Conversation

@cdoublev
Copy link
Collaborator

@cdoublev cdoublev commented Nov 1, 2024

tokenize maps a stream of code points into a stream of tokens:

To tokenize a stream of code points into a stream of CSS tokens input, repeatedly consume a token from input until an <EOF-token> is reached, pushing each of the returned tokens into a stream.

But a stream of tokens is also created with its result:

  1. [...], tokenize the result, then create a new token stream with those tokens as its tokens, and return it.
  1. Let tokens be the result of tokenizing [...]. Let input be a new token stream from tokens.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant