css-tokenizer.md

Home > @csstools/css-tokenizer

css-tokenizer package

Tokenize CSS following the CSS Syntax Level 3 specification.

Remarks

The tokenizing and parsing tools provided by CSS Tools are designed to be low level and generic with strong ties to their respective specifications.

Any analysis or mutation of CSS source code should be done with the least powerful tool that can accomplish the task. For many applications it is sufficient to work with tokens. For others you might need to use @csstools/css-parser-algorithms or a more specific parser.

Example

Tokenize a string of CSS into an array of tokens:

import { tokenize } from '@csstools/css-tokenizer';

const myCSS = `@media only screen and (min-width: 768rem) {
	.foo {
		content: 'Some content!' !important;
	}
}
`;

const tokens = tokenize({
	css: myCSS,
});

console.log(tokens);

Classes

Class	Description
ParseError	The CSS Tokenizer is forgiving and will never throw on invalid input. Any errors are reported through the onParseError callback.
ParseErrorWithToken

Enumerations

Enumeration	Description
HashType	The type of hash token
NumberType	The type of number token Either integer or number
TokenType	All possible CSS token types

Functions

Function	Description
cloneTokens(tokens)	Deep clone a list of tokens. Useful for mutations without altering the original list.
isToken(x)	Assert that a given value has the general structure of a CSS token: 1. is an array. 2. has at least four items. 3. has a known token type. 4. has a string representation. 5. has a start position. 6. has an end position.
isTokenAtKeyword(x)
isTokenBadString(x)
isTokenBadURL(x)
isTokenCDC(x)
isTokenCDO(x)
isTokenCloseCurly(x)
isTokenCloseParen(x)
isTokenCloseSquare(x)
isTokenColon(x)
isTokenComma(x)
isTokenComment(x)
isTokenDelim(x)
isTokenDimension(x)
isTokenEOF(x)
isTokenFunction(x)
isTokenHash(x)
isTokenIdent(x)
isTokenNumber(x)
isTokenNumeric(x)	Assert that a token is a numeric token
isTokenOpenCurly(x)
isTokenOpenParen(x)
isTokenOpenSquare(x)
isTokenPercentage(x)
isTokenSemicolon(x)
isTokenString(x)
isTokenUnicodeRange(x)
isTokenURL(x)
isTokenWhitespace(x)
isTokenWhiteSpaceOrComment(x)	Assert that a token is a whitespace or comment token
mirrorVariant(token)	Get the mirror variant of a given token
mirrorVariantType(type)	Get the mirror variant type of a given token type
mutateIdent(ident, newValue)	Set the ident value and update the string representation. This handles escaping.
mutateUnit(ident, newUnit)	Set the unit and update the string representation. This handles escaping.
stringify(tokens)	Concatenate the string representation of a list of tokens. This is not a proper serializer that will handle escaping and whitespace. It only produces valid CSS for a token list that is also valid.
tokenize(input, options)	Tokenize a CSS string into a list of tokens.
tokenizer(input, options)	Create a tokenizer for a CSS string.

Interfaces

Interface	Description
Token	The CSS Token interface
TokenAtKeyword
TokenBadString
TokenBadURL
TokenCDC
TokenCDO
TokenCloseCurly
TokenCloseParen
TokenCloseSquare
TokenColon
TokenComma
TokenComment
TokenDelim
TokenDimension
TokenEOF
TokenFunction
TokenHash
TokenIdent
TokenNumber
TokenOpenCurly
TokenOpenParen
TokenOpenSquare
TokenPercentage
TokenSemicolon
TokenString
TokenUnicodeRange
TokenURL
TokenWhitespace

Variables

Variable	Description
ParseErrorMessage

Type Aliases

Type Alias	Description
CSSToken	The union of all possible CSS tokens
NumericToken	The union of all possible CSS tokens that represent a numeric value