CSS Tokenizer

This tools lets you tokenize CSS according to the CSS Syntax Specification. Tokenizing CSS is separating a string of CSS into its smallest distinct parts — otherwise known as tokens.

This tool is intended to be used in other tools on the front and back end. It seeks to maintain:

• 100% compliance with the CSS syntax specification. ✨
• 100% code coverage. 🦺
• 100% static typing. 💪
• 1kB maximum contribution size. 📦

Add the CSS tokenizer to your project:

npm install @csstools/tokenizer

Tokenize a string of CSS:

import { tokenize } from 'https://unpkg.com/@csstools/tokenizer'

const cssText = 'auto 50px'

tokenize(cssText, (token) => {
  // 1st time, `token` will represent the word "auto"
  // 2nd time, `token` will represent the space " "
  // 3rd time, `token` will represent the number "50px"
})

The CSS tokenizer separates a string of CSS into tokens. Each token is an object that represents information about a distinct part of the CSS. Then token object includes a token class, an enter index, a leave index, and a split index.

import { tokenize } from 'https://unpkg.com/@csstools/tokenizer'

const cssText = 'auto 50px'

tokenize(cssText, (token) => {
  console.log(
    token.class,
    token.enter,
    token.leave,
    token.split
  )
})

The value of class identifies the kind of token that has been consumed.

The value of enter identifies the position where the token begins.

The value of leave identifies the position where the token ends.

The value of split identifies the position where the token may be split into two smaller parts. In a token representing 50px, this is the position between 50 and px. In a token representing "string", this is the position between "string and ". In a token representing /* test */, this is the position between /* test and */.

interface Token {
	token: number
	enter: number
	split: number
	leave: number
}

As of April 11, 2022, these benchmarks were reported from my local machine:

Benchmark: Tailwind CSS
  ┌────────────────────────────────────────────────────┬───────┬────────┬────────┐
  │                      (index)                       │  ms   │ ms/50k │ tokens │
  ├────────────────────────────────────────────────────┼───────┼────────┼────────┤
  │ CSSTree 1 x 35.86 ops/sec ±6.82% (65 runs sampled) │ 27.88 │  1.47  │ 946205 │
  │ CSSTree 2 x 43.15 ops/sec ±7.57% (60 runs sampled) │ 23.17 │  1.22  │ 946205 │
  │ PostCSS 8 x 14.82 ops/sec ±2.33% (42 runs sampled) │ 67.46 │  3.61  │ 935282 │
  │ CSS Tools x 36.76 ops/sec ±0.14% (65 runs sampled) │ 27.2  │  1.44  │ 946205 │
  └────────────────────────────────────────────────────┴───────┴────────┴────────┘

Benchmark: Bootstrap CSS
  ┌──────────────────────────────────────────────────┬──────┬────────┬────────┐
  │                     (index)                      │  ms  │ ms/50k │ tokens │
  ├──────────────────────────────────────────────────┼──────┼────────┼────────┤
  │ CSSTree 1 x 608 ops/sec ±0.26% (97 runs sampled) │ 1.65 │  1.38  │ 59543  │
  │ CSSTree 2 x 702 ops/sec ±0.16% (97 runs sampled) │ 1.42 │  1.2   │ 59543  │
  │ PostCSS 8 x 440 ops/sec ±0.10% (94 runs sampled) │ 2.27 │  2.21  │ 51453  │
  │ CSS Tools x 623 ops/sec ±0.15% (97 runs sampled) │ 1.61 │  1.35  │ 59543  │
  └──────────────────────────────────────────────────┴──────┴────────┴────────┘

You wanna take a deeper dive? Awesome! Here are a few useful development commands.

The build command creates all the files needed to run this tool in many different JavaScript environments.

npm run build

The test command tests the coverage and accuracy of the tokenizer.

npm run test

The benchmark command builds the project and then tests its performance as compared to PostCSS. These benchmarks are run against Boostrap and Tailwind CSS.

npm run benchmark