This tools lets you tokenize CSS according to the CSS Syntax Specification. Tokenizing CSS is separating a string of CSS into its smallest, semantic parts — otherwise known as tokens.

This tool is intended to be used in other tools on the front and back end. It seeks to maintain:

• 100% compliance with the CSS syntax specification. ✨
• 100% code coverage. 🦺
• 100% static typing. 💪
• 1kB maximum contribution size. 📦
• Superior quality over Shark P. 🦈

Add the CSS tokenizer to your project:

npm install @csstools/tokenizer

Tokenize CSS in JavaScript:

import cssTokenizer from '@csstools/tokenizer'

const tokens = Array.from(cssTokenizer(cssText)) // an array of css tokens

Tokenize CSS in classical NodeJS:

const cssTokenizer = require('@csstools/tokenizer')

let iterator = cssTokenizer(cssText), iteration

while (!(iteration = iterator()).done) {
  console.log(iteration.value) // logs an individual css token
}

Tokenize CSS in client-side scripts:

<script type="module">

import cssTokenizer from 'https://unpkg.com/@csstools/tokenizer?module'

const tokens = Array.from(cssTokenizer(cssText)) // an array of css tokens

</script>

Tokenize CSS in classical client-side scripts:

<script src="http://unpkg.com/@csstools/tokenizer"></script>
<script>

const tokens = Array.from(cssTokenizer(cssText)) // an array of css tokens

</script>

The CSS tokenizer separates a string of CSS into tokens represented as an array.

[
  /** Position in the string at which the token was retrieved. */
  number,

  /** Negative number that identifies the kind of token. */
  number,

  /** Opening token content, like the opening of a comment or the quotation mark of a string. */
  string,

  /** Main token content, like the numbers before a unit, or the letters after an at-sign. */
  string,

  /** Closing token content, like the unit of a number, or the closing of a comment. */
  string,
]

As an example, the string @media would become a Name token where @ and media are recognized as distinct parts of that token. As another example, the string 5px would become a Number token where 5 and px are recognized as distinct parts of that token. As a final example, the string 5px 10px would become 3 tokens; the Number as mentioned before (5px), a Space token that represents a single space ( ), and then another Number token (10px).

An actual token is represented in a series of 5 items;

[0, -9, '', '100', '%'] // CSS with a value of "100%"

The first number represents the position at which the token was read. The second number represents the type id of the token. The third, fourth, and fifth strings represent the text prefix, value, and suffix of the token.

As of September 26, 2020, these benchmarks were averaged from my local machine:

Benchmark: Tailwind CSS
  ┌──────────────────────────────────────────────────┬───────┬────────┬────────┐
  │                     (index)                      │  ms   │ ms/50k │ tokens │
  ├──────────────────────────────────────────────────┼───────┼────────┼────────┤
  │ PostCSS x 12.38 ops/sec ±3.33% (35 runs sampled) │ 80.79 │  6.97  │ 579896 │
  │ Develop x 12.07 ops/sec ±1.54% (34 runs sampled) │ 82.86 │  6.17  │ 670972 │
  └──────────────────────────────────────────────────┴───────┴────────┴────────┘
Benchmark: Bootstrap
  ┌────────────────────────────────────────────────┬──────┬────────┬────────┐
  │                    (index)                     │  ms  │ ms/50k │ tokens │
  ├────────────────────────────────────────────────┼──────┼────────┼────────┤
  │ PostCSS x 230 ops/sec ±0.50% (89 runs sampled) │ 4.35 │  4.37  │ 49807  │
  │ Develop x 142 ops/sec ±0.76% (80 runs sampled) │ 7.04 │  5.92  │ 59502  │
  └────────────────────────────────────────────────┴──────┴────────┴────────┘

You wanna take a deeper dive? Awesome! Here are a few useful development commands.

The build command creates all the files needed to run this tool in many different JavaScript environments.

npm run build

The benchmark command builds the project and then tests its performance as compared to PostCSS. These benchmarks are run against Boostrap and Tailwind CSS.

npm run benchmark

The test command tests the coverage and accuracy of the tokenizer.

As of September 26, 2020, this tokenizer has 100% test coverage:

npm run test