@csstools/css-tokenizer

What is @csstools/css-tokenizer?

@csstools/css-tokenizer is a library for tokenizing CSS. It breaks down CSS code into a stream of tokens, which can be useful for building CSS parsers, linters, or other tools that need to analyze or manipulate CSS code.

What are @csstools/css-tokenizer's main functionalities?

Tokenizing CSS

This feature allows you to tokenize a CSS string. The `tokenize` function takes a CSS string and returns an iterable of tokens, which can then be processed or analyzed.

const { tokenize } = require('@csstools/css-tokenizer');

const css = 'body { color: red; }';
const tokens = tokenize(css);

for (const token of tokens) {
  console.log(token);
}

Handling different token types

This feature demonstrates how to handle different types of tokens. The tokens returned by the `tokenize` function are arrays where the first element is the type of token and the second element is the value. This allows you to differentiate between different token types such as words, whitespace, etc.

const { tokenize } = require('@csstools/css-tokenizer');

const css = 'body { color: red; }';
const tokens = tokenize(css);

for (const token of tokens) {
  if (token[0] === 'word') {
    console.log('Word token:', token[1]);
  } else if (token[0] === 'whitespace') {
    console.log('Whitespace token');
  }
}

Other packages similar to @csstools/css-tokenizer

postcss

PostCSS is a tool for transforming CSS with JavaScript plugins. It includes a tokenizer and parser, making it a more comprehensive tool compared to @csstools/css-tokenizer, which focuses solely on tokenization.

css-tree

CSSTree is a toolset for CSS including a fast detailed parser, walker, generator, and lexer. It provides more advanced features like AST generation and manipulation, whereas @csstools/css-tokenizer focuses on tokenization.

css

The 'css' package is a CSS parser and stringifier. It provides a full parser that converts CSS into an AST and back, offering more functionality than just tokenization, which is the primary focus of @csstools/css-tokenizer.

README

CSS Tokenizer

Implemented from : https://www.w3.org/TR/2021/CRD-css-syntax-3-20211224/

Usage

Add CSS Tokenizer to your project:

npm install @csstools/css-tokenizer --save-dev

import { tokenizer, TokenType } from '@csstools/css-tokenizer';

const myCSS =  `@media only screen and (min-width: 768rem) {
	.foo {
		content: 'Some content!' !important;
	}
}
`;

const t = tokenizer({
	css: myCSS,
});

while (true) {
	const token = t.nextToken();
	if (token[0] === TokenType.EOF) {
		break;
	}

	console.log(token);
}

Or use the tokenize helper function:

import { tokenize } from '@csstools/css-tokenizer';

const myCSS =  `@media only screen and (min-width: 768rem) {
	.foo {
		content: 'Some content!' !important;
	}
}
`;

const tokens = tokenize({
	css: myCSS,
});

console.log(tokens);

Options

{
	onParseError?: (error: ParseError) => void
}

onParseError

The tokenizer is forgiving and won't stop when a parse error is encountered. Parse errors also aren't tokens.

To receive parsing error information you can set a callback.

import { tokenizer, TokenType } from '@csstools/css-tokenizer';

const t = tokenizer({
	css: '\\',
}, { onParseError: (err) => console.warn(err) });

while (true) {
	const token = t.nextToken();
	if (token[0] === TokenType.EOF) {
		break;
	}
}

Parser errors will try to inform you about the point in the tokenizer logic the error happened. This tells you the kind of error.

Goals and non-goals

Things this package aims to be:

• specification compliant CSS tokenizer
• a reliable low level package to be used in CSS parsers

What it is not:

• opinionated
• fast
• small