@csstools/css-tokenizer

CSS Tokenizer

Implemented from : https://drafts.csswg.org/css-syntax/

API

Read the API docs

Usage

Add CSS Tokenizer to your project:

npm install @csstools/css-tokenizer --save-dev

import { tokenize } from '@csstools/css-tokenizer';

const myCSS =  `@media only screen and (min-width: 768rem) {
	.foo {
		content: 'Some content!' !important;
	}
}
`;

const tokens = tokenize({
	css: myCSS,
});

console.log(tokens);

Or use the streaming interface:

import { tokenizer, TokenType } from '@csstools/css-tokenizer';

const myCSS = `@media only screen and (min-width: 768rem) {
	.foo {
		content: 'Some content!' !important;
	}
}
`;

const t = tokenizer({
	css: myCSS,
});

while (true) {
	const token = t.nextToken();
	if (token[0] === TokenType.EOF) {
		break;
	}

	console.log(token);
}

Options

{
	onParseError?: (error: ParseError) => void
}

onParseError

The tokenizer is forgiving and won't stop when a parse error is encountered.

To receive parsing error information you can set a callback.

import { tokenizer, TokenType } from '@csstools/css-tokenizer';

const t = tokenizer({
	css: '\\',
}, { onParseError: (err) => console.warn(err) });

while (true) {
	const token = t.nextToken();
	if (token[0] === TokenType.EOF) {
		break;
	}
}

Parser errors will try to inform you where in the tokenizer logic the error happened. This tells you what kind of error occurred.

Order of priorities

• specification compliance
• correctness
• reliability
• tokenizing and serializing must round trip losslessly
• exposing useful aspects about the source code
• runtime performance
• package size

Similar packages

Other packages similar to @csstools/css-tokenizer

postcss

PostCSS is a tool for transforming CSS with JavaScript plugins. It includes a tokenizer and parser, making it a more comprehensive tool compared to @csstools/css-tokenizer, which focuses solely on tokenization.

css-tree

CSSTree is a toolset for CSS including a fast detailed parser, walker, generator, and lexer. It provides more advanced features like AST generation and manipulation, whereas @csstools/css-tokenizer focuses on tokenization.

css

The 'css' package is a CSS parser and stringifier. It provides a full parser that converts CSS into an AST and back, offering more functionality than just tokenization, which is the primary focus of @csstools/css-tokenizer.