vscode-textmate

What is vscode-textmate?

The vscode-textmate package is a library that allows you to tokenize text using TextMate grammars. It is the same tokenizer that Visual Studio Code uses for syntax highlighting. The package can be used to implement syntax highlighting and other text analysis features based on TextMate grammars outside of Visual Studio Code.

What are vscode-textmate's main functionalities?

Tokenization

Tokenize a line of text using a TextMate grammar. This is useful for syntax highlighting or other forms of text analysis.

{"const registry = new tm.Registry();
const grammar = await registry.loadGrammar('source.js');
const text = 'var x = 5;';
const ruleStack = tm.INITIAL;
const lineTokens = grammar.tokenizeLine(text, ruleStack);
console.log(lineTokens.tokens);"}

Loading Grammars

Load a TextMate grammar from a file. This allows you to use custom or third-party grammars for tokenization.

{"const registry = new tm.Registry();
const grammar = await registry.loadGrammarFromPathSync('./syntaxes/javascript.tmLanguage.json');
console.log(grammar.scopeName);"}

Grammar Scopes

Associate a TextMate grammar with a specific scope name. This is useful for managing multiple grammars and switching between them based on the file type or language.

{"const registry = new tm.Registry();
registry.addGrammar(scopeName, './syntaxes/myGrammar.tmLanguage.json');
const grammar = registry.grammarForScopeName(scopeName);
console.log(grammar.scopeName);"}

Other packages similar to vscode-textmate

oniguruma

A package that provides bindings to the Oniguruma regular expressions library. It is used for regex matching within textmate grammars but does not provide the full tokenization capabilities that vscode-textmate offers.

shiki

A syntax highlighter powered by vscode-textmate. It uses the same TextMate grammars and themes as Visual Studio Code for consistent highlighting. Shiki is higher-level than vscode-textmate and provides out-of-the-box rendering of syntax-highlighted code.

highlight.js

A syntax highlighter written in JavaScript. It is self-contained and supports many languages and themes out of the box. Unlike vscode-textmate, it does not use TextMate grammars and instead relies on its own language definitions and parsing logic.

README

VSCode TextMate

An interpreter for grammar files as defined by TextMate. Supports loading grammar files from JSON or PLIST format. Grammar injection is currently not supported.

Installing

npm install vscode-textmate

Using

var Registry = require('vscode-textmate').Registry;
var registry = new Registry();
var grammar = registry.loadGrammarFromPathSync('./javascript.tmbundle/Syntaxes/JavaScript.plist');

var lineTokens = grammar.tokenizeLine('function add(a,b) { return a+b; }');
for (var i = 0; i < lineTokens.tokens.length; i++) {
	var token = lineTokens.tokens[i];
	console.log('Token from ' + token.startIndex + ' to ' + token.endIndex + ' with scopes ' + token.scopes);
}

Using asynchronously

Sometimes, it is necessary to manage the list of known grammars outside of vscode-textmate. The sample below shows how this works:

var Registry = require('vscode-textmate').Registry;

var registry = new Registry({
	getFilePath: function (scopeName) {
		// Return here the path to the grammar file for `scopeName`
		if (scopeName === 'source.js') {
			return './javascript.tmbundle/Syntaxes/JavaScript.plist';
		}
		return null;
	}
});

// Load the JavaScript grammar and any other grammars included by it async.
registry.loadGrammar('source.js', function(err, grammar) {
	if (err) {
		console.error(err);
		return;
	}

	// at this point `grammar` is available...
});

Tokenizing multiple lines

To tokenize multiple lines, you must pass in the previous returned ruleStack.

var ruleStack = null;
for (var i = 0; i < lines.length; i++) {
	var r = grammar.tokenizeLine(lines[i], ruleStack);
	console.log('Line: #' + i + ', tokens: ' + r.tokens);
	ruleStack = r.ruleStack;
}

API doc

See the .d.ts file

Developing

• Clone the repository
• Run npm install
• Compile in the background with npm run watch
• Run tests with npm test
• Run benchmark with npm run benchmark