vscode-textmate

What is vscode-textmate?

The vscode-textmate package is a library that allows you to tokenize text using TextMate grammars. It is the same tokenizer that Visual Studio Code uses for syntax highlighting. The package can be used to implement syntax highlighting and other text analysis features based on TextMate grammars outside of Visual Studio Code.

What are vscode-textmate's main functionalities?

Tokenization

Tokenize a line of text using a TextMate grammar. This is useful for syntax highlighting or other forms of text analysis.

{"const registry = new tm.Registry();
const grammar = await registry.loadGrammar('source.js');
const text = 'var x = 5;';
const ruleStack = tm.INITIAL;
const lineTokens = grammar.tokenizeLine(text, ruleStack);
console.log(lineTokens.tokens);"}

Loading Grammars

Load a TextMate grammar from a file. This allows you to use custom or third-party grammars for tokenization.

{"const registry = new tm.Registry();
const grammar = await registry.loadGrammarFromPathSync('./syntaxes/javascript.tmLanguage.json');
console.log(grammar.scopeName);"}

Grammar Scopes

Associate a TextMate grammar with a specific scope name. This is useful for managing multiple grammars and switching between them based on the file type or language.

{"const registry = new tm.Registry();
registry.addGrammar(scopeName, './syntaxes/myGrammar.tmLanguage.json');
const grammar = registry.grammarForScopeName(scopeName);
console.log(grammar.scopeName);"}

Other packages similar to vscode-textmate

oniguruma

A package that provides bindings to the Oniguruma regular expressions library. It is used for regex matching within textmate grammars but does not provide the full tokenization capabilities that vscode-textmate offers.

shiki

A syntax highlighter powered by vscode-textmate. It uses the same TextMate grammars and themes as Visual Studio Code for consistent highlighting. Shiki is higher-level than vscode-textmate and provides out-of-the-box rendering of syntax-highlighted code.

highlight.js

A syntax highlighter written in JavaScript. It is self-contained and supports many languages and themes out of the box. Unlike vscode-textmate, it does not use TextMate grammars and instead relies on its own language definitions and parsing logic.

README

VSCode TextMate

An interpreter for grammar files as defined by TextMate. Supports loading grammar files from JSON or PLIST format. Cross - grammar injections are currently not supported.

Installing

npm install vscode-textmate

Using

var vsctm = require('vscode-textmate');
var grammarPaths = {
	'source.js': './javascript.tmbundle/Syntaxes/JavaScript.plist'
};

var registry = new vsctm.Registry({
	loadGrammar: function (scopeName) {
		var path = grammarPaths[scopeName];
		if (path) {
			return new Promise((c,e) => {
				fs.readFile(path, (error, content) => {
					if (error) {
						e(error);
					} else {
						var rawGrammar = vsctm.parseRawGrammar(content.toString(), path);
						c(rawGrammar);
					}
				});
			});
		}
		return null;
	}
});

// Load the JavaScript grammar and any other grammars included by it async.
registry.loadGrammar('source.js').then(grammar => {
	// at this point `grammar` is available...
	var lineTokens = grammar.tokenizeLine('function add(a,b) { return a+b; }');
	for (var i = 0; i < lineTokens.tokens.length; i++) {
		var token = lineTokens.tokens[i];
		console.log('Token from ' + token.startIndex + ' to ' + token.endIndex + ' with scopes ' + token.scopes);
	}
});

Tokenizing multiple lines

To tokenize multiple lines, you must pass in the previous returned ruleStack.

var ruleStack = null;
for (var i = 0; i < lines.length; i++) {
	var r = grammar.tokenizeLine(lines[i], ruleStack);
	console.log('Line: #' + i + ', tokens: ' + r.tokens);
	ruleStack = r.ruleStack;
}

API doc

See the main.ts file

Developing

• Clone the repository
• Run npm install
• Compile in the background with npm run watch
• Run tests with npm test
• Run benchmark with npm run benchmark
• Troubleshoot a grammar with npm run inspect -- PATH_TO_GRAMMAR PATH_TO_FILE

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [e-mail] with any additional questions or comments.

License

MIT