@shikijs/vscode-textmate

What is @shikijs/vscode-textmate?

@shikijs/vscode-textmate is a library that provides TextMate grammar support for syntax highlighting in JavaScript and TypeScript applications. It allows developers to use TextMate grammars to tokenize source code and apply syntax highlighting, similar to how Visual Studio Code does it.

What are @shikijs/vscode-textmate's main functionalities?

Loading a TextMate Grammar

This code demonstrates how to load a TextMate grammar for JavaScript using the @shikijs/vscode-textmate package. It reads a grammar file and parses it into a usable format.

const vscodeTextmate = require('@shikijs/vscode-textmate');
const fs = require('fs');
const path = require('path');

async function loadGrammar() {
  const registry = new vscodeTextmate.Registry({
    loadGrammar: (scopeName) => {
      if (scopeName === 'source.js') {
        return new Promise((resolve, reject) => {
          fs.readFile(path.join(__dirname, 'JavaScript.tmLanguage.json'), 'utf8', (err, data) => {
            if (err) {
              reject(err);
            } else {
              resolve(vscodeTextmate.parseRawGrammar(data, path.join(__dirname, 'JavaScript.tmLanguage.json')));
            }
          });
        });
      }
      return null;
    }
  });

  const grammar = await registry.loadGrammar('source.js');
  console.log(grammar);
}

loadGrammar();

Tokenizing a Line of Code

This code demonstrates how to tokenize a line of JavaScript code using a loaded TextMate grammar. It shows how to initialize the rule stack and tokenize a given line of code.

const vscodeTextmate = require('@shikijs/vscode-textmate');

async function tokenizeLine(grammar, line) {
  const ruleStack = vscodeTextmate.INITIAL;
  const lineTokens = grammar.tokenizeLine(line, ruleStack);
  console.log(lineTokens);
}

async function main() {
  const registry = new vscodeTextmate.Registry({
    loadGrammar: async (scopeName) => {
      if (scopeName === 'source.js') {
        const grammar = await fetch('path/to/JavaScript.tmLanguage.json').then(res => res.json());
        return vscodeTextmate.parseRawGrammar(JSON.stringify(grammar));
      }
      return null;
    }
  });

  const grammar = await registry.loadGrammar('source.js');
  tokenizeLine(grammar, 'const x = 42;');
}

main();

Other packages similar to @shikijs/vscode-textmate

vscode-textmate

The vscode-textmate package is the original library used by Visual Studio Code for TextMate grammar support. It provides similar functionality to @shikijs/vscode-textmate, allowing developers to load and tokenize source code using TextMate grammars. The main difference is that @shikijs/vscode-textmate is a fork of vscode-textmate with additional features and improvements.

highlight.js

highlight.js is a popular library for syntax highlighting in web applications. Unlike @shikijs/vscode-textmate, which uses TextMate grammars, highlight.js uses its own language definitions. It is easier to set up and use but may not offer the same level of customization and accuracy as TextMate-based solutions.

prismjs

Prism is another popular syntax highlighting library for web applications. It is similar to highlight.js in that it uses its own language definitions rather than TextMate grammars. Prism is known for its lightweight and extensible design, making it a good choice for projects that need a simple and fast syntax highlighting solution.

README

Fork of microsoft/vscode-textmate

Changes make in this fork:

• Change all async operations to sync; onigLib option now required to be resolved instead of a promise.
• Use tsup to bundle the lib, ship as a single file ES module.
• Remove debug flags and some other unnecessary exports.
• Convert EncodedTokenAttributes from namespace to class, rename to EncodedTokenMetadata