tree-sitter

What is tree-sitter?

Tree-sitter is a parser generator tool and an incremental parsing library. It is used to build parsers for programming languages and to parse code into syntax trees. Tree-sitter is designed to be fast and efficient, making it suitable for real-time applications like code editors.

What are tree-sitter's main functionalities?

Parsing Code

This feature allows you to parse source code into a syntax tree. In this example, we parse a simple JavaScript code snippet and print the resulting syntax tree.

const Parser = require('tree-sitter');
const JavaScript = require('tree-sitter-javascript');

const parser = new Parser();
parser.setLanguage(JavaScript);

const sourceCode = 'const x = 1 + 2;';
const tree = parser.parse(sourceCode);

console.log(tree.rootNode.toString());

Querying Syntax Trees

This feature allows you to query syntax trees using a pattern-matching language. In this example, we query the syntax tree for binary expressions with an identifier on the left and a number on the right.

const Parser = require('tree-sitter');
const JavaScript = require('tree-sitter-javascript');
const { query } = require('tree-sitter-query');

const parser = new Parser();
parser.setLanguage(JavaScript);

const sourceCode = 'const x = 1 + 2;';
const tree = parser.parse(sourceCode);

const q = query(JavaScript, '(binary_expression left: (identifier) right: (number))');
const matches = q.matches(tree.rootNode);

console.log(matches);

Incremental Parsing

This feature allows you to incrementally parse code, which is useful for real-time applications like code editors. In this example, we first parse a JavaScript code snippet and then update the syntax tree with a modified version of the code.

const Parser = require('tree-sitter');
const JavaScript = require('tree-sitter-javascript');

const parser = new Parser();
parser.setLanguage(JavaScript);

let sourceCode = 'const x = 1 + 2;';
let tree = parser.parse(sourceCode);

sourceCode = 'const x = 1 + 2 + 3;';
tree = parser.parse(sourceCode, tree);

console.log(tree.rootNode.toString());

Other packages similar to tree-sitter

esprima

Esprima is a high-performance, standard-compliant ECMAScript parser. It parses JavaScript code into an abstract syntax tree (AST). Compared to Tree-sitter, Esprima is specifically focused on JavaScript and does not support incremental parsing.

acorn

Acorn is a small, fast, JavaScript-based JavaScript parser. It generates an abstract syntax tree (AST) and is known for its performance and modularity. Unlike Tree-sitter, Acorn is limited to JavaScript and does not support incremental parsing.

README

Node Tree-sitter

This module provides Node.js bindings to the [tree-sitter] parsing library.

Installation

npm install tree-sitter

Basic Usage

Prerequisites

First, you'll need a Tree-sitter grammar for the language you want to parse. There are many existing grammars, such as tree-sitter-javascript. These grammars can typically be installed with a package manager like NPM, so long as the author has published them.

npm install tree-sitter-javascript

You can also develop a new grammar by using the Tree-sitter CLI and following the docs.

Parsing Source Code

Once you've got your grammar, create a parser with that grammar.

const Parser = require('tree-sitter');
const JavaScript = require('tree-sitter-javascript');

const parser = new Parser();
parser.setLanguage(JavaScript);

Then you can parse some source code,

const sourceCode = 'let x = 1; console.log(x);';
const tree = parser.parse(sourceCode);

and inspect the syntax tree.

console.log(tree.rootNode.toString());

// (program
//   (lexical_declaration
//     (variable_declarator (identifier) (number)))
//   (expression_statement
//     (call_expression
//       (member_expression (identifier) (property_identifier))
//       (arguments (identifier)))))

const callExpression = tree.rootNode.child(1).firstChild;
console.log(callExpression);

// {
//   type: 'call_expression',
//   startPosition: {row: 0, column: 16},
//   endPosition: {row: 0, column: 30},
//   startIndex: 0,
//   endIndex: 30
// }

If your source code changes, you can update the syntax tree. This is much faster than the first parse.

// In the code, we replaced 'let' with 'const'.
// So, we set our old end index to 3, and our new end index to 5.
// Note that the end index is exclusive.
const newSourceCode = 'const x = 1; console.log(x);';
//                        ^ ^
// indices:               3 5
// points:            (0,3) (0,5)

tree.edit({
  startIndex: 0,
  oldEndIndex: 3,
  newEndIndex: 5,
  startPosition: {row: 0, column: 0},
  oldEndPosition: {row: 0, column: 3},
  newEndPosition: {row: 0, column: 5},
});

const newTree = parser.parse(newSourceCode, tree);

Parsing Text From a Custom Data Structure

If your text is stored in a data structure other than a single string, such as a rope or array, you can parse it by supplying a callback to parse instead of a string:

const sourceLines = [
  'let x = 1;',
  'console.log(x);'
];

const tree = parser.parse((index, position) => {
  let line = sourceLines[position.row];
  if (line) {
    return line.slice(position.column);
  }
});

Further Reading

It's recommended that you read the Tree-sitter documentation on using parsers to get a higher-level overview of the API. Once you're comfortable with the basics, you can explore the full API documentation, which should map closely to the C API, though there are some differences.