tree-sitter-languages-freed-wu-pr

================================================== Python Bindings for Tree Sitter with All Languages

This is a fork!

It is based on @Freed_Wu's pull request_ which adds support for many additional tree-sitter language parsers beyond the original.

As the package seems unmaintained, I decided to fork it based on the PR. Note that this version may not see additional support and you should switch back to the original once it gets going again.

The package has been renamed to avoid collision with the original.

The Windows and ARM Arch64 packages still only get the original languages, for Windows because the extended commandline busted the link.exe command line length limit, and for ARM Arch64 because the Github action uses QEMU to build on ARM, which is slow, so the additional languages cause the Github runner to time out.

Original README

Binary Python wheels for all tree sitter languages.

py-tree-sitter_ is a fantastic library that provides Python bindings for the even more fantastic tree-sitter_ parsing library.

py-tree-sitter-languages_ provides binary Python wheels for all tree sitter languages. The binary wheels remove the need to download and compile support for individual languages.

This package is based on

.. _py-tree-sitter-languages: https://github.com/grantjenks/py-tree-sitter-languages .. _@Freed_Wu's pull request: https://github.com/grantjenks/py-tree-sitter-languages/pull/53

Install

::

pip install tree_sitter_languages_freed_wu_pr

Source installs are not supported. To see how the binary wheels are built, look at:

1. setup.py — Python package setup.

2. parsers.json, lockfile.json — Json files that contain a list of included language repositories and their commit hashes.

3. build.py — Python script to download and build the language repositories.

4. .github/workflows/release.yml — GitHub action to invoke cibuildwheel_ and release to PyPI.

.. _cibuildwheel: https://github.com/pypa/cibuildwheel

Usage

.. code:: python

from tree_sitter_languages_freed_wu_pr import get_language, get_parser

language = get_language('python') parser = get_parser('python')

That's the whole API!

Refer to py-tree-sitter_ for the language and parser API. Notice the Language.build_library(...) step can be skipped! The binary wheel includes the language binary.

.. _py-tree-sitter: https://github.com/tree-sitter/py-tree-sitter

Demo

Want to know something crazy? Python lacks multi-line comments. Whhaaa!?!

It's really not such a big deal. Instead of writing

.. code:: python

""" My awesome multi-line comment. """

Simply write

.. code:: python

My awesome

multi-line

comment.

So multi-line comments are made by putting multiple single-line comments in sequence. Amazing!

Now, how to find all the strings being used as comments?

Start with some example Python code

.. code:: python

example = """ #!shebang

License blah blah (Apache 2.0)

"This is a module docstring."

a = 1

'''This is not a multiline comment.'''

b = 2

class Test: "This is a class docstring."

   'This is bogus.'

   def test(self):
       "This is a function docstring."

       "Please, no."

       return 1

c = 3 """

Notice a couple things:

1. Python has module, class, and function docstrings that bare a striking resemblance to the phony string comments.

2. Python supports single-quoted, double-quoted, triple-single-quoted, and triple-double-quoted strings (not to mention prefixes for raw strings, unicode strings, and more).

Creating a regular expression to capture the phony string comments would be exceedingly difficult!

Enter tree-sitter_

.. code:: python

from tree_sitter_languages_freed_wu_pr import get_language, get_parser

language = get_language('python') parser = get_parser('python')

Tree-sitter creates an abstract syntax tree (actually, a concrete syntax tree_) and supports queries

.. code:: python

tree = parser.parse(example.encode()) node = tree.root_node print(node.sexp())

.. _concrete syntax tree: https://stackoverflow.com/q/1888854/232571

Look for statements that are a single string expression

.. code:: python

stmt_str_pattern = '(expression_statement (string)) @stmt_str' stmt_str_query = language.query(stmt_str_pattern) stmt_strs = stmt_str_query.captures(node) stmt_str_points = set( (node.start_point, node.end_point) for node, _ in stmt_strs ) print(stmt_str_points)

Now, find those statement string expressions that are actually module, class, or function docstrings

.. code:: python

doc_str_pattern = """ (module . (comment)* . (expression_statement (string)) @module_doc_str)

   (class_definition
       body: (block . (expression_statement (string)) @class_doc_str))

   (function_definition
       body: (block . (expression_statement (string)) @function_doc_str))

""" doc_str_query = language.query(doc_str_pattern) doc_strs = doc_str_query.captures(node) doc_str_points = set( (node.start_point, node.end_point) for node, _ in doc_strs )

With the set of string expression statements and the set of docstring statements, the locations of all phony string comments is

.. code:: python

comment_strs = stmt_str_points - doc_str_points print(sorted(comment_strs))

License

Copyright 2022-2023 Grant Jenks

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

The project also includes the following other projects distributed in binary form:

• https://github.com/tree-sitter/tree-sitter — licensed under the MIT License.

• https://github.com/WhatsApp/tree-sitter-erlang — licensed under the Apache License, Version 2.0.

• https://github.com/Azganoth/tree-sitter-lua — licensed under the MIT License.

• https://github.com/Wilfred/tree-sitter-elisp — licensed under the MIT License.

• https://github.com/alemuller/tree-sitter-make — licensed under the MIT License.

• https://github.com/camdencheek/tree-sitter-dockerfile — licensed under the MIT License.

• https://github.com/camdencheek/tree-sitter-go-mod — licensed under the MIT License.

• https://github.com/elixir-lang/tree-sitter-elixir — licensed under the Apache License, Version 2.0.

• https://github.com/elm-tooling/tree-sitter-elm — licensed under the MIT License.

• https://github.com/fwcd/tree-sitter-kotlin — licensed under the MIT License.

• https://github.com/ganezdragon/tree-sitter-perl — licensed under the MIT License.

• https://github.com/ikatyang/tree-sitter-markdown — licensed under the MIT License.

• https://github.com/ikatyang/tree-sitter-yaml — licensed under the MIT License.

• https://github.com/jiyee/tree-sitter-objc — licensed under the MIT License.

• https://github.com/m-novikov/tree-sitter-sql — licensed under the MIT License.

• https://github.com/r-lib/tree-sitter-r — licensed under the MIT License.

• https://github.com/rydesun/tree-sitter-dot — licensed under the MIT License.

• https://github.com/slackhq/tree-sitter-hack — licensed under the MIT License.

• https://github.com/theHamsta/tree-sitter-commonlisp — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-bash — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-c — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-c-sharp — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-cpp — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-css — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-embedded-template — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-go — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-haskell — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-html — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-java — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-javascript — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-jsdoc — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-json — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-julia — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-ocaml — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-php — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-python — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-ql — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-regex — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-ruby — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-rust — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-scala — licensed under the MIT License.

• https://github.com/dhcmrlchtdj/tree-sitter-sqlite - licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-toml — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-tsq — licensed under the MIT License.

• https://github.com/tree-sitter/tree-sitter-typescript — licensed under the MIT License.

• https://github.com/stsewd/tree-sitter-rst - licensed under the MIT License.

• https://github.com/MichaHoffmann/tree-sitter-hcl - licensed under the Apache License, Version 2.0.

• https://github.com/stadelmanma/tree-sitter-fortran - licensed under the MIT License.

• https://github.com/stadelmanma/tree-sitter-fixed-form-fortran - licensed under the MIT License.

.. _tree-sitter: https://tree-sitter.github.io/