The tiny, regex powered, lenient, almost spec-compliant JavaScript tokenizer that never fails.
const jsTokens = require("js-tokens");
const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\\t")';
Array.from(jsTokens(jsString), (token) => token.value).join("|");
// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)
Version 9.0.1 (2024-11-22)
/a[/]/ is now parsed as a RegularExpressionLiteral again (regression from 8.0.3). Thanks to No Two (@noootwo) for reporting!Acorn is a full JavaScript parser that can parse source code into an abstract syntax tree (AST). While js-tokens only tokenizes the code, Acorn can parse it and provide more detailed information about the structure of the code.
Esprima is another JavaScript parser that can convert code into an AST. It offers similar functionality to Acorn but has different API and extension points. Compared to js-tokens, Esprima provides a more comprehensive analysis of the code.
Babylon is the parser used by Babel, and it's capable of handling modern JavaScript features. It's more feature-rich than js-tokens, offering AST generation and the ability to handle experimental syntax through plugins.
FAQs
Tiny JavaScript tokenizer.
The npm package js-tokens receives a total of 65,851,101 weekly downloads. As such, js-tokens popularity was classified as popular.
We found that js-tokens demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Python 3.14 adds template strings, deferred annotations, and subinterpreters, plus free-threaded mode, an experimental JIT, and Sigstore verification.
Security News
Former RubyGems maintainers have launched The Gem Cooperative, a new community-run project aimed at rebuilding open governance in the Ruby ecosystem.
Security News
An opt-in lazy import keyword aims to speed up Python startups, especially CLIs, without the ecosystem-wide risks that sank PEP 690.