js-tokens
Advanced tools
Package description
The js-tokens npm package is a lightweight, regex-based lexer for JavaScript tokens. It can tokenize JavaScript code without the overhead of a full parser, making it useful for tasks like syntax highlighting or extracting specific tokens from code.
Tokenizing JavaScript code
This feature allows you to tokenize a string of JavaScript code into an array of token objects. Each token object describes a syntactic element of the code, such as a keyword, identifier, number, string, or operator.
const jsTokens = require('js-tokens');
const tokens = [...jsTokens('var x = 42;')];
console.log(tokens);Acorn is a full JavaScript parser that can parse source code into an abstract syntax tree (AST). While js-tokens only tokenizes the code, Acorn can parse it and provide more detailed information about the structure of the code.
Esprima is another JavaScript parser that can convert code into an AST. It offers similar functionality to Acorn but has different API and extension points. Compared to js-tokens, Esprima provides a more comprehensive analysis of the code.
Babylon is the parser used by Babel, and it's capable of handling modern JavaScript features. It's more feature-rich than js-tokens, offering AST generation and the ability to handle experimental syntax through plugins.
Changelog
Version 8.0.2 (2023-09-09)
export default, but apparently export = is the correct syntax to use for packages that export a single function, which can be used both in CJS and MJS. (Read more about [Incorrect default export]). It should now be possible to do const jsTokens = require("js-tokens") in a @ts-checkeded JS file without TypeScript complaining. Note: This requires "esModuleInterop": true in your tsconfig.json, but as far as I can tell that’s not a breaking change, since importing js-tokens with "esModuleInterop": false didn’t work at runtime anyway.Readme
The tiny, regex powered, lenient, almost spec-compliant JavaScript tokenizer that never fails.
const jsTokens = require("js-tokens");
const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\\t")';
Array.from(jsTokens(jsString), (token) => token.value).join("|");
// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)
FAQs
Tiny JavaScript tokenizer.
The npm package js-tokens receives a total of 40,978,741 weekly downloads. As such, js-tokens popularity was classified as popular.
We found that js-tokens demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
NIST's new AI Risk Management Framework aims to enhance the security and reliability of generative AI systems and address the unique challenges of malicious AI exploits.
Security News
This episode of the Risky Biz podcast discusses how the rise of small open source packages and the shift towards individual maintainers makes the ecosystem more vulnerable to supply chain attacks.
Product
Streamline your login process and enhance security by enabling Single Sign-On (SSO) on the Socket platform, now available for all customers on the Enterprise plan, supporting 20+ identity providers.