jsdoc-tokenizer

jsdoc-tokenizer

Tokenizer (Scanner) for JSDoc. This project only operate on/with Node.js Buffer and Uint8Array (No conversion to String required).

Requirements

• Node.js v10 or higher

Getting Started

This package is available in the Node Package Repository and can be easily installed with npm or yarn.

$ npm i jsdoc-tokenizer
# or
$ yarn add jsdoc-tokenizer

Usage example

const { scan, TOKENS } = require("jsdoc-tokenizer");

const it = scan(Buffer.from("/** @type {String} **/"));
for (const [token, value] of it) {
    if (value instanceof Uint8Array) {
        console.log(token, String.fromCharCode(...value));
    }
    else {
        const tValue = typeof value === "number" ? String.fromCharCode(value) : value;
        console.log(token, tValue);
    }
}

API

scan(buf: Buffer): IterableIterator< [Symbol, Uint8Array | number] >

Scan (tokenize) JSDoc block. The scanner only take single instance of block (not build to detect start and end). To extract JSDoc block as buffer, please take a look at jsdoc-extractor.

TOKENS

Available tokens are described by the following interface:

interface Tokens {
    KEYWORD: Symbol,
    IDENTIFIER: Symbol,
    SYMBOL: Symbol
}

Tokens are exported in the module.

Caveats

• There is a room for improvement on supporting more chars for Identifier.

License

MIT