javadoc-tokenizer

javadoc-tokenizer

Tokenize source code documentation created to document non-component functions in StoryBook.js. StoryBook generates component documentations by adding a __docgenInfo property to components. This package can add a __docgenInfo to most functions.

Installation

npm install javadoc-tokenizer

Demo

Download and run npm start.

Goals

1. Portable way to extract doc blocks into JSON objects. Other documentation tools such as esdoc2 do not have a way to use their tokenizers separately.
2. Support a wide range of common tags including @property
3. Normalize type names such as bool => Boolean
4. Extract default values specified in brackets
5. Generate simple signature strings such as bytesToText(bytes, precision = 'auto') ⇒ {String}

Recognized tags

Tag	Output	Description
@name	String	The name of the function (if not present, name will be inferred)
@description	String	The name of the function (if not present, top text will be used)
@param	Object[]	type, description, default value and properties of an argument
@property	Object[]	type, description and default value of an argument's property
@throws	Object[]	type, description and properties of an Error that may be thrown
@examples	Object[]	type, text, description and language of a code example
@access	String	May be public, private, protected or a custom value
@api	String	Alias for @access
@public	String	Same as @access public
@private	String	Same as @access private
@protected	String	Same as @access protected
@chainable	Boolean	True if function is chainable
@deprecated	Boolean	True if function is deprecated
@version	String	Function version
@since	String	Version of library when function was added
@todo	String[]	A list of TODOs
@see	String[]	A list of text/links for more information
@returns	Object	type, description and properties of the return value
@ignore	Boolean	True if function should be omitted from displayed documentation

Note: Other tags will be put into a customTags array.

Usage

Tokenize source code

const fs = require('fs');
const { extract } = require('javadoc-tokenizer');

let src = fs.readFileSync(path, 'utf8');
const functionDocs = extract(src);

Add a __docgenInfo property to all functions possible

const fs = require('fs');
const { getDocgenCode } = require('javadoc-tokenizer');

let src = fs.readFileSync(path, 'utf8');
const docgenCode = getDocgenCode(src);
if (docgenCode) {
	src += '\n\n' + docgenCode;
}

Example Input and Output

Input:

/**
 * Convert numeric bytes to a rounded number with label
 * @example
 * bytesToText(23 * 1024 + 35); // 23.4 KB
 * @param {Number} bytes  The number of bytes
 * @param {Number|String} precision  The decimal precision or "auto"
 * @returns {String}
 */
export default function bytesToText(bytes, precision = 'auto') {
	// ...
}

Output:

[
	{
		access: 'public',
		canAddDocgen: true,
		chainable: null,
		contextCode:
			"export default function bytesToText(bytes, precision = 'auto')",
		customTags: [],
		deprecated: null,
		description: 'Convert numeric bytes to a rounded number with label',
		examples: [
			{
				description: '',
				language: 'js',
				text: 'bytesToText(23 * 1024 + 35); // 23.4 KB',
				type: 'javadoc',
			},
		],
		ignore: false,
		name: 'bytesToText',
		params: [
			{
				default: undefined,
				description: 'The number of bytes',
				name: 'bytes',
				properties: [],
				required: true,
				type: 'Number',
			},
			{
				default: undefined,
				description: 'The decimal precision or "auto"',
				name: 'precision',
				properties: [],
				required: true,
				type: 'Number|String',
			},
		],
		returns: {
			description: '',
			properties: [],
			type: 'String',
		},
		see: [],
		signature: "bytesToText(bytes, precision = 'auto') ⇒ {String}",
		since: null,
		subtype: null,
		throws: [],
		todos: [],
		type: 'function',
		version: null,
	},
];

Limitations

1. Only tested on JavaScript
2. Uses regular expressions instead of code tokenization
3. Tokenizer is not aware of context; e.g. the name of the class they belong to

Unit Tests and Code Coverage

Powered by jest

npm test
npm run coverage

Contributing

Contributions are welcome. Please open a GitHub ticket for bugs or feature requests. Please make a pull request for any fixes or new code you'd like to be incorporated.

License

Open Source under the ISC License.