wikiparser-node

WikiParser-Node

Other Languages

• 简体中文

Introduction

WikiParser-Node is an offline Wikitext parser developed by Bhsd for the Node.js environment. It can parse almost all wiki syntax and generate an Abstract Syntax Tree (AST) (Try it online). It also allows for easy querying and modification of the AST, and returns the modified wikitext.

Although WikiParser-Node is not originally designed to convert Wikitext to HTML, it provides a limited capability to do so. Here is a list of example HTML pages rendered using this package.

Other Versions

Mini (also known as WikiLint)

This version provides a CLI, but only retains the parsing functionality and linting functionality. The parsed AST cannot be modified. It powers the Wikitext LSP, which provides multiple language services for editors such as VS Code, Sublime Text, and Helix.

A list of available linting rules can be found here.

Browser-compatible

A browser-compatible version, which can be used for code highlighting or as a linting plugin in conjunction with editors such as CodeMirror and Monaco (Usage example). It has been integrated into the MediaWiki official CodeMirror extension since Release 1.45.

Installation

Node.js

Please install the corresponding version as needed (WikiParser-Node or WikiLint), for example:

npm i wikiparser-node

or

npm i wikilint

Browser

You can download the code via CDN, for example:

<script src="//cdn.jsdelivr.net/npm/wikiparser-node"></script>

or

<script src="//unpkg.com/wikiparser-node/bundle/bundle-lsp.min.js"></script>

For more browser extensions, please refer to the corresponding documentation.

Usage

CLI usage

For MediaWiki sites with the CodeMirror extension installed, such as different language editions of Wikipedia and other Wikimedia Foundation-hosted sites, you can use the following command to obtain the parser configuration:

npx getParserConfig <site> <script path> [user] [force]
# For example:
npx getParserConfig jawiki https://ja.wikipedia.org/w [e-mail]

The generated configuration file will be saved in the config directory. You can then use the site name for Parser.config.

// For example:
Parser.config = 'jawiki';

API usage

Please refer to the Wiki.

Performance

A full database dump (*.xml.bz2) scan of English Wikipedia's ~19 million articles (parsing and linting) on a personal MacBook Air takes about 5 hours.

Known issues

Parser

• Memory leaks may occur in rare cases.
• Invalid page names with unicode characters are treated like valid ones (Example).
• Preformatted text with a leading space is only processed by Token.prototype.toHtml.
• BCP 47 language codes are not supported in language conversion (Example).

HTML conversion

Expand

Extension

• Many extensions are not supported, such as <indicator> and <ref>.
• & needs to be escaped in <syntaxhighlight> (Example).

Transclusion

• Some parser functions are not supported.
• New lines in {{localurl:}} are not handled correctly (Example).

Heading

• The table of contents (TOC) is not supported.

HTML tag

• Style sanitization is sometimes different (Example).
• Table fostered content from <table> HTML tags (Example).

Table

• <caption> elements are wrapped in <tbody> elements (Example).
• Unclosed HTML tags in the table fostered content (Example).
• <tr> elements should not be fostered (Example).

Link

• Link trail is not supported (Example).
• Block elements inside a link should break it into multiple links (Example).
• Invalid or missing images (Examples 1, 2).
• Link starting with ../ on a subpage (Example).

External link

• External images are not supported (Examples 1, 2).
• No percent-encoding in displayed free external links (Example).

Block element

• Incomplete <p> wrapping when there are block elements (e.g., <pre>, <div> or even closing tags).
• Mixed lists (Example).

Language conversion

• Automatic language conversion is not supported.
• Support for manual language conversion is minimal (Example).

Miscellaneous

• Illegal HTML entities (Example).