node-html-markdown-cloudflare

node-html-markdown

NHM is a fast HTML to markdown converter, compatible with both node and the browser.

It was built with the following two goals in mind:

1. Speed

We had a need to convert gigabytes of HTML daily very quickly. All libraries we found were too slow with node. We considered using a low-level language but decided to attempt to write something that would squeeze every bit of performance out of the JIT that we could. The end result was fast enough to make the cut!

2. Human Readability

The other libraries we tested produced output that would break in numerous conditions and produced output with many repeating linefeeds, etc. Generally speaking, outside of a markdown viewer, the result was not easy to read.

We took the approach of producing a clean, concise result with consistent spacing rules.

Install

<yarn|npm|pnpm> add node-html-markdown

Benchmarks

-----------------------------------------------------------------------------

Estimated processing times (fastest to slowest):

  [node-html-markdown (reused instance)]
    100 kB:  17ms
    1 MB:    176ms
    50 MB:   8.80sec
    1 GB:    3min, 0sec
    50 GB:   2hr, 30min, 14sec

  [turndown (reused instance)]
    100 kB:  27ms
    1 MB:    280ms
    50 MB:   13.98sec
    1 GB:    4min, 46sec
    50 GB:   3hr, 58min, 35sec

-----------------------------------------------------------------------------

Speed comparison - node-html-markdown (reused instance) is:

  1.02 times as fast as node-html-markdown
  1.57 times as fast as turndown
  1.59 times as fast as turndown (reused instance)

-----------------------------------------------------------------------------

Usage

import { NodeHtmlMarkdown, NodeHtmlMarkdownOptions } from 'node-html-markdown'


/* ********************************************************* *
 * Single use
 * If using it once, you can use the static method
 * ********************************************************* */

// Single file
NodeHtmlMarkdown.translate(
  /* html */ `<b>hello</b>`, 
  /* options (optional) */ {}, 
  /* customTranslators (optional) */ undefined,
  /* customCodeBlockTranslators (optional) */ undefined
);

// Multiple files
NodeHtmlMarkdown.translate(
  /* FileCollection */ { 
    'file1.html': `<b>hello</b>`, 
    'file2.html': `<b>goodbye</b>` 
  }, 
  /* options (optional) */ {}, 
  /* customTranslators (optional) */ undefined,
  /* customCodeBlockTranslators (optional) */ undefined
);


/* ********************************************************* *
 * Re-use
 * If using it several times, creating an instance saves time
 * ********************************************************* */

const nhm = new NodeHtmlMarkdown(
  /* options (optional) */ {}, 
  /* customTransformers (optional) */ undefined,
  /* customCodeBlockTranslators (optional) */ undefined
);

// Single file
nhm.translate(/* html */ `<b>hello</b>`);

// Multiple Files
nhm.translate(
  /* FileCollection */ { 
    'file1.html': `<b>hello</b>`, 
    'file2.html': `<b>goodbye</b>` 
  }, 
);

Options

export interface NodeHtmlMarkdownOptions {
  /**
   * Use native window DOMParser when available
   * @default false
   */
  preferNativeParser: boolean,

  /**
   * Code block fence
   * @default ```
   */
  codeFence: string,

  /**
   * Bullet marker
   * @default *
   */
  bulletMarker: string,

  /**
   * Style for code block
   * @default fence
   */
  codeBlockStyle: 'indented' | 'fenced',

  /**
   * Emphasis delimiter
   * @default _
   */
  emDelimiter: string,

  /**
   * Strong delimiter
   * @default **
   */
  strongDelimiter: string,

  /**
   * Strong delimiter
   * @default ~~
   */
  strikeDelimiter: string,

  /**
   * Supplied elements will be ignored (ignores inner text does not parse children)
   */
  ignore?: string[],

  /**
   * Supplied elements will be treated as blocks (surrounded with blank lines)
   */
  blockElements?: string[],

  /**
   * Max consecutive new lines allowed
   * @default 3
   */
  maxConsecutiveNewlines: number,

  /**
   * Line Start Escape pattern
   * (Note: Setting this will override the default escape settings, you might want to use textReplace option instead)
   */
  lineStartEscape: [ pattern: RegExp, replacement: string ]

  /**
   * Global escape pattern
   * (Note: Setting this will override the default escape settings, you might want to use textReplace option instead)
   */
  globalEscape: [ pattern: RegExp, replacement: string ]

  /**
   * User-defined text replacement pattern (Replaces matching text retrieved from nodes)
   */
  textReplace?: [ pattern: RegExp, replacement: string ][]

  /**
   * Keep images with data: URI (Note: These can be up to 1MB each)
   * @example
   * <img src="data:image/gif;base64,R0lGODlhEAAQAMQAAORHHOVSK......0o/">
   * @default false
   */
  keepDataImages?: boolean

  /**
   * Place URLS at the bottom and format links using link reference definitions
   *
   * @example
   * Click <a href="/url1">here</a>. Or <a href="/url2">here</a>. Or <a href="/url1">this link</a>.
   *
   * Becomes:
   * Click [here][1]. Or [here][2]. Or [this link][1].
   *
   * [1]: /url
   * [2]: /url2
   */
  useLinkReferenceDefinitions?: boolean

  /**
   * Wrap URL text in < > instead of []() syntax.
   * 
   * @example
   * The input <a href="https://google.com">https://google.com</a>
   * becomes <https://google.com>
   * instead of [https://google.com](https://google.com)
   * 
   * @default true
   */
  useInlineLinks?: boolean
}

Custom Translators

Custom translators are an advanced option to allow handling certain elements a specific way.

These can be modified via the NodeHtmlMarkdown#translators property, or added during creation.

For detail on how to use them see:

• translator.ts - Documentation for TranslatorConfig
• config.ts - Translators in defaultTranslators

The NodeHtmlMarkdown#codeBlockTranslators property is a collection of translators which handles elements within a <pre><code> block.

Further improvements

Being a performance-centric library, we're always interested in further improvements. There are several probable routes by which we could gain substantial performance increases over the current model.

Such methods include:

• Writing a custom parser
• Integrating an async worker-thread based model for multi-threading
• Fully replacing any remaining regex

These would be fun to implement; however, for the time being, the present library is fast enough for my purposes. That said, I welcome discussion and any PR toward the effort of further improving performance, and I may ultimately do more work in that capacity in the future!

Help Wanted!

Looking to contribute? Check out our help wanted list for a good place to start!

Changelog

1.3.0 (2022-12-13)

Features

• Added useInlineLinks option (c318667)
• Added option to customize custom strike tag. (82cf1e3)

Fixes

• Can't get text from TextNode in the browser (fixes #40) (feec660)
• perfStart and perfStop breaking code in the browser (fixes #36 (7d52c92)