Big News: Socket raises $60M Series C at a $1B valuation to secure software supply chains for AI-driven development.Announcement
Sign In

@mdream/js

Package Overview
Dependencies
Maintainers
1
Versions
19
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

@mdream/js

JavaScript HTML-to-Markdown engine for mdream. Escape hatch for hooks and edge runtimes.

Source
npmnpm
Version
1.2.1
Version published
Maintainers
1
Created
Source

@mdream/js

npm version npm downloads License

Installation

# pnpm
pnpm add @mdream/js@beta

# npm
npm install @mdream/js@beta

# yarn
yarn add @mdream/js@beta

Entry Points

ImportDescription
@mdream/jsCore htmlToMarkdown and streamHtmlToMarkdown APIs
@mdream/js/pluginsPlugin utilities: createPlugin, extractionPlugin, extractionCollectorPlugin, filterPlugin, frontmatterPlugin, isolateMainPlugin, tailwindPlugin
@mdream/js/preset/minimalwithMinimalPreset for declarative config combining frontmatter, isolateMain, tailwind, and filter plugins
@mdream/js/negotiateHTTP content negotiation: shouldServeMarkdown, parseAcceptHeader
@mdream/js/parseLow-level HTML parser: parseHtml, parseHtmlStream
@mdream/js/splitterSingle-pass markdown splitter: htmlToMarkdownSplitChunks, htmlToMarkdownSplitChunksStream
@mdream/js/llms-txtllms.txt artifact generation: generateLlmsTxtArtifacts, createLlmsTxtStream

API Reference

htmlToMarkdown(html, options?)

Converts an HTML string to Markdown synchronously.

import { htmlToMarkdown } from '@mdream/js'

const md = htmlToMarkdown('<h1>Hello</h1><p>World</p>')
// # Hello\n\nWorld

Parameters:

ParameterTypeDescription
htmlstringThe HTML string to convert
optionsPartial<MdreamOptions>Optional configuration (see MdreamOptions)

Returns: string

streamHtmlToMarkdown(htmlStream, options?)

Converts an HTML stream to Markdown incrementally. Useful for large documents or streaming HTTP responses.

import { streamHtmlToMarkdown } from '@mdream/js'

const stream = streamHtmlToMarkdown(response.body, {
  origin: 'https://example.com',
})

for await (const chunk of stream) {
  process.stdout.write(chunk)
}

Parameters:

ParameterTypeDescription
htmlStreamReadableStream<Uint8Array | string> | nullA web ReadableStream of HTML content
optionsPartial<MdreamOptions>Optional configuration (see MdreamOptions)

Returns: AsyncIterable<string>

Options

MdreamOptions

OptionTypeDefaultDescription
originstringundefinedOrigin URL for resolving relative image paths and internal links
pluginsBuiltinPluginsundefinedDeclarative built-in plugin configuration (see BuiltinPlugins)
cleanboolean | CleanOptionsundefinedPost-processing cleanup. Pass true for all cleanup rules or an object for specific features (see CleanOptions). Sync API only for fragments.
hooksTransformPlugin[]undefinedImperative hook-based transform plugins for custom behavior (see Plugins)

BuiltinPlugins

Declarative configuration for built-in plugins. Works with both the JavaScript and Rust engines.

OptionTypeDefaultDescription
frontmatterboolean | ((fm: Record<string, string>) => void) | FrontmatterConfigundefinedExtract metadata from HTML <head> into YAML frontmatter. Pass true for defaults, a callback to receive structured data, or a config object.
isolateMainbooleanundefinedIsolate main content area. Prioritizes <main> elements, then falls back to header-to-footer heuristic.
tailwindbooleanundefinedConvert Tailwind utility classes (bold, italic, hidden, etc.) to semantic Markdown formatting.
filter{ include?, exclude?, processChildren? }undefinedFilter elements by CSS selectors, tag names, or TAG_* constants (see Filter Plugin).
extractionRecord<string, (element: ExtractedElement) => void>undefinedExtract elements matching CSS selectors during conversion. Each key is a CSS selector; the handler is called for every match.
tagOverridesRecord<string, TagOverride | string>undefinedDeclarative tag overrides. String values act as aliases (e.g., { "x-heading": "h2" }). Object values override specific handler properties.

FrontmatterConfig

OptionTypeDefaultDescription
additionalFieldsRecord<string, string>undefinedExtra key-value pairs to inject into the frontmatter
metaFieldsstring[]['description', 'keywords', 'author', 'date', 'og:title', 'og:description', 'twitter:title', 'twitter:description']Meta tag names to extract beyond the standard set
onExtract(frontmatter: Record<string, string>) => voidundefinedCallback to receive structured frontmatter data after conversion

TagOverride

OptionTypeDescription
enterstringMarkdown string to emit when entering the tag
exitstringMarkdown string to emit when exiting the tag
spacing[number, number]Newlines to add [before, after] the tag
isInlinebooleanWhether the tag is inline (no block-level spacing)
isSelfClosingbooleanWhether the tag is self-closing
collapsesInnerWhiteSpacebooleanWhether inner whitespace should be collapsed

CleanOptions

Post-processing cleanup options. Pass true to clean to enable all of these.

OptionTypeDefaultDescription
urlsbooleanfalseStrip tracking query parameters (utm_*, fbclid, gclid, etc.) from URLs
fragmentsbooleanfalseStrip fragment-only links that do not match any heading slug in the output
emptyLinksbooleanfalseStrip links with meaningless hrefs (#, javascript:void(0), data:, vbscript:) and replace with plain text
blankLinesbooleanfalseCollapse 3+ consecutive blank lines to 2
redundantLinksbooleanfalseStrip links where text equals URL: [https://x.com](https://x.com) becomes https://x.com
selfLinkHeadingsbooleanfalseStrip self-referencing heading anchors: ## [Title](#title) becomes ## Title
emptyImagesbooleanfalseStrip images with no alt text (decorative images, tracking pixels)
emptyLinkTextbooleanfalseDrop links that produce no visible text: [](url) is removed entirely

When clean: true is passed, all options except urls and blankLines are enabled.

Plugins

createPlugin(plugin)

Factory function for creating type-safe transform plugins. All hooks are optional.

import { createPlugin } from '@mdream/js/plugins'

const myPlugin = createPlugin({
  beforeNodeProcess(event, state) {
    // Return { skip: true } to skip this node entirely
  },
  onNodeEnter(element, state) {
    // Return a string to inject markdown at this position
  },
  onNodeExit(element, state) {
    // Return a string to inject markdown at this position
  },
  processAttributes(element, state) {
    // Inspect or modify element attributes
  },
  processTextNode(textNode, state) {
    // Return { content: string, skip: boolean } to transform text
    // Return undefined for no transformation
  },
})

Plugin Hook Reference

HookParametersReturn TypeDescription
beforeNodeProcess(event: NodeEvent, state){ skip: boolean } | voidCalled before any node processing. Return { skip: true } to skip the node.
onNodeEnter(element: ElementNode, state)string | voidCalled when entering an element. Return a string to inject markdown.
onNodeExit(element: ElementNode, state)string | voidCalled when exiting an element. Return a string to inject markdown.
processAttributes(element: ElementNode, state)voidCalled to inspect or modify element attributes.
processTextNode(textNode: TextNode, state){ content: string, skip: boolean } | voidCalled for each text node. Return an object to transform text or skip it.

filterPlugin(options)

Filters elements by CSS selectors, tag names, or TAG_* constants.

import { TAG_FOOTER, TAG_NAV } from '@mdream/js'
import { filterPlugin } from '@mdream/js/plugins'

// Exclude navigation and footer
const plugin = filterPlugin({
  exclude: [TAG_NAV, TAG_FOOTER, '.sidebar', '#ads'],
})

// Include only specific elements
const plugin2 = filterPlugin({
  include: ['article', 'main'],
  processChildren: true, // default: true
})
OptionTypeDefaultDescription
include(string | number)[][]CSS selectors, tag names, or TAG_* constants for elements to include (all others excluded)
exclude(string | number)[][]CSS selectors, tag names, or TAG_* constants for elements to exclude
processChildrenbooleantrueWhether to also process children of matching elements

frontmatterPlugin(options?)

Extracts metadata from HTML <head> into YAML frontmatter. Collects <title> and <meta> tags.

import { frontmatterPlugin } from '@mdream/js/plugins'

const plugin = frontmatterPlugin({
  additionalFields: { source: 'crawler' },
  metaFields: ['robots', 'viewport'],
})

Default meta fields extracted: description, keywords, author, date, og:title, og:description, twitter:title, twitter:description.

isolateMainPlugin()

Isolates the main content area of a page using a priority-based strategy:

  • If an explicit <main> element exists (within 5 depth levels), use its content exclusively.
  • Otherwise, find content between the first heading (h1-h6) that is not inside a <header> tag and the first <footer>.
  • The <head> section is always passed through for other plugins (e.g., frontmatter extraction).
import { isolateMainPlugin } from '@mdream/js/plugins'

const plugin = isolateMainPlugin()

tailwindPlugin()

Converts Tailwind utility classes to Markdown formatting. Supports mobile-first responsive breakpoints (sm:, md:, lg:, xl:, 2xl:).

Tailwind ClassMarkdown Output
font-bold, font-semibold, font-black, font-extrabold, font-medium**text**
italic, font-italic*text*
line-through~~text~~
hidden, invisibleElement skipped
absolute, fixed, stickyElement skipped
import { tailwindPlugin } from '@mdream/js/plugins'

const plugin = tailwindPlugin()

extractionPlugin(selectors) (Deprecated)

Deprecated. Use plugins.extraction config for declarative extraction that works with both JS and Rust engines.

Extracts elements matching CSS selectors during conversion. Callbacks receive matching elements with their accumulated text content.

import { extractionPlugin } from '@mdream/js/plugins'

const plugin = extractionPlugin({
  'h2': (element, state) => {
    console.log('Heading:', element.textContent)
  },
  'img[alt]': (element, state) => {
    console.log('Image:', element.attributes.src)
  },
})

extractionCollectorPlugin(selectors)

Internal extraction collector for the plugins.extraction config. Collects results during processing and calls callbacks post-conversion to match Rust engine behavior.

Presets

withMinimalPreset(options?)

Returns a declarative config combining frontmatter, isolateMain, tailwind, and filter plugins. Also enables clean: true by default. You can override any option.

import { htmlToMarkdown } from '@mdream/js'
import { withMinimalPreset } from '@mdream/js/preset/minimal'

const md = htmlToMarkdown(html, withMinimalPreset({
  origin: 'https://example.com',
}))

The minimal preset excludes these elements by default: <form>, <fieldset>, <object>, <embed>, <footer>, <aside>, <iframe>, <input>, <textarea>, <select>, <button>, <nav>.

You can override or extend the plugin config:

const md = htmlToMarkdown(html, withMinimalPreset({
  origin: 'https://example.com',
  clean: { urls: true, fragments: true },
  plugins: {
    frontmatter: {
      additionalFields: { source: 'my-crawler' },
    },
  },
}))

Content Negotiation

shouldServeMarkdown(acceptHeader?, secFetchDest?)

Determines if a client prefers Markdown over HTML using HTTP content negotiation.

  • Returns true when text/markdown or text/plain has higher quality than text/html in the Accept header.
  • If qualities are equal, earlier position wins.
  • Bare wildcards (*/*) do not trigger Markdown (prevents breaking OG crawlers).
  • sec-fetch-dest: document always returns false (browser navigation).
import { shouldServeMarkdown } from '@mdream/js/negotiate'

if (shouldServeMarkdown(request.headers.accept, request.headers['sec-fetch-dest'])) {
  return new Response(markdown, {
    headers: { 'content-type': 'text/markdown' },
  })
}

Parameters:

ParameterTypeDescription
acceptHeaderstring | undefinedThe HTTP Accept header value
secFetchDeststring | undefinedThe Sec-Fetch-Dest header value

Returns: boolean

parseAcceptHeader(accept)

Parses an HTTP Accept header into an ordered list of media types with quality values.

import { parseAcceptHeader } from '@mdream/js/negotiate'

const entries = parseAcceptHeader('text/markdown;q=0.9, text/html')
// [{ type: 'text/markdown', q: 0.9, position: 0 }, { type: 'text/html', q: 1, position: 1 }]

Returns: Array<{ type: string, q: number, position: number }>

Markdown Splitter

htmlToMarkdownSplitChunks(html, options?)

Converts HTML to Markdown and returns an array of chunks. Compatible with LangChain's Document structure.

import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'

const chunks = htmlToMarkdownSplitChunks(html, {
  chunkSize: 1000,
  chunkOverlap: 200,
  origin: 'https://example.com',
})

for (const chunk of chunks) {
  console.log(chunk.content)
  console.log(chunk.metadata.headers) // e.g., { h2: 'Section Title' }
  console.log(chunk.metadata.code) // e.g., 'typescript'
  console.log(chunk.metadata.loc) // e.g., { lines: { from: 1, to: 10 } }
}

Returns: MarkdownChunk[]

htmlToMarkdownSplitChunksStream(html, options?)

Generator version that yields chunks during processing for better memory efficiency.

import { htmlToMarkdownSplitChunksStream } from '@mdream/js/splitter'

for (const chunk of htmlToMarkdownSplitChunksStream(html, options)) {
  process.stdout.write(chunk.content)
}

Returns: Generator<MarkdownChunk, void, undefined>

SplitterOptions

Extends EngineOptions with chunking-specific settings.

OptionTypeDefaultDescription
headersToSplitOnnumber[][TAG_H2, TAG_H3, TAG_H4, TAG_H5, TAG_H6]TAG_* header constants to split on
returnEachLinebooleanfalseReturn each non-empty line as an individual chunk
stripHeadersbooleantrueStrip header lines from chunk content
chunkSizenumber1000Maximum chunk size (measured by lengthFunction)
chunkOverlapnumber200Overlap between chunks for context preservation. Must be less than chunkSize.
lengthFunction(text: string) => number(text) => text.lengthFunction to measure chunk length. Replace with a token counter for LLM applications.
keepSeparatorbooleanfalseKeep separators in the split chunks

MarkdownChunk

FieldTypeDescription
contentstringThe markdown content of the chunk
metadata.headersRecord<string, string> | undefinedHeader hierarchy at this chunk position (e.g., { h2: 'API', h3: 'Methods' })
metadata.codestring | undefinedCode block language if chunk contains code
metadata.loc{ lines: { from: number, to: number } } | undefinedLine number range in original document

llms.txt Generation

generateLlmsTxtArtifacts(options)

Generates llms.txt content, optionally including llms-full.txt and individual markdown files.

import { generateLlmsTxtArtifacts } from '@mdream/js/llms-txt'

const result = await generateLlmsTxtArtifacts({
  files: processedPages,
  siteName: 'My Site',
  description: 'A description of my site',
  origin: 'https://example.com',
  generateFull: true,
  generateMarkdown: true,
  sections: [
    {
      title: 'Documentation',
      description: 'API reference and guides',
      links: [
        { title: 'Getting Started', href: '/docs/start', description: 'Quick start guide' },
      ],
    },
  ],
  notes: 'Generated by mdream',
})

// result.llmsTxt         -- index file with links to pages
// result.llmsFullTxt     -- single file with all page content (if generateFull: true)
// result.markdownFiles   -- array of { path, content } (if generateMarkdown: true)
// result.processedFiles  -- the input files passed through

LlmsTxtArtifactsOptions

OptionTypeDefaultDescription
filesProcessedFile[](required)Array of processed page files
siteNamestring'Site'Site name for the header
descriptionstringundefinedSite description (rendered as blockquote)
originstring''Origin URL to prepend to relative URLs
generateFullbooleanfalseGenerate llms-full.txt with complete page content
generateMarkdownbooleanfalseGenerate individual markdown files under md/ directory
outputDirstringundefinedOutput directory (used to compute relative file paths)
sectionsLlmsTxtSection[]undefinedCustom sections to write before the Pages section
notesstring | string[]undefinedNotes to append at the end

ProcessedFile

FieldTypeDescription
titlestringPage title
contentstringMarkdown content of the page
urlstringURL path of the page
filePathstring | undefinedOptional file path on disk
metadata{ title?, description?, keywords?, author? }Optional metadata extracted from the page

LlmsTxtSection

FieldTypeDescription
titlestringSection heading
descriptionstring | string[]Section description (can be multiple paragraphs)
linksLlmsTxtLink[]Links in the section
FieldTypeDescription
titlestringLink title
hrefstringLink URL
descriptionstring | undefinedOptional description shown after the link

createLlmsTxtStream(options)

Creates a WritableStream<ProcessedFile> that generates llms.txt artifacts incrementally. Writes files to disk as pages are streamed in, never keeping full content in memory. Pages in llms.txt are sorted by URL path hierarchy on close.

import { createLlmsTxtStream } from '@mdream/js/llms-txt'

const stream = createLlmsTxtStream({
  outputDir: './output',
  siteName: 'My Site',
  origin: 'https://example.com',
  generateFull: true,
})

const writer = stream.getWriter()
await writer.write({ title: 'Home', content: '# Home\nWelcome', url: '/' })
await writer.write({ title: 'About', content: '# About\nInfo', url: '/about' })
await writer.close()
// Writes llms.txt and llms-full.txt to ./output/

Returns: WritableStream<ProcessedFile>

Low-Level Parser

parseHtml(html, options?)

Parses HTML into a list of DOM events. Returns the events and any remaining unparsed HTML.

import { parseHtml } from '@mdream/js/parse'

const { events, remainingHtml } = parseHtml('<p>Hello</p>')

Returns: { events: NodeEvent[], remainingHtml: string }

parseHtmlStream(html, state, onEvent)

Streaming parser that calls onEvent for each DOM event. Returns any remaining unparsed HTML (useful for processing partial chunks).

import { parseHtmlStream } from '@mdream/js/parse'

const state = { depthMap: new Uint8Array(1024), depth: 0 }
const remaining = parseHtmlStream(htmlChunk, state, (event) => {
  // Process each event
})

Returns: string (remaining unparsed HTML)

CLI

Reads HTML from stdin and outputs Markdown to stdout.

# Basic conversion
curl -s https://example.com | npx @mdream/js

# With origin URL for resolving relative paths
curl -s https://example.com | npx @mdream/js --origin https://example.com

# With minimal preset
curl -s https://example.com | npx @mdream/js --origin https://example.com --preset minimal

CLI Options

FlagDescription
--origin <url>Origin URL for resolving relative image paths and links
--preset <preset>Conversion preset. Currently supports: minimal
-v, --versionShow version number
-h, --helpShow help

Usage Examples

Custom Tag Overrides

Map custom HTML elements to standard Markdown behavior:

import { htmlToMarkdown } from '@mdream/js'

const md = htmlToMarkdown('<x-heading>Title</x-heading>', {
  plugins: {
    tagOverrides: {
      // String alias: make <x-heading> behave like <h2>
      'x-heading': 'h2',

      // Object override: custom enter/exit strings
      'callout': {
        enter: '> **Note:** ',
        exit: '\n',
        spacing: [2, 2],
      },
    },
  },
})

Declarative Extraction

Extract data from elements during conversion:

import { htmlToMarkdown } from '@mdream/js'

const images: { src: string, alt: string }[] = []

const md = htmlToMarkdown(html, {
  plugins: {
    extraction: {
      'img[alt]': (element) => {
        images.push({
          src: element.attributes.src,
          alt: element.attributes.alt,
        })
      },
    },
  },
})

console.log('Found images:', images)

Frontmatter with Callback

Receive structured frontmatter data for further processing:

import { htmlToMarkdown } from '@mdream/js'

let metadata: Record<string, string> = {}

const md = htmlToMarkdown(html, {
  plugins: {
    frontmatter: {
      additionalFields: { source: 'crawler' },
      metaFields: ['robots'],
      onExtract: (fm) => {
        metadata = fm
      },
    },
  },
})

console.log('Title:', metadata.title)
console.log('Description:', metadata.description)

Custom Hooks for Content Filtering

import { htmlToMarkdown } from '@mdream/js'
import { createPlugin } from '@mdream/js/plugins'

const md = htmlToMarkdown(html, {
  hooks: [
    createPlugin({
      beforeNodeProcess(event) {
        const { node } = event
        if (node.type === 1 /* ELEMENT_NODE */) {
          const el = node as any
          // Skip ads and promotional content
          if (el.attributes?.class?.includes('ad')
            || el.attributes?.id?.includes('promo')) {
            return { skip: true }
          }
        }
      },
      processTextNode(textNode) {
        // Uppercase all TODO markers
        if (textNode.value.includes('TODO')) {
          return { content: textNode.value.toUpperCase(), skip: false }
        }
      },
    }),
  ],
})

Token-Based Chunking for LLMs

import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
import { encoding_for_model } from 'tiktoken'

const enc = encoding_for_model('gpt-4')

const chunks = htmlToMarkdownSplitChunks(html, {
  chunkSize: 512,
  chunkOverlap: 64,
  lengthFunction: text => enc.encode(text).length,
  origin: 'https://example.com',
})

Exported Types

import type {
  BuiltinPlugins,
  CleanOptions,
  ElementNode,
  EngineOptions,
  ExtractedElement,
  FrontmatterConfig,
  MarkdownChunk,
  MdreamOptions,
  Node,
  NodeEvent,
  PluginContext,
  SplitterOptions,
  TagOverride,
  TextNode,
  TransformPlugin,
} from '@mdream/js'

Exported Constants

import {
  ELEMENT_NODE, // Node type for HTML elements (1)
  NodeEventEnter, // Event type for entering a node (0)
  NodeEventExit, // Event type for exiting a node (1)
  TAG_H1, // Tag ID constant for <h1>
  TAG_H2, // Tag ID constant for <h2>
  TAG_H3, // Tag ID constant for <h3>
  TAG_H4, // Tag ID constant for <h4>
  TAG_H5, // Tag ID constant for <h5>
  TAG_H6, // Tag ID constant for <h6>
  TEXT_NODE, // Node type for text content (3)
} from '@mdream/js'

FAQs

Package last updated on 20 May 2026

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts