@mdedit/markdown-parser

@mdedit/markdown-parser

AST-based markdown parsing utilities for the mdedit project.

Overview

This package provides utilities for parsing and transforming markdown content using Abstract Syntax Trees (AST) instead of manual line-by-line parsing. It leverages the mdast (Markdown Abstract Syntax Tree) ecosystem including:

• mdast-util-from-markdown - Parse markdown to AST
• mdast-util-to-markdown - Serialize AST back to markdown
• mdast-util-gfm - GitHub Flavored Markdown support
• mdast-util-frontmatter - YAML and TOML frontmatter support
• micromark-extension-frontmatter - Frontmatter tokenization
• unist-util-visit - Traverse and manipulate AST nodes

Benefits

The AST-based approach provides several advantages over manual parsing:

• More Reliable: Handles edge cases and complex markdown structures correctly
• Maintainable: Clear, declarative code that's easier to understand and modify
• Extensible: Easy to add new transformations or extractions
• Standards-Compliant: Uses the unified/remark ecosystem standards
• Type-Safe: Written in TypeScript with full type definitions
• Frontmatter-Aware: Preserves YAML and TOML frontmatter during transformations

API

extractCodeBlocks(markdown: string): CodeBlock[]

Extracts all code blocks from markdown content.

const blocks = extractCodeBlocks(markdown);
blocks.forEach(block => {
  console.log(`Language: ${block.lang}, Code: ${block.value}`);
});

extractHeadings(markdown: string): Heading[]

Extracts all headings from markdown content.

const headings = extractHeadings(markdown);
headings.forEach(heading => {
  console.log(`Level ${heading.depth}: ${heading.text}`);
});

getTextWithoutCodeBlocks(markdown: string): string

Returns the markdown content with all code blocks removed.

const text = getTextWithoutCodeBlocks(markdown);

transformCodeBlocks(markdown: string, transformer: (code: string, lang: string) => Promise<string>): Promise<string>

Transforms all code blocks in the markdown using the provided transformer function.

const result = await transformCodeBlocks(markdown, async (code, lang) => {
  // Transform code to screenshot
  const screenshot = await codeToScreenshot(code, lang);
  return screenshot;
});

transformHeadings(markdown: string, transformer: (text: string, depth: number) => string): string

Transforms all heading text using the provided transformer function.

const result = transformHeadings(markdown, (text, depth) => {
  return titleCase(text);
});

generateTableOfContents(markdown: string, options?: { maxDepth?: number }): string

Generates a table of contents from the markdown headings.

const toc = generateTableOfContents(markdown, { maxDepth: 3 });

Migration Guide

Before (Manual Parsing)

const lines = article.split("\n");
let codeBlockStarted = false;
let codeBlock = "";

for (let i = 0; i < lines.length; i++) {
  const line = lines[i];
  if (line.startsWith("```")) {
    codeBlockStarted = !codeBlockStarted;
  } else if (codeBlockStarted) {
    codeBlock += line + "\n";
  }
}

After (AST-Based)

import { extractCodeBlocks } from '@mdedit/markdown-parser';

const blocks = extractCodeBlocks(article);
blocks.forEach(block => {
  const codeBlock = block.value;
  // Process code block
});

Refactored Functions

The following functions in apps/app/src/utils/articleUtils.ts have been refactored to use AST-based parsing:

• cleanupIpynbOutput - Uses transformCodeBlocks
• detectCodeSnippetLanguages - Uses transformCodeBlocks
• codePrettify - Uses transformCodeBlocks
• codeScreenShots - Uses transformCodeBlocks
• gistify - Uses transformCodeBlocks
• getTextWithoutCodeBlocks - Uses getTextWithoutCodeBlocks
• convertHeadingsToTitleCase - Uses transformHeadings
• generateToC - Uses generateTableOfContents

Testing

Run the manual tests:

cd packages/markdown-parser
ts-node src/test-manual.ts