notion-md-crawler

notion-md-crawler

A library to recursively retrieve and serialize Notion pages and databases with customization for machine learning applications.

🌟 Features

• 🕷️ Crawling Pages and Databases: Dig deep into Notion's hierarchical structure with ease.
• 📝 Serialize to Markdown: Seamlessly convert Notion pages to Markdown for easy use in machine learning and other.
• 🛠️ Custom Serialization: Adapt the serialization process to fit your specific machine learning needs.
• ⏳ Async Generator: Yields results on a page-by-page basis, so even huge documents can be made memory efficient.

🛠️ Installation

@notionhq/client must also be installed.

Using npm 📦:

npm install notion-md-crawler @notionhq/client

Using yarn 🧶:

yarn add notion-md-crawler @notionhq/client

Using pnpm 🚀:

pnpm add notion-md-crawler @notionhq/client

🚀 Quick Start

⚠️ Note: Before getting started, create an integration and find the token. Details on methods can be found in API section

Leveraging the power of JavaScript generators, this library is engineered to handle even the most extensive Notion documents with ease. It's designed to yield results page-by-page, allowing for efficient memory usage and real-time processing.

import { Client } from "@notionhq/client";
import { crawler, pageToString } from "notion-md-crawler";

// Need init notion client with credential.
const client = new Client({ auth: process.env.NOTION_API_KEY });

const crawl = crawler({ client });

const main = async () => {
  const rootPageId = "****";
  for await (const result of crawl(rootPageId)) {
    if (result.success) {
      const pageText = pageToString(result.page);
      console.log(pageText);
    }
  }
};

main();

🌐 API

crawler

Recursively crawl the Notion Page. dbCrawler should be used if the Root is a Notion Database.

Note: It tries to continue crawling as much as possible even if it fails to retrieve a particular Notion Page.

Parameters:

• options (CrawlerOptions): Crawler options.
• rootPageId (string): Id of the root page to be crawled.

Returns:

• AsyncGenerator<CrawlingResult>: Crawling results with failed information.

dbCrawler

Recursively crawl the Notion Database. crawler should be used if the Root is a Notion Page.

Parameters:

• options (CrawlerOptions): Crawler options.
• rootDatabaseId (string): Id of the root page to be crawled.

Returns:

• AsyncGenerator<CrawlingResult>: Crawling results with failed information.

CrawlerOptions

Option	Description	Type	Default
client	Instance of Notion Client. Set up an instance of the Client class imported from @notionhq/client.	Notion Client	-
serializers?	Used for custom serialization of Block and Property objects.	Object	undefined
serializers?.block?	Map of Notion block type and BlockSerializer.	BlockSerializers	undefined
serializers?.property?	Map of Notion Property Type and PropertySerializer.	PropertySerializers	undefined
metadataBuilder?	The metadata generation process can be customize.	MetadataBuilder	undefined
urlMask?	If specified, the url is masked with the string.	string | false	false
skipPageIds?	List of page Ids to skip crawling (also skips descendant pages)	string[]	undefined

BlockSerializers

Map with Notion block type (like "heading_1", "to_do", "code") as key and BlockSerializer as value.

BlockSerializer

BlockSerializer that takes a Notion block object as argument. Returning false will skip serialization of that Notion block.

[Type]

type BlockSerializer = (
  block: NotionBlock,
) => string | false | Promise<string | false>;

PropertySerializers

Map with Notion Property Type (like "heading_1", "to_do", "code") as key and PropertySerializer as value.

PropertySerializer

PropertySerializer that takes a Notion property object as argument. Returning false will skip serialization of that Notion property.

[Type]

type PropertySerializer = (
  name: string,
  block: NotionBlock,
) => string | false | Promise<string | false>;

MetadataBuilder

Retrieving metadata is sometimes very important, but the information you want to retrieve will vary depending on the context. MetadataBuilder allows you to customize it according to your use case.

[Example]

import { crawler, MetadataBuilderParams } from "notion-md-crawler";

const getUrl = (id: string) => `https://www.notion.so/${id.replace(/-/g, "")}`;

const metadataBuilder = ({ page }: MetadataBuilderParams) => ({
  url: getUrl(page.metadata.id),
});

const crawl = crawler({ client, metadataBuilder });

for await (const result of crawl("notion-page-id")) {
  if (result.success) {
    console.log(result.page.metadata.url); // "https://www.notion.so/********"
  }
}

📊 Use Metadata

Since crawler returns Page objects and Page object contain metadata, you can be used it for machine learning.

🛠️ Custom Serialization

notion-md-crawler gives you the flexibility to customize the serialization logic for various Notion objects to cater to the unique requirements of your machine learning model or any other use case.

Define your custom serializer

You can define your own custom serializer. You can also use the utility function for convenience.

import { BlockSerializer, crawler, serializer } from "notion-md-crawler";

const customEmbedSerializer: BlockSerializer<"embed"> = (block) => {
  if (block.embed.url) return "";

  // You can use serializer utility.
  const caption = serializer.utils.fromRichText(block.embed.caption);

  return `<figure>
  <iframe src="${block.embed.url}"></iframe>
  <figcaption>${caption}</figcaption>
</figure>`;
};

const serializers = {
  block: {
    embed: customEmbedSerializer,
  },
};

const crawl = crawler({ client, serializers });

Skip serialize

Returning false in the serializer allows you to skip the serialize of that block. This is useful when you want to omit unnecessary information.

const image: BlockSerializer<"image"> = () => false;
const crawl = crawler({ client, serializers: { block: { image } } });

Advanced: Use default serializer in custom serializer

If you want to customize serialization only in specific cases, you can use the default serializer in a custom serializer.

import { BlockSerializer, crawler, serializer } from "notion-md-crawler";

const defaultImageSerializer = serializer.block.defaults.image;

const customImageSerializer: BlockSerializer<"image"> = (block) => {
  // Utility function to retrieve the link
  const { title, href } = serializer.utils.fromLink(block.image);

  // If the image is from a specific domain, wrap it in a special div
  if (href.includes("special-domain.com")) {
    return `<div class="special-image">
      ${defaultImageSerializer(block)}
    </div>`;
  }

  // Use the default serializer for all other images
  return defaultImageSerializer(block);
};

const serializers = {
  block: {
    image: customImageSerializer,
  },
};

const crawl = crawler({ client, serializers });

🔍 Supported Blocks and Database properties

Blocks

Block Type	Supported
Text	✅ Yes
Bookmark	✅ Yes
Bulleted List	✅ Yes
Numbered List	✅ Yes
Heading 1	✅ Yes
Heading 2	✅ Yes
Heading 3	✅ Yes
Quote	✅ Yes
Callout	✅ Yes
Equation (block)	✅ Yes
Equation (inline)	✅ Yes
Todos (checkboxes)	✅ Yes
Table Of Contents	✅ Yes
Divider	✅ Yes
Column	✅ Yes
Column List	✅ Yes
Toggle	✅ Yes
Image	✅ Yes
Embed	✅ Yes
Video	✅ Yes
Figma	✅ Yes
PDF	✅ Yes
Audio	✅ Yes
File	✅ Yes
Link	✅ Yes
Page Link	✅ Yes
External Page Link	✅ Yes
Code (block)	✅ Yes
Code (inline)	✅ Yes

Database Properties

Property Type	Supported
Checkbox	✅ Yes
Created By	✅ Yes
Created Time	✅ Yes
Date	✅ Yes
Email	✅ Yes
Files	✅ Yes
Formula	✅ Yes
Last Edited By	✅ Yes
Last Edited Time	✅ Yes
Multi Select	✅ Yes
Number	✅ Yes
People	✅ Yes
Phone Number	✅ Yes
Relation	✅ Yes
Rich Text	✅ Yes
Rollup	✅ Yes
Select	✅ Yes
Status	✅ Yes
Title	✅ Yes
Unique Id	✅ Yes
Url	✅ Yes
Verification	□ No

💬 Issues and Feedback

For any issues, feedback, or feature requests, please file an issue on GitHub.

📜 License

MIT

Made with ❤️ by TomPenguin.