@jldb/web-to-md

🕷️ Web-to-MD: Your Friendly Neighborhood Web Crawler and Markdown Converter 🕸️

Welcome to Web-to-MD, the CLI tool that turns websites into your personal Markdown library! 🚀

🌟 Why Web-to-MD?

Ever wished you could magically transform entire websites into neatly organized Markdown files? Well, wish no more! Web-to-MD is here to save the day (and your sanity)!

🎭 Features That'll Make You Go "Wow!"

• 🔍 Crawls websites like a pro detective
• 🧙‍♂️ Magically transforms HTML into beautiful Markdown
• 🏃‍♂️ Resumes interrupted crawls (because life happens!)
• 📚 Creates separate Markdown files or one big book of knowledge
• 🎨 Shows fancy progress bars (because who doesn't love those?)
• 🚦 Respects rate limits (we're polite crawlers here!)
• 🌳 Preserves directory structure (if you're into that sort of thing)
• 🔒 Handles authentication gracefully (no trespassing allowed!)
• 👥 Multi-worker support (because teamwork makes the dream work!)
• 🔄 Smart content change detection (no need to crawl what hasn't changed!)

🛠️ Installation

1. Clone this repo (it won't bite, promise!)
2. Run npm install (sit back and watch the magic happen)
3. Run npm run build to compile the TypeScript code

🚀 Usage

Fire up Web-to-MD with this incantation:

npm start -- -u <url> -o <output_directory> [options]

🎛️ Options (Mix and Match to Your Heart's Content)

• -u, --url <url>: The URL of your web treasure trove (required)
• -o, --output <output>: Where to stash your Markdown gold (required)
• -c, --combine: Merge all pages into one massive scroll of knowledge
• -e, --exclude <paths>: Comma-separated list of paths to skip (shh, we won't tell)
• -r, --rate <rate>: Max pages per second (default: 5, for the speed demons)
• -d, --depth <depth>: How deep should we dig? (default: 3, watch out for dragons)
• -m, --max-file-size <size>: Max file size in MB for combined output (default: 2)
• -n, --name <name>: Name your combined file (get creative!)
• -p, --preserve-structure: Keep the directory structure (for the neat freaks)
• -t, --timeout <timeout>: Timeout in seconds for page navigation (default: 3.5)
• -i, --initial-timeout <initialTimeout>: Initial timeout for the first page (default: 60)
• -re, --retries <retries>: Number of retries for initial page load (default: 3)
• -w, --workers <workers>: Number of concurrent workers (default: 1, for the multitaskers)

🌟 Example (Because We All Need a Little Guidance)

npm start -- -u https://docs.example.com -o ./my_docs -c -d 5 -r 3 -n "ExampleDocs" -w 3

This will:

1. Crawl https://docs.example.com
2. Save Markdown files to ./my_docs
3. Combine all pages into one file
4. Crawl up to 5 levels deep
5. Respect a rate limit of 3 pages per second
6. Name the combined file "ExampleDocs"
7. Use 3 concurrent workers for faster crawling

🔧 Config Magic: Resuming and Customizing Your Crawls

Web-to-MD comes with a nifty config feature that lets you resume interrupted crawls and customize your crawling experience. Here's how it works:

📁 Config File

After a crawl (complete or interrupted), Web-to-MD saves a config.json file in your output directory. This file contains all the settings and state information from your last crawl.

🔄 Resuming a Crawl

To resume an interrupted crawl, simply run Web-to-MD with the same output directory. The tool will automatically detect the config.json file and pick up where it left off.

🎛️ Customizing Your Crawl

You can manually edit the config.json file to customize your next crawl. Here are the available options and their default values:

Option	Description	Default Value
url	Starting URL for the crawl	(Required)
outputDir	Output directory for Markdown files	(Required)
excludePaths	Paths to exclude from crawling	[]
maxPagesPerSecond	Maximum pages to crawl per second	5
maxDepth	Maximum depth to crawl	3
maxFileSizeMB	Maximum file size in MB for combined output	2
combine	Combine all pages into a single file	false
name	Name for the combined output file	undefined
preserveStructure	Preserve directory structure	false
timeout	Timeout in seconds for page navigation	3.5
initialTimeout	Initial timeout in seconds for the first page load	60
retries	Number of retries for initial page load	3
numWorkers	Number of concurrent workers	1

You can modify these settings in the config.json file to customize your crawl. For example:

{
  "settings": {
    "url": "https://docs.example.com",
    "outputDir": "./my_docs",
    "excludePaths": ["/blog", "/forum"],
    "maxPagesPerSecond": 5,
    "maxDepth": 4,
    "numWorkers": 3
  }
}

🌟 Example Workflow

1. Start an initial crawl:

   npm start -- -u https://docs.example.com -o ./my_docs -d 3 -w 2
   

2. If the crawl is interrupted, Web-to-MD will save the state in ./my_docs/config.json.

3. To resume, simply run:

   npm start -- -o ./my_docs
   

4. To customize, edit ./my_docs/config.json to change the crawl settings as needed. For example:

{
  "settings": {
    "url": "https://docs.example.com",
    "outputDir": "./my_docs",
    "excludePaths": ["/blog", "/forum"],
    "maxPagesPerSecond": 5,
    "maxDepth": 4,
    "numWorkers": 3
  }
}

1. Run the crawl again with the updated config:

   npm start -- -o ./my_docs
   

This workflow allows you to fine-tune your crawls and easily pick up where you left off!

🎭 Contributing

Got ideas? Found a bug? We're all ears! Open an issue or send a pull request. Let's make Web-to-MD even more awesome together! 🤝

📜 License

ISC (It's So Cool) License

🙏 Acknowledgements

A big thank you to all the open-source projects that made Web-to-MD possible. You rock! 🎸

Now go forth and crawl some docs! 🕷️📚