Beautiful Command-Line Interface for Documentation Scraping
Transform any documentation site into AI-ready Markdown with an elegant developer experience
The Aimdoc CLI transforms the complex process of documentation scraping into a beautiful, interactive experience:
🎯 Zero Configuration - Just run aimdoc scrape and follow the prompts
⚡ Real-Time Progress - Watch your documentation being scraped with live updates
🎨 Beautiful Interface - Elegant spinners, progress bars, and colored output
🔍 Smart Diagnostics - Detailed error reporting when things go wrong
📁 Organized Output - Clean folder structure with automatic README generation
# Using npm
npm install -g aimdoc
# Using pnpm
pnpm install -g aimdoc
# Using yarn
yarn global add aimdoc
# Clone and install
git clone https://github.com/clemeverger/aimdoc.git
cd aimdoc/cli
npm install
# Build and link locally
npm run build
npm link
# Or run directly with ts-node
npm run dev scrape --help
aimdoc scrape - Scrape DocumentationThe main command that does all the magic ✨
# Interactive mode - just follow the prompts!
aimdoc scrape
# Specify URL directly
aimdoc scrape https://nextjs.org/docs
# Full customization
aimdoc scrape https://react.dev \
--name "React Official Docs" \
--output-dir ./references/react
Options:
[url] - Documentation site URL (optional, will prompt if not provided)-n, --name <name> - Project name (defaults to domain name)-o, --output-dir <dir> - Output directory (defaults to ./docs)What happens:
aimdoc jobs - List All JobsSee all your scraping jobs at a glance:
# List all jobs
aimdoc jobs
# Show only recent jobs
aimdoc jobs --limit 10
Output:
📋 Recent Scraping Jobs
✅ nextjs-docs (a1b2c3d4) Completed 2 hours ago 247 files
🔄 react-docs (e5f6g7h8) Running Started 5m ago
❌ vue-docs (i9j0k1l2) Failed 1 day ago Connection timeout
⏳ tailwind-docs (m3n4o5p6) Pending Just now
aimdoc download - Download Job ResultsDownload results from a previous job:
# Download specific job
aimdoc download a1b2c3d4-e5f6-7890-abcd-ef1234567890
# Specify custom output directory
aimdoc download a1b2c3d4-e5f6-7890-abcd-ef1234567890 --output-dir ./my-docs
aimdoc diagnose - Debug Failed JobsGet detailed diagnostics for failed or problematic scrapes:
# Basic diagnosis
aimdoc diagnose a1b2c3d4-e5f6-7890-abcd-ef1234567890
# Verbose output with failed pages
aimdoc diagnose a1b2c3d4-e5f6-7890-abcd-ef1234567890 --verbose
Sample Output:
=== Job Diagnosis: a1b2c3d4 ===
Status: ❌ FAILED
Created: 2024-01-15T10:30:00Z
Started: 2024-01-15T10:30:05Z
=== Results Summary ===
📄 Files created: 0
✅ Pages scraped: 0
❌ Pages failed: 5
🔍 Pages discovered: Unknown
=== Detailed Diagnostics ===
❌ Discovery Errors (2):
1. https://example.com/robots.txt
Error: DNSLookupError: DNS lookup failed
2. https://example.com/sitemap.xml
Error: TimeoutError: Request timed out
💡 Tip: This website appears to have no sitemap or an inaccessible sitemap.
Consider trying a different website that has a sitemap.xml file.
aimdoc config - Configuration ManagementManage CLI configuration and API connection:
# Show current configuration
aimdoc config show
# Set API server URL
aimdoc config set api-url http://localhost:8000
# Reset to defaults
aimdoc config reset
The CLI is designed to be beautiful and intuitive. Here's what you'll see:
? Documentation URL: https://nextjs.org/docs
? Project name: (nextjs) Next.js Official
? Output directory: (./docs) ./references/nextjs
✓ Job created: a1b2c3d4-e5f6-7890-abcd-ef1234567890
✓ Connected to job
⠋ Discovering sitemap and pages...
✓ Found 247 pages to scrape
Scraping |████████████████████████████████████████| 247/247 pages
✓ Scraping completed!
⠋ Converting to markdown... 247 files created
✓ Converting completed!
⠋ Downloading and organizing files...
✓ Downloaded all 247 files
✅ Scraping completed successfully!
✓ Created 247 files from 247 pages
ℹ️ All 247 discovered pages scraped successfully
📁 Documentation organized in: ./references/nextjs
✓ Generated README.md index with 247 files
The CLI stores configuration in:
~/.config/aimdoc/config.yaml~/.config/aimdoc/config.yaml%APPDATA%\aimdoc\config.yamlapi:
url: http://localhost:8000
timeout: 30000
output:
default_dir: ./docs
create_readme: true
display:
show_progress: true
use_colors: true
verbose: false
# Override API server URL
export AIMDOC_API_URL=http://localhost:8000
# Set default output directory
export AIMDOC_OUTPUT_DIR=./my-docs
# Disable colors (for CI environments)
export NO_COLOR=1
When you scrape documentation, the CLI creates a clean, organized structure:
docs/
└── nextjs/ # Project name
├── README.md # Auto-generated index
├── getting-started/
│ ├── installation.md
│ └── quick-start.md
├── app-router/
│ ├── routing.md
│ ├── pages.md
│ └── layouts.md
└── api-reference/
├── components.md
└── functions.md
# Documentation Index
Generated on 2024-01-15T14:30:00.000Z
## Structure
- [installation](./getting-started/installation.md)
- [quick-start](./getting-started/quick-start.md)
### app-router
- [routing](./app-router/routing.md)
- [pages](./app-router/pages.md)
- [layouts](./app-router/layouts.md)
### api-reference
- [components](./api-reference/components.md)
- [functions](./api-reference/functions.md)
---
_Generated with [aimdoc](https://github.com/clemeverger/aimdoc)_
The CLI communicates with the Aimdoc API server. Make sure it's running:
# Start the API server (from project root)
python start_api.py
# Check if it's running
curl http://localhost:8000/health
The CLI uses WebSocket for real-time updates during scraping:
# Get the latest Next.js docs for your AI assistant
aimdoc scrape https://nextjs.org/docs -n "NextJS-Latest" -o ./ai-context
# Scrape multiple frameworks for comparison
aimdoc scrape https://react.dev -n "React" -o ./frameworks
aimdoc scrape https://vuejs.org/guide -n "Vue" -o ./frameworks
aimdoc scrape https://svelte.dev/docs -n "Svelte" -o ./frameworks
# Monitor competitor documentation
aimdoc scrape https://competitor.com/docs -n "Competitor-Docs"
# Archive documentation versions
aimdoc scrape https://v4.react.dev -n "React-v4" -o ./archives
# Create local documentation library
mkdir ~/dev-docs
aimdoc scrape https://docs.python.org -o ~/dev-docs
aimdoc scrape https://docs.rust-lang.org -o ~/dev-docs
aimdoc scrape https://golang.org/doc -o ~/dev-docs
❌ Command not found: aimdoc
# Make sure it's installed globally
npm list -g aimdoc
# Or install it
npm install -g aimdoc
❌ API connection failed
# Check if API server is running
curl http://localhost:8000/health
# Start the API server
cd .. && python start_api.py
❌ Permission denied writing files
# Check directory permissions
ls -la ./docs
# Create directory with proper permissions
mkdir -p ./docs && chmod 755 ./docs
❌ WebSocket connection failed
# This is usually due to API server not running
# Check the server logs for WebSocket errors
# Run with debug logging
DEBUG=aimdoc:* aimdoc scrape https://example.com
# Or use verbose flag
aimdoc scrape https://example.com --verbose
# Show help for any command
aimdoc --help
aimdoc scrape --help
aimdoc diagnose --help
# Show version
aimdoc --version
/docs/ section (that's what we target)# 1. Test with a small site first
aimdoc scrape https://small-docs-site.com
# 2. Use diagnose to understand any issues
aimdoc diagnose <job-id> --verbose
# 3. Scale up to larger documentation sites
aimdoc scrape https://large-docs-site.com
# Check current version
aimdoc --version
# Check for updates (npm)
npm outdated -g aimdoc
# Update to latest version
npm update -g aimdoc
# Clone and setup
git clone https://github.com/clemeverger/aimdoc.git
cd aimdoc/cli
# Install dependencies
npm install
# Start development server
npm run dev
# Build for production
npm run build
# Run tests
npm test
# Run tests in watch mode
npm run test:watch
# Test CLI commands
npm run dev scrape --help
MIT License - see the LICENSE file for details.
Built with ❤️ for developers who love beautiful CLIs
FAQs
Command line interface for Aimdoc API
We found that aimdoc demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Multiple high-impact npm maintainers confirm they have been targeted in the same social engineering campaign that compromised Axios.
Security News
Axios compromise traced to social engineering, showing how attacks on maintainers can bypass controls and expose the broader software supply chain.
Security News
Node.js has paused its bug bounty program after funding ended, removing payouts for vulnerability reports but keeping its security process unchanged.