wikimem

wikimem

Self-improving wiki IDE. Ingest anything. Query with any LLM. Three automations.

npx wikimem@latest

What is WikiMem?

WikiMem takes Karpathy's LLM wiki concept and turns it into a full IDE. Drop any file — PDF, audio, video, slides, spreadsheet, URL — and watch it compile into structured, interlinked wiki pages via Claude, GPT-4o, or Ollama. Three automations keep your knowledge base growing and self-improving while you sleep.

raw/                        wiki/
  paper.pdf                   index.md ........... content catalog
  podcast.mp3    ──LLM──>    sources/paper.md ... summary + citations
  screenshot.png              entities/openai.md . people, orgs, tools
  meeting.docx                concepts/rag.md .... ideas + frameworks
  blog-url                    syntheses/ ......... cross-cutting analysis

Quick Start

# Create a vault and start the IDE
npx wikimem init my-wiki
cd my-wiki
npx wikimem serve

Open http://localhost:3141. That's it — you have a running wiki IDE.

# Or ingest from the CLI
wikimem ingest paper.pdf
wikimem ingest https://en.wikipedia.org/wiki/Large_language_model
wikimem query "What are the key themes across my sources?"

Features

13+ Format Ingestion

Drop anything. WikiMem detects the file type, runs the right processor, and produces wiki pages with cross-references and citations.

Format	Extensions	Processor
Text	.md, .txt	Direct read
Structured	.json, .csv, .yaml	Schema-aware extraction
PDF	.pdf	Built-in text extraction
Office	.docx, .pptx, .xlsx	Document parsing
HTML	.html, .htm	Tag stripping + content extraction
Image	.png, .jpg, .gif, .webp	Claude Vision description
Audio	.mp3, .wav, .m4a, .ogg, .flac	Whisper / Deepgram transcription
Video	.mp4, .mov, .avi, .mkv, .webm	ffmpeg → Whisper transcription
URL	https://...	Firecrawl / fetch → markdown

Knowledge Graph

D3-powered interactive force-directed graph. Click a node to highlight its neighbors, double-click to open. Community detection clusters related pages. Hub nodes sized by connection count.

Time-Lapse

Watch your knowledge base grow commit-by-commit. Every wiki change is checkpointed in git — scrub through the timeline to see pages appear, links form, and the graph densify.

WYSIWYG Editing

Click any wiki page to edit it inline. Markdown shortcuts, live preview, Cmd+S to save. Changes are auto-committed to git.

Three Automations

Automation	Trigger	What it does
Ingest	File watcher on raw/	New file detected → process → wiki pages → git commit
Scrape	Cron schedule or manual	RSS feeds, GitHub trending, URLs → fetch → deposit in raw/ → triggers Ingest
Observe	Nightly or manual	LLM Council scores wiki quality (coverage, consistency, cross-linking, freshness, organization) → proposes and applies improvements

Git Checkpointing

Every change committed automatically. Browse history, restore snapshots, see diffs. Your wiki is a git repo from day one.

Pipeline Visualization

See exactly how your document flows through the system — file detection, text extraction, LLM processing, page generation, cross-linking, indexing — step by step in the web UI.

Connectors

Sync external sources into your vault automatically.

Connector	Status
Local folders	✅ Shipped
Git repos	✅ Shipped
GitHub	✅ Shipped
Webhooks	✅ Shipped
Slack	🔜 Coming soon
Gmail	🔜 Coming soon

MCP Server

Use WikiMem as a tool inside Claude Code, Cursor, or any MCP-compatible client.

wikimem mcp

Multiple LLMs

Provider	Flag	Default Model
Claude	-p claude	claude-sonnet-4-20250514
OpenAI	-p openai	gpt-4o
Ollama	-p ollama	llama3.2

Ollama runs fully local — no API keys, no network, no data leaves your machine.

CLI Reference

Command	Description
wikimem init [dir]	Create a new vault (--template research|business|codebase, --from-folder, --from-repo)
wikimem serve	Start the web IDE on port 3141
wikimem ingest <source>	Process a file or URL into wiki pages
wikimem search <term>	BM25 full-text search across wiki pages
wikimem ask <question>	Ask a question, get an answer from your wiki
wikimem query <question>	Ask a question and optionally save as synthesis page (--file)
wikimem lint	Health-check: orphan pages, broken links, missing summaries (--fix)
wikimem status	Vault statistics: pages, words, sources, links, orphans
wikimem watch	Auto-ingest files dropped into raw/
wikimem scrape	Fetch from configured RSS/GitHub/URL sources
wikimem improve	Run self-improvement cycle (--dry-run, --threshold 90)
wikimem export	Export wiki to other formats
wikimem open	Open vault in Obsidian
wikimem history	Browse audit trail, restore snapshots
wikimem mcp	Start MCP server for Claude Code / Cursor
wikimem publish	Publish wiki as static site, RSS, JSON feed, or digest (--format html,rss,json-feed,digest)
wikimem duplicates	Detect and manage near-duplicate sources

Web UI

wikimem serve opens a full IDE at localhost:3141:

• File tree — browse wiki pages with collapsible folders
• Tabbed editor — open multiple pages, WYSIWYG markdown editing
• Knowledge graph — interactive D3 force-directed visualization
• Pipeline view — drag-and-drop file ingestion with step-by-step progress
• Time-lapse — scrub through git history to watch your wiki grow
• Search — Cmd+K fuzzy search across all pages
• Command palette — Cmd+P for quick actions
• Settings — configure API keys, models, and automations from the UI
• Ask your knowledge — query your wiki from the browser

MCP Server

WikiMem ships with a built-in MCP server so Claude Code and Cursor can read, search, and query your wiki directly.

Add to Claude Code (.mcp.json):

{
  "mcpServers": {
    "wikimem": {
      "command": "npx",
      "args": ["-y", "wikimem", "mcp"],
      "env": {
        "WIKIMEM_VAULT": "/path/to/your/vault"
      }
    }
  }
}

Or run standalone:

wikimem-mcp

Configuration

After wikimem init, your vault contains config.yaml:

provider: claude                    # claude | openai | ollama
model: claude-sonnet-4-20250514

sources:
  - name: "HN Front Page"
    type: rss
    url: "https://hnrss.org/frontpage"

  - name: "GitHub Trending TS"
    type: github
    query: "stars:>100 created:>7d language:typescript"

improvement:
  threshold: 80
  schedule: "0 3 * * *"            # 3am nightly

Environment Variables

Variable	Purpose
ANTHROPIC_API_KEY	Claude API access (default provider)
OPENAI_API_KEY	OpenAI API access
OLLAMA_BASE_URL	Ollama server URL (default: http://localhost:11434)
FIRECRAWL_API_KEY	Enhanced URL-to-markdown (optional)
DEEPGRAM_API_KEY	Audio transcription (optional, falls back to Whisper)

Architecture

vault/
├── wiki/           ← LLM-generated pages (sources/, entities/, concepts/, syntheses/)
├── raw/            ← Immutable source documents (date-stamped subdirectories)
├── AGENTS.md       ← Schema — wiki structure + conventions
├── config.yaml     ← Configuration — provider, sources, schedules
└── index.md        ← Content catalog (auto-maintained)

Three layers: raw/ (immutable sources) → LLM processing → wiki/ (structured knowledge). AGENTS.md is the schema file that tells the LLM how to structure output — it co-evolves with your wiki.

Three automations: Ingest (file watcher → process → wiki pages), Scrape (RSS/GitHub/URLs → raw/), Observe (LLM Council → score → improve).

Obsidian Integration

WikiMem vaults are Obsidian vaults. Open any wikimem directory in Obsidian — no plugins, no configuration:

• [[wikilinks]] rendered as backlinks
• YAML frontmatter as page metadata
• Graph view showing all connections
• Tag view from frontmatter tags: arrays

Privacy

• Everything runs locally. Your wiki is a folder of markdown files.
• No data sent anywhere except LLM API calls (and those are optional with Ollama).
• raw/ excluded from git by default — your source documents stay private.
• config.yaml excluded from git — API keys never committed.

Path	Safe to commit?	Why
wiki/	✅	LLM-generated summaries, no raw personal data
AGENTS.md	✅	Schema file, no personal data
raw/	❌	Original source files
config.yaml	❌	May contain API keys

Credits

Inspired by Andrej Karpathy's LLM Wiki pattern — the idea that LLMs should compile knowledge into structured, interlinked wikis rather than just answering questions from raw chunks.

Built with Express, D3, simple-git, and the Claude / OpenAI / Ollama APIs.

License

MIT — see LICENSE.