
Company News
AWS Security Hub Adds Socket for Supply Chain Security
Socket is now in the AWS Security Hub Extended plan. Adopt it through AWS, apply committed spend, and block malicious open source packages.
sourceharbor
Advanced tools
Turn YouTube, Bilibili, and RSS sources into readable digests, traceable job runs, and searchable knowledge for operators and agents.
Proof-first AI knowledge control tower for source intake, grounded search, and agent workflows, with one shared Web/API/MCP truth and strong YouTube/Bilibili lanes.
See It In 30 Seconds · Run Locally · No-Boot Tour · Docs Home · MCP Quickstart · Builders · Distribution Status · Starter Packs · Media Kit · Sample Corpus · Proof · Project Status · Why It Stands Out · Discussions
SourceHarbor helps you turn long-form sources into grounded search results, knowledge cards, traceable job runs, and MCP-ready operations. It stays source-first and proof-first: you can inspect it, run it locally, and verify each surface instead of trusting product copy on vibes alone.
Three quick reasons developers keep reading:
The honest intake boundary today is:
That intake split now lives behind one shared template catalog instead of
separate Web-only presets: the /subscriptions front door, HTTP API, and MCP
surface all point at the same strong-supported vs generalized intake contract.
It is strongest when you read it as a control tower for source intake:
SourceHarbor is a multi-surface product repo, not a single skill package.
Public starter packs and plugin-grade skill surfaces are adoption layers inside
that repo. They are not the whole product, and they are not raw exports of the
internal .agents/skills tree.
This is the part that makes SourceHarbor more than a pretty README:
| If you are... | What pulls you in | Why it feels different |
|---|---|---|
| A builder chasing Codex / Claude Code workflows | one repo already exposes MCP, HTTP API, and a shared operator truth | you do not have to invent a fake assistant shell just to reach real jobs, artifacts, and retrieval |
| An operator who cares about proof | job trace, ops inbox, watchlists, trends, and bundle exports all point back to the same pipeline | the repo keeps receipts, not just summaries |
| A maintainer deciding whether to contribute | the product story, runtime truth, and testing story now line up | you can tell what is real, what is gated, and what is still a deliberate bet without re-reading the whole archive |
The hook is simple:
You do not need every door on day one.
Pick the first path that matches why you are here:
| If you want to... | Open this first | Why this is the right first door |
|---|---|---|
| See whether the product is real | docs/see-it-fast.md, then docs/proof.md | start with the shop window, then inspect the proof ladder before you commit to a longer run |
| Run SourceHarbor as an operator | docs/start-here.md, then /subscriptions, /search, and /ops after local boot | this is the shortest truthful path from clone to intake, evidence, and triage |
| Build on top of SourceHarbor | docs/builders.md, docs/mcp-quickstart.md, and docs/public-distribution.md | these pages separate MCP, API, CLI, SDK, starter packs, and official-surface submission truth without mixing them into the newcomer path |
The fastest way to understand the product is to open the highest-value rooms first:
| Front door | What it means | Current truth |
|---|---|---|
| Subscriptions | Source-universe intake front door with one shared template catalog for strong-supported YouTube/Bilibili lanes plus generalized RSSHub/RSS intake | Real Web route after local boot: /subscriptions + shared catalog through API and MCP |
| Search | Operator-facing evidence search over digests, knowledge cards, transcripts, and related artifacts | Real Web route after local boot: /search |
| Ask your sources | Story-aware, briefing-backed Ask front door: with watchlist and story context it returns the current answer, recent changes, and citation drill-down through a server-owned page payload; without context it falls back to grounded retrieval | Real Web route after local boot: /ask + grounded contract |
| Briefings | Lowest-cognitive-load unified story view for one watchlist: summary first, then differences, then evidence drill-down, with one canonical selected-story page payload that Ask reuses instead of parallel browser-side aliases | Real Web route after local boot: /briefings; grounded in watchlists, merged stories, jobs, and knowledge |
| Watchlists | Durable tracking object surface for saved topics, claim kinds, platform slices, and source matchers | Real Web route after local boot: /watchlists |
| Trends | Compounder front door that turns repeated watchlist hits into merged stories plus recent evidence runs | Real Web route after local boot: /trends |
| MCP | Agent-facing surface on top of the same API and pipeline state | docs/mcp-quickstart.md + ./bin/dev-mcp |
| Ops / doctor | First-run diagnosis, operator triage, and next-step guidance for runtime truth, failed jobs, ingest issues, and live-hardening gates | ./bin/doctor + /ops after local boot + docs/runtime-truth.md |
| Playground | Clearly labeled sample-proof lane for demo corpus, example jobs, retrieval results, and use-case navigation without pretending to be live operator truth | Real Web route after local boot: /playground + docs/samples/README.md |
SourceHarbor is not just a Web app. It already has multiple access layers for builders and agent workflows:
The builder-facing mental map should follow the same product line:
/subscriptions establishes the intake contract/watchlists stores the tracking object/trends turns repeated runs into the compounder front door/briefings and /ask share the story-aware page payload/mcp reuses that same system truth for agents| Entry point | Who it is for | Current truth |
|---|---|---|
| Codex / Claude Code workflows | local operators who want an AI coding or operations agent to query and drive the same system truth | honest fit today through MCP + HTTP API, documented in docs/builders.md |
| OpenClaw workflows | builders who want a dedicated OpenClaw first hop instead of only generic MCP theory | first-cut fit today through docs/compat/openclaw.md plus starter-packs/openclaw/README.md |
| Plugin-grade bundles and official-surface templates | builders who want the strongest public bundles or distribution templates beyond docs-only starters | real today through starter-packs/codex/sourceharbor-codex-plugin/, starter-packs/claude-code/sourceharbor-claude-plugin/, starter-packs/openclaw/clawhub.package.template.json, and starter-packs/mcp-registry/sourceharbor-server.template.json; readiness differs by platform |
| Packaged public CLI | builders who want an installable command surface before memorizing repo entrypoints | real today in packages/sourceharbor-cli; it stays thinner than the repo-local runtime CLI and delegates inside a checkout |
| Repo-local CLI substrate | operators already inside a checkout who want the direct runtime and governance command truth | real today via ./bin/sourceharbor help, which remains the underlying local command truth |
| MCP surface | agent workflows and assistant clients that need governed access to jobs, artifacts, retrieval, ingest, reports, and notifications | real surface today via ./bin/dev-mcp |
| HTTP API contract | product builders, automation, and SDK consumers | real contract today via contracts/source/openapi.yaml |
| Public TypeScript SDK | TypeScript builders who want a thin client over the same HTTP contract | real today in packages/sourceharbor-sdk; it stays contract-first and builder-facing |
| Public starter packs | builders who want reproducible Codex / Claude Code / OpenClaw / SDK starting templates | available today as a first-cut surface in starter-packs/; these are public templates, starter skills, and compatibility notes, not raw internal .agents/skills exports |
The packaging story is intentionally thin: the packaged CLI stays repo-aware and
delegates to ./bin/sourceharbor inside a checkout, the TypeScript SDK stays a
contract-first wrapper over the HTTP API, and Python SDK support still stays
later.
Skill-repo criteria apply only partially here:
| Surface | Skill-repo criteria apply? | Why |
|---|---|---|
| Whole SourceHarbor repo | No | this repo ships Web, API, MCP, runtime, and builder layers together; it is a multi-surface product repo |
starter-packs/** | Yes, mostly | these are newcomer-facing starter surfaces with copyable prompts, templates, and clear adoption paths |
| plugin-grade bundle directories | Yes, strongly | these are the closest repo-owned artifacts to official marketplace or registry submission |
internal .agents/skills/** | No public applicability | they stay internal operating aids, not newcomer-facing exports |
Plugin-grade distribution is now one layer stronger than raw docs:
If you want the shortest truthful answer to "what already exists vs what still needs official submit/read-back proof," open docs/public-distribution.md.
These packages are the public box around the same repo-owned logic:
packages/sourceharbor-cli when you want one thin command surface for the repo-local command substrate.packages/sourceharbor-sdk when you want a typed HTTP client instead of inventing a second fetch stack.sourceharbor-mcp console script and the PyPI-shaped install artifact that the MCP Registry template now targets.starter-packs/README.md when you want reproducible Codex / Claude Code / OpenClaw / SDK starting templates rather than raw internal skill files; this surface is available today, but it is still first-cut.Registry ownership marker:
mcp-name: io.github.xiaojiou176-open/sourceharbor-mcp
Do not read every container-shaped artifact in this repo as a newcomer-facing product container.
| Container surface | What it is for | Current truth |
|---|---|---|
infra/compose/core-services.compose.yml | repo-local core services for Postgres and Temporal | local operator/runtime helper for SourceHarbor boot; not a public product container distribution |
.devcontainer/devcontainer.json + .devcontainer/Dockerfile | contributor workspace parity | local development environment for people working inside a checkout; not a packaged newcomer product artifact |
infra/docker/sourceharbor-api.Dockerfile + ghcr.io/xiaojiou176-open/sourceharbor-api | dedicated product API image route | builder-facing API container lane; the GHCR package URL now exists, but anonymous pull is still blocked while the package visibility remains private |
infra/config/strict_ci_contract.json + ghcr.io/xiaojiou176-open/sourceharbor-ci-standard | strict CI and devcontainer parity image | infrastructure image for CI, attestation, and repeatable tooling; not newcomer-facing product container distribution |
So the honest newcomer path stays:
docs/start-here.mdMinimal examples:
npm install --global ./packages/sourceharbor-cli
cd /path/to/sourceharbor
sourceharbor help
npm install ./packages/sourceharbor-sdk
uv build
Think of this as the label on the box, not fine print:
If you need the explicit bet boundaries, read:
Keep these truth layers separate when you read or share the repo:
main truth can move ahead of the latest release tagmain are already greenThat is why SourceHarbor keeps proof.md, project-status.md, runbook-local.md, and the public-reference docs as separate ledgers instead of one blanket “ready” claim.
These are the surfaces that make SourceHarbor reusable instead of one-and-done:
| Compounder | What it does | Current truth |
|---|---|---|
| Watchlists | Save a topic, claim kind, or source matcher as a durable tracking object | Real route: /watchlists |
| Trends | Compare recent matched runs for a watchlist and show what was added or removed | Real route: /trends |
| Briefings | Collapse one watchlist into a unified story surface that starts with the current summary, highlights recent deltas, and keeps evidence one click away | Real route: /briefings; now backed by a server-owned briefing page payload that shares one canonical selected-story object with Ask |
| Evidence bundle | Export one job as a reusable internal bundle with digest, trace summary, knowledge cards, and artifact manifest | Real route on demand: /api/v1/jobs/<job-id>/bundle |
| Playground | Explore clearly labeled sample corpus and demo outputs without pretending they are live operator state | Real route: /playground + docs/samples/README.md |
| Use-case pages | Route newcomer traffic into truthful capability stories for YouTube, Bilibili, RSS, MCP, and research workflows | Real routes: /use-cases/youtube, /use-cases/bilibili, /use-cases/rss, /use-cases/mcp-use-cases, /use-cases/research-pipeline |
These are real directions, but they are not current product claims:
Choose the shortest honest path for the result you want first:
| I want to... | Do this first | What I get |
|---|---|---|
| discover the repo-local command surface first | ./bin/sourceharbor help | a thin menu over the existing bin/* entrypoints without inventing a second CLI stack |
| evaluate without booting anything | docs/see-it-fast.md | the fastest public tour of the command center, digest feed, and job trace |
| run a real local flow | docs/start-here.md | the shortest repo-documented path to a local stack and a queued or completed job |
| inspect the trust boundary first | docs/proof.md | the current proof map, including what is locally provable and where the public boundary stops |
There are three honest first paths:
Truth route, in plain English:
README.mdis the front door,docs/start-here.mdis the first real run,docs/proof.mdis the proof ladder,docs/generated/*pages are render-only pointers, and.agents/Plans/*files are historical execution archives rather than current public truth.
Current non-promises:
If you want the shortest honest summary of what is already real, what is still gated, and what remains future direction, read docs/project-status.md.
If you only have half a minute, do not start with setup. Start with the three surfaces that explain the product fastest:
AI Weekly and Digest One become reusable summaries instead of lost links.Source -> queued job -> digest feed -> searchable artifact -> MCP / API reuse
Representative result shape, based on the current digest template and UI surfaces:
# AI Weekly
> Source: [Original video](https://www.youtube.com/watch?v=abc)
> Platform: youtube | Video ID: video-uid-123 | Generated at: 2026-02-10T00:00:00Z
## One-Minute Summary
- This episode focuses on agent workflows, operator visibility, and job trace.
## Key Takeaways
- Every job carries a step summary, artifacts index, and pipeline final status.
For the lightweight evaluation path, go to docs/see-it-fast.md.
| Surface | What you can do | Why it matters |
|---|---|---|
| Subscriptions | Start from strong YouTube/Bilibili templates or widen into RSSHub and generic RSS intake through the shared backend template catalog | Build a durable intake layer without pretending every source family is equally proven |
| Digest feed | Read generated summaries in a single operator flow | Turn long-form content into an actionable daily reading stream |
| Search & Ask | Search raw evidence and turn a watchlist or selected story briefing into an answer + change + citation flow on one page, with Briefings and Ask now sharing a server-owned story read-model instead of parallel browser-side selection glue | Make the knowledge layer visible without pretending every question already has a global answer engine |
| Job trace | Inspect pipeline status, retries, degradations, and artifacts | Debug with evidence instead of guessing what happened |
| Notifications | Configure and send digests outward when the notification lane is enabled | Push results outward instead of trapping them in a database |
| Retrieval | Search over generated artifacts | Reuse digests as a searchable knowledge layer |
| MCP tools | Expose subscriptions, ingestion, jobs, artifacts, search, and notifications to agents | Let assistants act on the same system without custom glue code |
Think of this like walking past a storefront window before deciding whether to step inside.
This path is for evaluation, not a hosted trial. You are inspecting the product shape, evidence surfaces, and result format before deciding whether a local run is worth it.
This is the shortest truthful local setup path. It starts when you are ready to install dependencies and boot the stack yourself; it is not a hosted "try now" flow.
By the end of this path, you should have:
./bin/sourceharbor help
cp .env.example .env
set -a
source .env
set +a
UV_PROJECT_ENVIRONMENT="${UV_PROJECT_ENVIRONMENT:-$SOURCE_HARBOR_CACHE_ROOT/project-venv}" \
uv sync --frozen --extra dev --extra e2e
bash scripts/ci/prepare_web_runtime.sh >/dev/null
./bin/bootstrap-full-stack
./bin/full-stack up
source .runtime-cache/run/full-stack/resolved.env
Read the resolved local routes:
${SOURCE_HARBOR_API_BASE_URL}http://127.0.0.1:${WEB_PORT}The clean local path is container-first for Postgres. By default .env.example
uses CORE_POSTGRES_PORT=15432 together with
postgresql+psycopg://postgres:postgres@127.0.0.1:${CORE_POSTGRES_PORT}/sourceharbor
so a host Postgres on 127.0.0.1:5432 does not silently become the active data
plane.
Open the operator UI at the resolved web URL:
http://127.0.0.1:${WEB_PORT}If you only need the repo-managed local proof, stop at the supervisor checks first:
./bin/full-stack status
./bin/doctor
curl -sS "${SOURCE_HARBOR_API_BASE_URL}/healthz"
curl -I "http://127.0.0.1:${WEB_PORT}/ops"
./bin/smoke-full-stack --offline-fallback 0 is the stricter long live-smoke
lane. It goes beyond local supervisor proof and can still stop on provider-side
YouTube preflight or Resend sender configuration even after the local stack is
healthy.
export SOURCE_HARBOR_API_KEY="${SOURCE_HARBOR_API_KEY:-sourceharbor-local-dev-token}"
If you launch the API outside ./bin/full-stack up, export both
SOURCE_HARBOR_API_KEY and WEB_ACTION_SESSION_TOKEN before starting the
API process so write routes and web actions share the same local token contract.
Replace the sample URL with any public YouTube or Bilibili video:
curl -sS -X POST "${SOURCE_HARBOR_API_BASE_URL}/api/v1/videos/process" \
-H "Content-Type: application/json" \
-H "X-API-Key: ${SOURCE_HARBOR_API_KEY}" \
-d '{
"video": {
"platform": "youtube",
"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
},
"mode": "full"
}'
curl -sS "${SOURCE_HARBOR_API_BASE_URL}/api/v1/videos" | jq
curl -sS "${SOURCE_HARBOR_API_BASE_URL}/api/v1/feed/digests" | jq
curl -sS "${SOURCE_HARBOR_API_BASE_URL}/api/v1/jobs/<job-id>" | jq
Open these front-door routes after the stack is up:
/search for grounded search/ask for the story-aware, briefing-backed Ask front door, now sharing the same server-owned selected-story read-model that /briefings uses/mcp for the in-product MCP front door/ops for operator diagnostics and hardening gatesThe truthful package story today is intentionally thin:
sourceharbor from packages/sourceharbor-cli stays repo-aware and delegates to the repo-local command substrate instead of pretending to manage the whole local runtime itself./bin/sourceharbor mcp and ./bin/sourceharbor doctor remain the repo-local operator entrypointspackages/sourceharbor-sdk is the public TypeScript SDK surface and still stays contract-first instead of becoming a second business-logic stack./bin/smoke-full-stack --offline-fallback 0
When you want the operator-side log trail, start at .runtime-cache/logs/components/full-stack.
For a guided version with operator notes and public-proof boundaries, go to docs/start-here.md.
Most repos in this space stop at one of these layers:
SourceHarbor is built around the full knowledge flow:
See the full comparison in docs/compare.md.
This repository does not ask you to trust product copy on its own.
GitHub profile intent is tracked in config/public/github-profile.json. Use
python3 scripts/github/apply_public_profile.py --verify to compare the live
description, homepage, and topics against the current tracked intent, and use
python3 scripts/github/apply_public_profile.py when you intentionally want to
sync those settings after current main truth is ready. Social preview upload
still requires a manual GitHub Settings check.
Generated docs under docs/generated/ can point you toward runtime-owned evidence, but they are not the current verdict themselves. Historical plans under .agents/Plans/ explain past execution context only and should not be read as the current public truth route.
SourceHarbor is a public, source-first engineering repository.
It is inspectable, and you can run it locally. It is not marketed as a turnkey hosted product, and external distribution claims are valid only when live remote workflows prove them for the current
maincommit.
For local verification, the repo-managed route snapshot under
.runtime-cache/run/full-stack/resolved.env is the runtime truth for API/Web
ports. Do not assume any process already listening on 9000, 3000, or
5432 belongs to the clean-path stack.
Start where you are:
No. SourceHarbor is a source-first repository you can inspect, run locally, adapt, and extend.
No. The public surface is strongest around long-form video today, but the feed and retrieval layers already model both video and article content types.
Because it sits at the intersection of source ingestion, digest pipelines, retrieval, operator UI, and MCP reuse. Even if you are not adopting it immediately, it is a strong reference point for how to turn long-form inputs into reusable knowledge products.
More questions are answered in docs/faq.md.
apps/api: FastAPI service for ingestion, jobs, artifacts, retrieval, notifications, and operator controlsapps/worker: pipeline runner, Temporal workflows, and delivery automationapps/mcp: MCP tool surface for agentsapps/web: browser command center for operatorscontracts: shared schemas and generated contract artifactsdocs: layered public navigation, proof, and architectureSourceHarbor is released under the MIT License. See LICENSE.
FAQs
Turn YouTube, Bilibili, and RSS sources into readable digests, traceable job runs, and searchable knowledge for operators and agents.
We found that sourceharbor demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Company News
Socket is now in the AWS Security Hub Extended plan. Adopt it through AWS, apply committed spend, and block malicious open source packages.

Research
/Security News
Popular npm packages keyv and cacheable compromised.

Security News
A misconfiguration gave three Anthropic models internet access, and one, believing it was in a simulation, shipped a credential-stealing package to PyPI.