dsh-stability-audit

dsh-stability-audit (plugin stability audit)

Scan installed DeepSeek Harness (dsh) plugins and get a stability risk grade + fix suggestion before they break your harness — static analysis, optional isolated install verification, and machine-readable output for any agent to surface problems and remedies fast.

Built from real incidents: every grading rule comes from actual dsh ecosystem failures — barricade's hooks killing the whole tool chain, codegraph freezing the event loop at startup, trailmap's version conflict breaking the tool layer, dsh-tools single-instance corruption, doc-guard/secret-guard missing-inject boot crashes, 7 unbuilt plugins crashing on load, memento/taskboard undeclared deps (Cannot find package)... The rules are the troubleshooting manual, codified.

Community plugins are often young projects. This plugin answers one question: will installing/keeping this plugin hurt my dsh? Each installed plugin gets a 🔴/🟡/🟢 grade, evidence, and a 🔧 fix suggestion, so you decide on facts, not vibes.

Why

• One plugin can kill the entire tool layer (hook surface — barricade incident; dsh-tools single-instance — trailmap incident)
• Startup scans can freeze the event loop for minutes on big workspaces (codegraph incident)
• Bad plugin config makes dsh silently half-activate plugins (preflight failure)
• @deepseek-ai/* version conflicts can kill the whole tool pipeline
• Every failure period pollutes active sessions (dangling tool-calls) that need cleanup after recovery

These are the real failure modes this plugin statically detects — before you step on them.

Install

# Option 1: npm (recommended — prebuilt, skips allowBuilds approval)
dsh plugin --profile web add dsh-stability-audit

# Option 2: GitHub source
dsh plugin --profile web add github:chunfenxiazhi-collab/dsh-stability-audit

After restarting dsh web, agents can call the stability_audit tool (just say "run the plugin stability audit"); or use the CLI:

git clone https://github.com/chunfenxiazhi-collab/dsh-stability-audit.git
cd dsh-stability-audit && npm test
node cli.mjs                    # scan the real web profile
node cli.mjs --dynamic          # also run isolated install verification (~2s/plugin)
node cli.mjs --json             # machine-readable output (for other agents)
node cli.mjs --remote owner/repo          # remote plugin static pre-audit (clone to temp dir, no install)
node cli.mjs --remote owner/repo --dynamic  # full remote pre-audit (clone + isolated install smoke)
node cli.mjs --remote owner/repo --json    # remote pre-audit JSON output

Grading dimensions

Signal	Grade	Case
Tool hook / middleware registration (pre/post-execute)	red	can intercept the whole tool chain (barricade)
Top-level sync scan / indexing at startup	red	blocks the event loop (codegraph)
Load-time sync service access missing from inject	red	boot crash (doc-guard/barricade incidents)
main points to unbuilt source (TS/missing file)	red	load-time crash (7 unbuilt plugins)
Preflight report critical	red	flagged by dsh at boot
setInterval polling	yellow	normal for update checks; watch frequency
>=10 global event listeners	yellow	every broadcast event gets slower
No dsh.bundle manifest	yellow	installed but never activated (research-plugins)
>50 dependencies	yellow	dependency-tree pollution risk
Non-build install scripts (prepare etc.)	yellow	npm lifecycle runs arbitrary code
Deferred service access missing from inject	yellow	may be undefined at runtime (cron-style)
@deepseek-ai/* range excludes host synced pkg version	yellow	pnpm hoists old build -> tools dead (trailmap)
Isolated install + boot smoke	pass/fail	temp DSH_HOME, dsh plugin add, headless boot, no loader errors
Entry imports undeclared deps (missing-dep)	red	Cannot find package after install (taskboard/memento)
No main and no candidate entry (no-entry)	red	loader fails importing by package name (verify)
Data exfiltration (process.env via fetch/WebSocket)	red	OpenClaw 341 malicious-skill top pattern
Hardcoded remote endpoints (URL/WebSocket)	yellow	for human review (normal API calls common)

Every hit carries a 🔧 fix suggestion (executable command or manual action) that can be handed to the user or another agent.

Batch testing: --remote accepts multiple specs (comma- or space-separated) to pre-audit a batch of online plugins in one run (clone → static → isolated install smoke, host untouched). Verified sample: docs/audit-report.md (18 plugins, post-verification verdicts).

How it works

• Collect plugins from ~/.dsh/profiles//package.json (node_modules + plugin-src)
• Static scan: entry file + package.json signals (regex-based, zero dependencies, no AST)
• Optional dynamic check (lib/dyncheck.js): temp DSH_HOME -> isolated dsh plugin add -> headless boot -> loader-error scan -> cleanup
• Optional remote pre-audit (lib/remote.js): git shallow clone to temp dir -> static rules -> (optional) isolated install smoke -> cleanup
• Render report: Markdown (human) or JSON (--json, schema dsh-stability-audit/v1, agent-readable)

Principles: read-only, never runs the audited plugin, zero side effects. Grades are suggestions for human confirmation, not verdicts.

Architecture diagram

The runtime architecture is drawn with dsh-archify (MIT — a dsh port of tt-a1i/archify, also MIT, itself derived from Cocoon AI's "architecture-diagram-generator"). The MIT notice is embedded in the rendered HTML head; spec and evaluation notes are kept in-repo:

• Spec (EN): runtime.architecture.en.json
• Rendered EN (interactive, guided views): docs/dsh-stability-audit-architecture.en.html
• 中文版（Chinese version）: README.zh.md — 架构图见中文版
• Evaluation notes: docs/ARCHIFY-EVALUATION.md

JSON output (for agents)

node cli.mjs --json

Per plugin: name / version / grade / source / dynamic / findings[], each finding carries ruleId / severity / desc / evidence / fix (remedy). Other agents can act on it directly.

Known limitations (honest list)

• Static analysis cannot measure runtime behavior (hook throws, real stall duration) — red means "needs a look", not "definitely broken"
• Isolated boot uses a headless profile: plugins depending on web-only services (e.g. storageDomain) show ❌ in isolation but may be fine in web — treat as an environment-dependency hint
• Runtime probes (event-loop latency, hook timing) are on the v2 roadmap
• Fix suggestions are hints, not auto-applied: environments differ (junction/overrides fixes depend on the specific mechanism), agents should confirm before executing

Isolated test vs real environment (what it can and cannot catch)

Isolated testing (temp DSH_HOME + headless smoke) answers: "can this plugin install on its own and boot cleanly?" — filtering out ~80% of the failure modes (unbuilt entry, missing inject, dependency conflicts, load crashes).

Dimension	Isolated env	Real web profile	Impact
Services	headless boot	web services + all host services	plugins depending on webServer/webRuntime stay pending in isolation; real behavior not observable
Co-installed plugins	only the audited one	many plugins interacting	inter-plugin conflicts not caught (hooks stepping on each other, service overrides)
Config	empty	real config (API keys, models, paths)	config-dependent plugin paths untested
Credentials/network	no API key	model APIs available	LLM-dependent features only smoke, never execute
Data	empty storage	real sessions/libraries	data-migration plugins untested
Runtime duration	seconds of boot smoke	days of residency	setInterval leaks, memory growth not caught
Permissions/build	same machine, same permissions	same machine, same permissions	✅ identical

Suggested usage: use isolated testing as bulk screening — exclude red/fail targets; for plugins you actually want, install manually and observe. Automatically testing plugins against the real environment is not recommended — plugins have real destructive power (see the dsh-troubleshooting incidents).

Releases

• Release flow: see RELEASE.md (GitHub + npm + Release, three channels in sync)
• npm: https://www.npmjs.com/package/dsh-stability-audit
• GitHub Releases (with tarball assets): https://github.com/chunfenxiazhi-collab/dsh-stability-audit/releases

License

MIT (c) 2026 chunfenxiazhi

Contributing

Issues and PRs welcome. Tests: npm test (node:test, zero dependencies).

Commit convention: English subject line + optional Chinese body (e.g. feat: remote plugin pre-audit).