dsh-session-eval

dsh-session-eval

English | 中文

Retrospective evaluation for DeepSeek Harness agent sessions — deterministic grade cards computed from sessions that already happened, plus cross-session regression diffs that answer "did my agent get better or worse after that plugin / prompt / model change?"

No benchmark suites to author. No LLM judges. No new runs. Every grade is a fixed, documented threshold over persisted session logs — fully reproducible.

The measurement layer of the session toolchain

Plugin	Layer	Answers
dsh-session-export	Evidence	"What exactly happened in this session?"
dsh-session-recall	Memory	"What did I do before, and where is it?"
dsh-session-eval	Measurement	"Was that session good? Is the trend improving?"

All three read through the same trusted ctx.sessionQuery seam, so any persistence backend (JSONL or SQLite) works without touching raw artifacts.

How it differs from benchmark-style evaluation

	dsh-eval (benchmark)	dsh-session-eval (this)
Evaluates	Runs you launch ahead of time	Sessions that already happened
Setup	Author benchmark YAML + cases	None — read existing logs
Judge	Metric folding over trials	Fixed thresholds, deterministic
Best at	Pre-deploy benchmarks	Regression tracking over real work

Both are useful; they don't compete. Benchmarks test what you predicted; retrospective evaluation watches what actually happened.

Quick Start

npm install -g dsh-session-eval

Then in DeepSeek Harness:

/eval

Session eval — a1b2c3d4… (dsh-session-eval v0.1.0)
Overall: A
  Reliability   A    2/87 tool results errored (2.3%)
  Re-ask        A    1 re-ask signal(s) over 23 turns (0.4/10 turns)
  Tool load     B    3.8 tool calls per turn
Notes:
  • Wall time 41.2 min over 23 turns.

Grade another session, or emit machine-readable JSON:

/eval --id <sessionId> --json

Compare two sessions — the regression workflow:

/eval-diff <beforeSessionId> <afterSessionId>

Session diff — a1b2c3d4… → e5f6a7b8… (dsh-session-eval v0.1.0)
Overall: regressed — 0 improved, 2 regressed, 1 flat.
  Reliability   2.1% → 11.8%   regressed
  Re-ask        0.5 → 3.2      regressed
  Tool load     3.8 → 4.1      flat

Typical use: grade the ten sessions before and after a plugin or prompt change; a persistent Reliability regression is an early warning that the change hurt real work, not just benchmarks.

Dimensions & thresholds

All grades derive from three deterministic metrics. n/a when the session lacks the relevant activity.

Dimension	Metric	A	B	C	D	E
Reliability	tool error rate	≤5%	≤10%	≤20%	≤35%	>35%
Re-ask	corrections per 10 turns¹	0	<1	<2	<4	≥4
Tool load	tool calls per turn	0.5–8	0.1–0.5 or just above 8	<0.1 or 8–20	>20	—

¹ A correction signal is each extra user message inside a consecutive-user run — the human re-asking before any assistant reply, a behavioral proxy for "the agent didn't get it right the first time."

Overall is the worst graded dimension — conservative by design. Thresholds are constants in src/grade.ts; they will be configurable in a future release.

Diff verdicts include a noise band (±2pp error rate, ±0.5 re-ask, ±1 tool load) so trivial jitter doesn't cry regression.

Determinism guarantee

Same session logs in → same grade card out, every time, forever. There is no model in the loop, no sampling, no clock dependence in the metrics (wall time is reported, never graded). This is what makes /eval-diff trustworthy as a regression signal.

Commands

Command	Effect
/eval	Grade card for the current session
/eval --id <sessionId>	Grade a specific session
/eval --json	Machine-readable metrics + card
/eval-diff <beforeId> <afterId>	Regression comparison, noise-tolerant
/eval-diff … --json	Machine-readable trend report

Development

npm install
npm run typecheck   # tsc --noEmit
npm test            # vitest run — 24 tests
npm run bundle      # tsdown → lib/

License

MIT