harness-evolver - npm Package Compare versions

Comparing version 6.3.2 to 6.4.0

skills/certify/SKILL.md

+---
+name: harness:certify
+description: "Use when the user wants to verify that the evolved agent's score is stable and reliable. Runs evaluation multiple times and reports mean ± std."
+allowed-tools: [Read, Bash, Glob]
+---
+
+# /harness:certify
+
+Verify score stability by running evaluation multiple times and reporting statistical confidence.
+
+## Resolve Tool Path
+
+```bash
+TOOLS="${EVOLVER_TOOLS:-$([ -d ".evolver/tools" ] && echo ".evolver/tools" || echo "$HOME/.evolver/tools")}"
+EVOLVER_PY="${EVOLVER_PY:-$([ -f "$HOME/.evolver/venv/bin/python" ] && echo "$HOME/.evolver/venv/bin/python" || echo "python3")}"
+```
+
+## What To Do
+
+Read `.evolver.json` to get the best experiment and dataset.
+
+Run evaluation 3 times on the current code (not a worktree — the best code is already merged):
+
+```bash
+for i in 1 2 3; do
+    $EVOLVER_PY $TOOLS/run_eval.py \
+        --config .evolver.json \
+        --worktree-path "." \
+        --experiment-prefix "certify-run-$i" \
+        --no-canary
+done
+```
+
+After all 3 runs complete, read results and compute statistics:
+
+```bash
+$EVOLVER_PY $TOOLS/read_results.py --experiments "certify-run-1-{suffix},certify-run-2-{suffix},certify-run-3-{suffix}" --config .evolver.json --format summary
+```
+
+Calculate mean and standard deviation from the 3 combined_scores.
+
+## Report
+
+```
+CERTIFICATION REPORT
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+Runs:  3
+Mean:  {mean:.3f}
+Std:   {std:.3f}
+Range: {min:.3f} — {max:.3f}
+
+Verdict: {STABLE|UNSTABLE}
+```
+
+**STABLE** (std < 0.05): Score is reliable. The agent performs consistently.
+
+**MARGINAL** (0.05 <= std < 0.10): Score varies moderately. Consider adding rubrics to reduce judge variance.
+
+**UNSTABLE** (std >= 0.10): Score is unreliable. The LLM judge interprets criteria differently across runs. Add few-shot examples or tighter rubrics.
+
+## After Certification
+
+If STABLE: suggest `/harness:deploy` to finalize.
+If UNSTABLE: suggest adding rubrics to dataset examples, or running `/harness:evolve` with `heavy` mode for more thorough evaluation.

tools/promote_learnings.py

+#!/usr/bin/env python3
+"""Promote proven evolution learnings to CLAUDE.md.
+
+Reads evolution_memory.md, extracts insights with recurrence >= threshold,
+and appends them to the project's CLAUDE.md as permanent rules.
+
+This implements "compound learning" — each evolution session permanently
+improves the project, not just the code.
+
+Usage:
+    python3 promote_learnings.py --memory evolution_memory.md --target CLAUDE.md --threshold 5
+    python3 promote_learnings.py --memory evolution_memory.md --dry-run
+
+Stdlib-only — no langsmith dependency.
+
+References:
+    - Compound Engineering (EveryInc): explicit codification of learnings
+    - Self-Improving Agent (pskoett): 3-tier promotion with quantitative thresholds
+"""
+
+import argparse
+import json
+import os
+import re
+import sys
+
+
+def parse_evolution_memory(memory_path):
+    """Parse evolution_memory.md and extract insights with recurrence counts."""
+    if not os.path.exists(memory_path):
+        return []
+
+    insights = []
+    with open(memory_path) as f:
+        content = f.read()
+
+    # Parse "Key Insights" section — format: "N. **text** [rec:N]"
+    # Also handles: "N. text [rec:N]" and "- text [rec:N]"
+    pattern = r'(?:^[\d]+\.\s+|\-\s+)\*{0,2}(.+?)\*{0,2}\s+\[rec:(\d+)\]'
+    # LLM consolidator output may also use "(seen Nx)" format
+    pattern2 = r'(?:^[\d]+\.\s+|\-\s+)\*{0,2}(.+?)\*{0,2}\s+\(seen\s+(\d+)x\)'
+
+    seen_texts = set()
+    for pat in (pattern, pattern2):
+        for match in re.finditer(pat, content, re.MULTILINE):
+            text = match.group(1).strip()
+            rec = int(match.group(2))
+            if text not in seen_texts:
+                seen_texts.add(text)
+                insights.append({"text": text, "recurrence": rec})
+
+    return insights
+
+
+def format_as_claude_rules(insights, project_name=""):
+    """Format insights as CLAUDE.md rules."""
+    if not insights:
+        return ""
+
+    lines = [
+        "",
+        f"## Evolution Learnings{' — ' + project_name if project_name else ''}",
+        "",
+        "Rules learned from automated evolution (promoted from evolution_memory.md):",
+        "",
+    ]
+    for insight in insights:
+        lines.append(f"- {insight['text']}")
+
+    lines.append("")
+    return "\n".join(lines)
+
+
+def append_to_claude_md(target_path, rules_text, dry_run=False):
+    """Append rules to CLAUDE.md. Creates file if it doesn't exist."""
+    if dry_run:
+        print("DRY RUN — would append to", target_path, file=sys.stderr)
+        print(rules_text, file=sys.stderr)
+        return True
+
+    # Check if rules already exist (prevent duplicates)
+    if os.path.exists(target_path):
+        with open(target_path) as f:
+            existing = f.read()
+        if "## Evolution Learnings" in existing:
+            print("Evolution Learnings section already exists in CLAUDE.md. Skipping to prevent duplicates.", file=sys.stderr)
+            return False
+
+    with open(target_path, "a") as f:
+        f.write(rules_text)
+
+    return True
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Promote evolution learnings to CLAUDE.md")
+    parser.add_argument("--memory", default="evolution_memory.md", help="Path to evolution_memory.md")
+    parser.add_argument("--target", default="CLAUDE.md", help="Path to CLAUDE.md to append to")
+    parser.add_argument("--threshold", type=int, default=5, help="Minimum recurrence to promote (default 5)")
+    parser.add_argument("--project", default="", help="Project name for section header")
+    parser.add_argument("--dry-run", action="store_true", help="Show what would be promoted without writing")
+    parser.add_argument("--output", default=None, help="Write promoted insights to JSON file")
+    args = parser.parse_args()
+
+    insights = parse_evolution_memory(args.memory)
+    if not insights:
+        print(json.dumps({"promoted": 0, "total_insights": 0}))
+        return
+
+    promotable = [i for i in insights if i["recurrence"] >= args.threshold]
+
+    if not promotable:
+        print(json.dumps({
+            "promoted": 0,
+            "total_insights": len(insights),
+            "max_recurrence": max(i["recurrence"] for i in insights),
+            "threshold": args.threshold,
+        }))
+        return
+
+    rules_text = format_as_claude_rules(promotable, args.project)
+    success = append_to_claude_md(args.target, rules_text, args.dry_run)
+
+    result = {
+        "promoted": len(promotable) if success else 0,
+        "total_insights": len(insights),
+        "threshold": args.threshold,
+        "insights": [i["text"] for i in promotable],
+    }
+
+    if args.output:
+        with open(args.output, "w") as f:
+            json.dump(result, f, indent=2)
+
+    print(json.dumps(result, indent=2))
+
+
+if __name__ == "__main__":
+    main()

.claude-plugin/plugin.json

 {
   "name": "harness-evolver",
-  "description": "LangSmith-native autonomous agent optimization — evolves LLM agent code using multi-agent proposers, LangSmith experiments, and git worktrees",
-  "version": "6.3.2",
+  "description": "LangSmith-native autonomous agent optimization \u2014 evolves LLM agent code using multi-agent proposers, LangSmith experiments, and git worktrees",
+  "version": "6.4.0",
   "author": {
@@ -6,0 +6,0 @@ "name": "Raphael Valdetaro"

agents/harness-consolidator.md

@@ -19,3 +19,3 @@ ---
 
-## Four-Phase Process
+## Five-Phase Process
 
@@ -39,4 +39,4 @@ ### Phase 1: Orient
 
-- **Promoted insights (rec >= 3)**: Copy verbatim from prior memory. Do NOT rephrase or re-summarize. These are stable knowledge.
-- **Rising insights (rec 1-2)**: Update recurrence count. If confirmed again, promote.
+- **Anchored insights (rec >= 3)**: Copy verbatim from prior memory. Do NOT rephrase or re-summarize. These are stable knowledge.
+- **Rising insights (rec 1-2)**: Update recurrence count. If confirmed again, anchor.
 - **New observations**: Extract from comparison.json and proposal.md. Use LITERAL text from proposal.md's `## Approach` and `## Expected Impact` sections — do not paraphrase. Paraphrasing loses fidelity (telephone game effect).
@@ -48,5 +48,17 @@ - **Contradictions**: Newer information wins. Mark old insight as superseded, don't delete.
 - **Garbage collection**: Remove observations that haven't recurred in 5+ iterations
-- Promoted insights are never pruned (they're proven patterns)
+- Anchored insights are never pruned (they're proven patterns)
 - Keep the markdown under 2KB
 
+### Phase 5: Promote (optional)
+
+If any insight has recurrence >= 5 (proven across 5+ iterations), flag it for promotion:
+
+```
+PROMOTION CANDIDATES:
+- "Never use vector search on KB < 50 lines" [rec:7]
+- "Input parsing: always extract from JSON, never pass raw path" [rec:5]
+```
+
+These candidates will be offered to the user during `/harness:deploy` for permanent addition to CLAUDE.md. Do NOT write to CLAUDE.md directly — promotion requires user consent.
+
 ## Constraints
@@ -53,0 +65,0 @@

package.json

 {
   "name": "harness-evolver",
-  "version": "6.3.2",
+  "version": "6.4.0",
   "description": "LangSmith-native autonomous agent optimization for Claude Code",
@@ -5,0 +5,0 @@ "author": "Raphael Valdetaro",

skills/deploy/SKILL.md

@@ -13,2 +13,7 @@ ---
 
+```bash
+TOOLS="${EVOLVER_TOOLS:-$([ -d ".evolver/tools" ] && echo ".evolver/tools" || echo "$HOME/.evolver/tools")}"
+EVOLVER_PY="${EVOLVER_PY:-$([ -f "$HOME/.evolver/venv/bin/python" ] && echo "$HOME/.evolver/venv/bin/python" || echo "python3")}"
+```
+
 ### 1. Show Results
@@ -46,3 +51,4 @@
       {"label": "Just review", "description": "Show the full diff of all changes made during evolution"},
-      {"label": "Clean up only", "description": "Remove temporary files (trace_insights.json, etc.) but don't push"}
+      {"label": "Clean up only", "description": "Remove temporary files (trace_insights.json, etc.) but don't push"},
+      {"label": "Promote learnings", "description": "Add proven evolution insights to CLAUDE.md (permanent knowledge)"}
     ]
@@ -73,2 +79,9 @@ }]
 
+**If "Promote learnings"**:
+```bash
+$EVOLVER_PY $TOOLS/promote_learnings.py --memory evolution_memory.md --target CLAUDE.md --threshold 5 --dry-run
+```
+
+Show the dry-run output. If the user approves, run without `--dry-run`.
+
 ### 4. Report
@@ -75,0 +88,0 @@

New alerts

AI-detected potential code anomaly

Supply chain risk

AI has identified unusual behaviors that may pose a security risk.

Fixed alerts

AI-detected potential code anomaly

Supply chain risk

AI has identified unusual behaviors that may pose a security risk.

Improved metrics

Total package byte prevSize

356795

2.26%

Number of package files

45

4.65%

Lines of code

6507

1.67%

No dependency changes