create-adsk
| # /run-skill-evals | ||
| Run with-skill vs without-skill output evals for a skill (Tier 2 style). Soft quality signal — not a merge gate. | ||
| ## Skill / playbook | ||
| Follow **`skills/skill-optimizer`** → `references/eval-loop.md` (or `.agents/skills/skill-optimizer` in adopter apps). Full runbook: `docs/evaluating-skills.md`. | ||
| ## Usage | ||
| ``` | ||
| /run-skill-evals | ||
| /run-skill-evals skills/spec-driven-workflow | ||
| /run-skill-evals .agents/skills/my-company-skill | ||
| ``` | ||
| ## When to use | ||
| - You **changed** a skill’s behavior or description, or switched models | ||
| - You are shipping a **company** skill and need with/without evidence before commit | ||
| **Do not** re-benchmark every PR. For stock first-party skills, trust published numbers in `docs/evals/SCORECARD.md` unless the above applies. | ||
| ## Behavior | ||
| 1. Locate the target skill (ask if unclear). Prefer kit `skills/<name>` or adopter `.agents/skills/<name>`. | ||
| 2. Confirm `evals/evals.json` exists (and ideally `evals/trigger/eval_queries.json`). If missing, stop and point to `/optimize-skill` first. | ||
| 3. **Package cases** (pick one): | ||
| - **This kit repo:** run `./scripts/run-skill-evals-soft.sh --skill <name>` → work under `.adsk-tier2-out/<name>/`. | ||
| - **Adopter app (no kit scripts):** read prompts from `.agents/skills/<name>/evals/evals.json` directly; create a gitignored workspace (e.g. `.adsk-tier2-out/<name>/`) with `with_skill/` and `without_skill/` arms per case. | ||
| 4. For **each** eval case, in a **clean** context: | ||
| - **with_skill** — skill available; paste the prompt; save output | ||
| - **without_skill** — same prompt without the skill (or prior version); save output | ||
| 5. Grade each assertion PASS/FAIL with evidence into `grading.json` (or equivalent). Prefer mechanical checks when possible. | ||
| 6. Summarize pass-rate Δ (and token/time Δ if available). For kit first-party skills, offer a SCORECARD paste row; for company skills, report in-chat / PR notes (do not invent kit SCORECARD edits unless asked). | ||
| 7. Reminder: Tier 1 (`./scripts/check-skills-ci.sh` in the kit) only checks harness shape — it does not replace this loop. |
@@ -17,6 +17,7 @@ # /implement-spec | ||
| 1. Read the spec or plan; follow the plan step-by-step when present. | ||
| 2. Implement with tests mapped to requirements (or justify non-behavioral exceptions). | ||
| 3. Update plan todo status as steps complete (Cursor Plan YAML `todos` and/or portable checklist — see `references/cursor-adapter.md` / `artifact-homes.md`). | ||
| 4. Run project verify commands; fix failures before claiming done. | ||
| 5. Suggest `/review` when implementation is complete. | ||
| 1. **Clear** (Medium+): confirm exploration is persisted in the living spec/plan; start this session lean — see skill gated procedure. | ||
| 2. Read the spec or plan; follow the plan step-by-step when present. | ||
| 3. Implement with tests mapped to requirements (or justify non-behavioral exceptions). | ||
| 4. Update plan todo status as steps complete (Cursor Plan YAML `todos` and/or portable checklist — see `references/cursor-adapter.md` / `artifact-homes.md`). | ||
| 5. **Fail-closed verify:** run `project-cmds` / project verify; fix failures before claiming done. If verify is missing, refuse “done” and point to `/quick-start` — do not invent a silent pass. | ||
| 6. Suggest `/review` when implementation is complete. |
@@ -22,2 +22,2 @@ # /optimize-skill | ||
| 3. Apply lean edits; move kit-meta/rationale into `references/` with hard when-to-load conditions. | ||
| 4. Report what changed and any remaining eval gaps. | ||
| 4. Report what changed and any remaining eval gaps. For a full with/without output pass, hand off to `/run-skill-evals`. |
@@ -18,7 +18,8 @@ # /plan-impl | ||
| 1. Read the spec; if open questions or unconfirmed assumptions remain, resolve those before planning. | ||
| 2. Break work into ordered, verifiable tasks (files, tests, risks). Prefer tasks that map to `REQ-XXX` and stay within a focused file set. | ||
| 3. **Resolve the plan path** via the skill. As a Cursor `/` command, default to `.cursor/plans/{feature-name}.plan.md` unless the project already uses `docs/plans/` or the user overrides the path. | ||
| 4. Write the plan with **trackable todos** (Cursor: non-empty frontmatter `todos`; never body-only REQ tables with empty Plan UI todos). | ||
| 5. Suggest `/implement-spec` only after the user approves the plan (medium+). | ||
| 2. Break work into ordered, verifiable tasks (files, tests, risks). Prefer tasks that map to `REQ-XXX` and stay within a focused file set. Prefer splitting **build** from **verify/review** so QA can run in parallel with the next slice. | ||
| 3. **Tracer gate** (Large, and Medium when architecture/integration is ambiguous): include a thin vertical slice + one verify before multi-phase implement, or “N/A — architecture proven” with justification — see skill / `problem-size-guide.md`. | ||
| 4. **Resolve the plan path** via the skill. As a Cursor `/` command, default to `.cursor/plans/{feature-name}.plan.md` unless the project already uses `docs/plans/` or the user overrides the path. | ||
| 5. Write the plan with **trackable todos** (Cursor: non-empty frontmatter `todos`; never body-only REQ tables with empty Plan UI todos). | ||
| 6. Suggest `/implement-spec` only after the user approves the plan (medium+). | ||
| Keep this command thin — planning depth lives in the skill / recommended upstream. |
@@ -19,3 +19,7 @@ # /quick-start | ||
| 2. Detect artifact homes (Cursor `.cursor/...` vs portable `docs/...`) via `references/artifact-homes.md`; when Cursor wiring is present, update or propose `.cursor/rules/project/` and `project-cmds` with real verify commands. | ||
| 3. Guide next steps: greenfield → `/draft-spec`; brownfield → `/extract-spec`. | ||
| 4. Confirm ADSK skills are discoverable under `.agents/skills/`. If missing, point to `docs/using-adsk.md` (`npx skills add rhyanvargas/agentic-development-starter-kit`). | ||
| 3. Confirm ADSK skills are discoverable under `.agents/skills/`. If missing, point to `docs/using-adsk.md` (`npx skills add rhyanvargas/agentic-development-starter-kit`). | ||
| 4. End with a short **where truth lives** checklist (all four required): | ||
| - Spec/plan artifact home (resolved path) | ||
| - Verify commands location (`project-cmds` or portable equivalent) | ||
| - Skills discovery path (`.agents/skills/`) | ||
| - Next command: greenfield → `/draft-spec`; brownfield → `/extract-spec` |
@@ -26,2 +26,3 @@ --- | ||
| # Skill Tier 2 fixtures:./scripts/run-skill-evals-soft.sh --self-test | ||
| # Skill Tier 2 agent loop (Cursor): /run-skill-evals # with/without; not a CI gate | ||
| # Dependency audit: npm ci && npm audit --audit-level=high | ||
@@ -28,0 +29,0 @@ # create-adsk snapshot: ./scripts/prepare-create-adsk-snapshot.sh |
@@ -18,3 +18,3 @@ --- | ||
| 3. Write `description` as what + when; add near-miss “Do not use for…” when adjacent skills share keywords. | ||
| 4. Add/update `evals/trigger/eval_queries.json` (~20 queries, include near-misses) and behavior `evals/evals.json` when instructions change. | ||
| 4. Add/update `evals/trigger/eval_queries.json` (~20 queries, include near-misses) and behavior `evals/evals.json` when instructions change. For a with/without pass, use `/run-skill-evals`. | ||
| 5. Run `npx --yes skills-ref validate <skill-dir>` before claiming done. | ||
@@ -21,0 +21,0 @@ |
+1
-1
| { | ||
| "name": "create-adsk", | ||
| "version": "0.3.3", | ||
| "version": "0.3.4", | ||
| "description": "Adopt the Agentic Development Starter Kit as a versioned profile (skills + Cursor wiring).", | ||
@@ -5,0 +5,0 @@ "type": "module", |
Sorry, the diff of this file is not supported yet
URL strings
Supply chain riskPackage contains fragments of external URLs or IP addresses, which the package may be accessing at runtime.
URL strings
Supply chain riskPackage contains fragments of external URLs or IP addresses, which the package may be accessing at runtime.
115098
2.81%71
1.43%