🎩 You're Invited:Meet the Socket team at Black Hat in Las Vegas, August 3-6.RSVP →

dcp-wrap

Advanced tools

Install Socket

Detect and block malicious and high-risk dependencies

Install

dcp-wrap - npm Package Compare versions

Comparing version

0.3.0

0.3.1

+408

docs/DCP_UNIVERSAL_DESIGN.md

		# DCP Universal Module — 設計ノート

		## 動機

		dcp-rag の core は RAG に依存していない。`DcpSchema`, `FieldMapping`, `DcpEncoder` は任意の dict → positional array 変換を行う汎用エンジン。RAG はスキーマとプリセットの組み合わせに過ぎない。

		## パッケージ構造案

		```
		dcp/ ← pip install dcp
		core/
		schema.py ← DcpSchema, SchemaRegistry, FieldType (現状のまま)
		mapping.py ← FieldMapping, resolve_path (現状のまま)
		encoder.py ← DcpEncoder, EncodedBatch (現状のまま)
		schemas/
		rag-chunk-meta.v1.json ← RAG用 (dcp[rag] extras)
		rag-query-hint.v1.json
		rag-rerank-signal.v1.json
		rag-result-summary.v1.json
		log-entry.v1.json ← ログ分析用 (dcp[log] extras — 将来)
		api-response.v1.json ← API応答用 (dcp[api] extras — 将来)
		presets/
		rag/ ← RAGプリセット (pinecone, qdrant, ...)
		log/ ← ログプリセット (structured-log, cloudwatch, ...)
		api/ ← APIプリセット (rest-json, graphql, ...)
		adapters/
		rag/ ← RAGフレームワーク (llamaindex, langchain, haystack, azure)
		log/ ← ログフレームワーク (将来)

		dcp-rag/ ← pip install dcp-rag (薄いラッパー、移行期間用)
		→ depends on dcp[rag]
		→ re-exports DcpEncoder, from_preset("pinecone") etc.
		```

		## 移行パス

		```
		Phase 1 (現在): dcp-rag 0.1.x — 単独パッケージとして公開
		Phase 2: dcp 0.1.0 — core を dcp に抽出、dcp-rag は dcp[rag] に依存
		Phase 3: dcp-rag 1.0 → deprecated, "pip install dcp[rag]" に誘導
		```

		## core の汎用性の証明

		encoder.py の主要メソッド:

		```python
		def encode(self, chunks: list[dict], texts: list[str] \| None = None) -> EncodedBatch:
		```

		- `chunks` = 任意の dict のリスト（Vector DB result に限らない）
		- `texts` = テキスト部分（ログのメッセージ本文、API response body、etc.）
		- `DcpSchema` = フィールド定義（RAG固有の知識なし）
		- `FieldMapping` = dot-notation パス解決（任意の nested dict 構造対応）

		RAG 固有のコードは core に存在しない。

		## 汎用ユースケース

		### 1. ログ分析 → LLM

		```json
		{
		"$dcp": "schema",
		"id": "log-entry:v1",
		"fields": ["level", "service", "timestamp", "error_code"],
		"fieldCount": 4,
		"types": {
		"level": {"type": "string", "enum": ["debug", "info", "warn", "error", "fatal"]},
		"service": {"type": "string"},
		"timestamp": {"type": "number", "description": "Unix epoch seconds"},
		"error_code": {"type": ["string", "null"]}
		}
		}
		```

		```python
		encoder = DcpEncoder(
		schema="log-entry:v1",
		mapping={
		"level": "level",
		"service": "service_name",
		"timestamp": "ts",
		"error_code": "error.code",
		},
		group_key="service",
		text_key="message",
		)

		# NL: "Error in auth-service at 2024-03-24 14:30: connection timeout (E_TIMEOUT)"
		# DCP: ["error", "auth-service", 1711284600, "E_TIMEOUT"]
		# connection timeout
		```

		### 2. DB結果 → LLM

		```json
		{
		"$dcp": "schema",
		"id": "db-row:v1",
		"fields": ["table", "column", "value_type", "row_count"],
		"fieldCount": 4
		}
		```

		### 3. API レスポンス → LLM

		```json
		{
		"$dcp": "schema",
		"id": "api-response:v1",
		"fields": ["status", "latency_ms", "endpoint", "method"],
		"fieldCount": 4
		}
		```

		### 4. 設定注入 → LLM

		```json
		{
		"$dcp": "schema",
		"id": "config-param:v1",
		"fields": ["key", "value", "source", "override"],
		"fieldCount": 4
		}
		```

		## 設計原則 (dcp-rag から継承)

		1. データ変換しない — 値は as-is。構造だけ変える
		2. カットダウン — 存在するフィールドだけ。null-fill しない
		3. $G グルーピング — 繰り返しキーの圧縮
		4. スキーマ教育 — $S ヘッダーで LLM にフィールド意味を伝達
		5. LLM境界のみ — 上流のプログラム処理は元データで行う

		## core に必要な変更

		なし。現状の core は汎用的。変更が必要なのは:

		- パッケージ名: `dcp_rag` → `dcp`
		- `from_preset()` のプリセットロード先: ドメイン別に分離
		- schemas ディレクトリ: ドメイン別のサブディレクトリ or タグベースのフィルタリング

		## `from_preset()` の拡張案

		```python
		# 現状 (RAG のみ)
		encoder = DcpEncoder.from_preset("pinecone")

		# 拡張 (ドメイン prefix)
		encoder = DcpEncoder.from_preset("rag:pinecone")
		encoder = DcpEncoder.from_preset("log:cloudwatch")
		encoder = DcpEncoder.from_preset("api:rest-json")

		# あるいは引数
		encoder = DcpEncoder.from_preset("pinecone", domain="rag")
		```

		prefix 方式が良い。理由:
		- 1引数で完結
		- 名前空間の衝突を防ぐ
		- `rag:` prefix がなければ RAG がデフォルト (後方互換)

		## SchemaRegistry の拡張案

		```python
		# 現状: ディレクトリ全部ロード
		registry = SchemaRegistry("schemas/")

		# 拡張: ドメインフィルタ
		registry = SchemaRegistry("schemas/", domain="rag") # rag-* のみロード
		registry = SchemaRegistry("schemas/", domain="log") # log-* のみロード
		registry = load_default_registry(domain="rag") # 便利関数
		```

		あるいはスキーマ ID の prefix 規約で十分:
		- `rag-chunk-meta:v1` → RAG ドメイン
		- `log-entry:v1` → ログドメイン
		- `api-response:v1` → API ドメイン

		Registry はフラットで全部持ち、`get()` で ID 指定するだけ。ドメインフィルタは不要。

		## まとめ

		dcp-rag の core は既に dcp 汎用モジュールそのもの。必要なのは:

		1. パッケージ名変更 (`dcp`)
		2. プリセットの名前空間分離 (`rag:pinecone` 形式)
		3. 新ドメインのスキーマ定義 (JSON ファイル追加のみ)
		4. 新ドメインのプリセット定義 (Python dict 追加のみ)

		core のロジック変更はゼロ。

		---

		## Interactive Schema — 実装済み設計 (2026-03-25)

		> engram gateway に Stage 0/1 を実装済み。以下は初期構想から実装を経て確定した設計。

		### 背景: エージェントは DCP を知っていても従わない

		実証済み: CLAUDE.md に DCP 仕様が記載され、engram が native 推奨警告を返しても、エージェントは自然言語で push した。LLM は仕様を「知っている」ことと「従う」ことの間にギャップがある。recency bias により、コンテキストウィンドウの拡大ではこの問題は解決しない。

		### 密度スペクトラム（3段階）

		schema は文脈に応じて自身の表現密度を変える:

		\| 密度 \| いつ \| 形式 \| コスト \|
		\|------\|------\|------\|--------\|
		\| Abbreviated \| consumer が schema を知っている \| `$S:knowledge:v1#fcbc [expand:GET /schemas/knowledge:v1]` \| ~5 tokens \|
		\| Expanded \| consumer にリマインダーが必要 \| `$S:knowledge:v1#fcbc [action(add\\|replace\\|flag\\|remove) target domain weight:0-1] [expand:...]` \| ~30 tokens \|
		\| Full \| consumer が初見 \| フィールド定義全体 + 型 + enum + 例 \| ~80+ tokens \|

		### スキーマヒントの実動作（engram 実装済み）

		engram gateway の `determineSchemaHint()` が push レスポンスに含めるヒントを決定:

		- DCP-native かつ schema-valid → abbreviated のみ（最小コスト）
		- 自然言語の push → expanded ヒント（パッシブ教育）
		- schema 違反 → データは受理 + 警告 + expanded ヒント

		パッシブ教育原則: reject しない、warn のみ。 gate.ts で schema violation は `warnings.push()` に変更済み（`errors.push()` ではない）。コスト勾配が自然なインセンティブ — DCP 準拠データは処理コストが低い。非準拠でも動く、ただしコストが高い。

		### スキーマレジストリ（engram 実装済み）

		`gateway/schemas/*.json` を SSOT として一元管理。API で動的参照可能:

		```
		GET /schemas → スキーマ一覧
		GET /schemas/:id → フル定義
		```

		tool description にスキーマをインライン埋め込み + push レスポンスにヒント + API で能動的参照 = 3層の教育動線。

		### スキーマプリメソッド（設計のみ・4つに凍結）

		\| メソッド \| 意味 \|
		\|----------\|------\|
		\| `$S?` \| schema query — "このスキーマは何？" \|
		\| `$S!` \| schema declaration — "このスキーマで送る" \|
		\| `$SV` \| schema validation — "準拠しているか？" \|
		\| `$S+` \| schema expansion — "フル定義をくれ" \|

		将来のマルチエージェントハンドシェイク用インフラ。現時点でエージェントが能動的にトリガすることはない。

		### エージェントプロファイル適応（設計のみ）

		エージェントごとの DCP 準拠率を観測し、ヒント密度を自動調整する:

		```
		agent_profile {
		agentId: string
		errorRate: float // 直近 N 回の非準拠率
		hintStage: 0 \| 1 \| 2 \| 3 // 適用中のヒント密度
		anchorDensity: number // リマインダー頻度 (0 = なし)
		}

		初見 → 保守的 (expanded + 高密度アンカー)
		高精度 → abbreviated + アンカーなし
		中精度 → expanded + 適度なアンカー
		低精度 → full + 高密度アンカー
		改善傾向 → 段階的に密度を下げる
		悪化傾向 → 即座に密度を上げる
		```

		TCP 輻輳制御と同構造: slow start → congestion avoidance。ペナルティではなくコスト最適化 — 低精度エージェントは罰を受けるのではなく、必要な情報量を提供されているだけ。

		### OUT 側フォーマッタとの協調

		Interactive Schema は入口の改善 — エージェントが DCP に近づく確率を上げる。
		OUT 側フォーマッタは出口の保証 — エージェントが何を出しても DCP になる。

		```
		Interactive Schema (入口) OUT Formatter (出口)
		abbreviated で周辺視野を提供 bitmask で入力を判定
		$S+ で展開を提供 cutdown で positional array に成型
		教育コストグラデーション streaming 適性（1 件ずつ処理）
		↓ ↓
		エージェントの改善を促す改善しなくても動く
		```

		両方あって完全 — 入口だけでは準拠を保証できない。出口だけでは改善が起きない。

		### bitmask cutdown のストリーミング適性

		\| 入力タイプ \| bitmask 判定 \| フォーマッタの処理 \|
		\|---\|---\|---\|
		\| 完全 DCP \| full_mask 一致 \| 素通し \|
		\| 部分 DCP \| 部分 bit \| cutdown 成型 \|
		\| structured NL \| キー名マッピング \| positional 化 \|
		\| 生 NL \| mask=0 \| summary + tags から最低限 array \|

		バッチ全体の OR で bitmask を判定すると、1 件の良いデータが全体の null 埋めを誘発する。ストリーミング（1 件ずつ判定）ではこの問題は起きない。マルチエージェントの push は本質的にストリーミング。

		### DB データ管理との構造的一致

		\| DCP \| DB equivalent \|
		\|---\|---\|
		\| bitmask \| カラムナー DB の null bitmap \|
		\| cutdown \| スキーマプロジェクション (`SELECT col1, col3`) \|
		\| `$G` \| `GROUP BY` \|
		\| ストリーミング単位処理 \| row-level evaluation \|
		\| abbreviated schema \| インデックスページ（B-Tree の内部ノード） \|
		\| interactive methods \| ストアドプロシージャ参照 \|

		偶然の一致ではなく、データを効率的に扱う問題の解は収束する。

		### 設計原則

		\| # \| 原則 \|
		\|---\|---\|
		\| 1 \| schema が全ての起点 — データ定義も操作方法も schema から生える \|
		\| 2 \| abbreviated が日常、展開は例外 — 常時コストは最小、必要時だけ拡大 \|
		\| 3 \| 準拠は経済的インセンティブ — 従えばコスト減、従わなければコスト増。reject しない \|
		\| 4 \| 入口は改善、出口は保証 — Interactive Schema + OUT フォーマッタの二重構造 \|
		\| 5 \| 観測 → 適応 — エージェント能力はシステムが観測し、密度を自動調整する \|

		---

		## Shadow の使い方と作法 (2026-03-31)

		### シャドウのライフタイム原則

		シャドウはスキーマID × 接続ID にバインドされる。それ以上でも以下でもない。

		```
		shadow → bound to → schemaId (e.g. "hotmemo:v1") + connectionId
		```

		- 接続IDをライフタイムにする — MCPコネクション確立でシャドウが生まれ、切断で廃棄される。セッションをまたいだ持ち越しは原則しない
		- スキーマIDが変わればシャドウは死ぬ — `hotmemo:v1` のシャドウは `hotmemo:v2` には使えない。IDが変わった時点でエラーとする
		- バージョン管理はしない — シャドウに独立したバージョン管理を持ち込まない。スキーマIDの更新がバージョン管理を兼ねる

		### 使い捨ての論理

		シャドウは本質的に使い捨て。この性質を利用する:

		```
		コネクション確立
		→ スキーマ解決 (schemaId 確定)
		→ シャドウをスキーマIDにアタッチ
		→ バリデーション/ルーティング動作
		→ コネクション切断 → シャドウ全廃棄
		```

		再利用したい場合は必ず明示的なガードを通す:

		```typescript
		// 再利用前に必ずスキーマID一致チェック
		if (shadow.schemaId !== currentSchema.id) {
		throw new Error(`shadow bound to ${shadow.schemaId}, current schema is ${currentSchema.id}`);
		}
		```

		### フォールバック設計

		バリデーションシャドウが未知フィールド・定義外の表現に出会った時の原則:

		- 未知フィールド → pass-through — 捨てずに通過させる。シャドウが知らないフィールドはシャドウの管轄外
		- 定義外の表現 → pass-through — silent drop ではなく `unknown` として扱う
		- バリデーション失敗 = ストリームの終わりではない — 1行が失敗しても後続行は処理を続ける

		```
		バリデーションシャドウの判定:
		known field, valid value → pass
		known field, invalid value → fail (log + route to AI)
		unknown field → pass-through (shadow の管轄外)
		定義外の表現 → pass-through (unknown として保持)
		```

		### スキーマ定義内でのシャドウ宣言

		`DcpSchemaDef` の `shadows?` フィールドにアタッチできる:

		```json
		{
		"$dcp": "schema",
		"id": "hotmemo:v1",
		"fields": ["summary", "tags", "domain", "weight"],
		"shadows": {
		"validation": {
		"schemaId": "hotmemo:v1",
		"fields": {
		"weight": { "minLength": 0 },
		"tags": { "pattern": "^[a-z0-9,]+$" }
		}
		},
		"routing": {
		"schemaId": "hotmemo:v1",
		"minLevel": 1,
		"access": ["ops"]
		}
		}
		}
		```

		シャドウはスキーマ定義内に宣言できるが、着脱可能。`shadows` を丸ごと除いてもボディは影響を受けない。

		### まとめ

		\| 原則 \| 内容 \|
		\|---\|---\|
		\| バインドキー \| `schemaId` — IDが変わればシャドウは無効 \|
		\| ライフタイム \| MCPコネクション単位 — 切断で廃棄 \|
		\| バージョン管理 \| しない — スキーマIDの更新で吸収 \|
		\| 未知フィールド \| pass-through — シャドウの管轄外として通過 \|
		\| 再利用時 \| schemaId一致チェックを必ず通す \|
		\| 設計思想 \| 使い捨てが正しい。セッション内で完結する \|

+343

docs/SCHEMA_GENERATION.md

		# DCP Schema Generation & Output Controller

		設計メモ。実装しながら固める。確定後に dcp-docs へ移植。

		## 前提

		```
		既存データ → DCP Schema 生成 → DCP Encoder → DCP Data

		スキーマ生成 = エンコーダ生成。スキーマが決まればエンコーダは自動的に決まる。
		```

		## 1. Schema Generator — 既存データからスキーマを生成する

		### 問題

		DCP スキーマには作法がある（フィールド順序、型定義、enum 検出、命名規則 `:v1` など）。現状はスキーマもマッピングも手書きで、作法の遵守は書く人間/AI に依存する:

		```python
		# 人間が JSON を書く — 作法を知っている前提
		schema = DcpSchema.from_file("schemas/my-domain.v1.json")
		# 人間がマッピングを書く
		mapping = FieldMapping(schema_id="my-domain:v1", paths={"field": "data.field", ...})
		```

		作法をコードに埋め込んだジェネレータがあれば、誰が使っても DCP 準拠のスキーマが出る。

		### 解決

		データサンプルからスキーマを推論する:

		```
		入力: データサンプル（JSON object, DB result, dict, ...）
		+ オプション: ドメイン名、フィールド選択、型ヒント

		出力: DcpSchema + FieldMapping（ドラフト）
		```

		### 推論すべきもの

		```
		1. フィールド抽出
		- トップレベルキー、ネストされたキー（dot-notation で展開）
		- 複数サンプルでの出現率 → 常在フィールド vs 稀なフィールド

		2. 型推定
		- 値の観察: string / number / boolean / null / array
		- enum 検出: 値の種類が少なければ enum 候補
		- range 検出: 数値フィールドの min/max

		3. フィールド順序（作法）
		- 高頻度フィールドを先頭に（cutdown 時に生き残りやすい）
		- 識別子系 → 分類系 → 数値系 → テキスト系
		- group_key 候補の検出（重複率が高いフィールド）

		4. マッピング自動生成
		- スキーマフィールド名とソースのキー名が同一 → 自動バインド
		- ネストされている場合は dot-notation パスを生成
		- 一致しないフィールドだけ人間/AI が補完
		```

		### API 案

		```python
		from dcp_rag.core.generator import SchemaGenerator

		# データサンプルから推論
		gen = SchemaGenerator()
		draft = gen.from_samples(
		samples=[chunk1, chunk2, chunk3, ...],
		domain="my-domain", # → スキーマ ID: "my-domain:v1"
		include=["field1", "field2"], # 含めるフィールド（省略時: 全フィールド）
		exclude=["internal_id"], # 除外するフィールド
		)

		# ドラフト確認
		print(draft.schema) # DcpSchema
		print(draft.mapping) # FieldMapping
		print(draft.report) # 推論レポート（型推定根拠、enum 候補など）

		# 確定 → ファイル保存
		draft.save("schemas/my-domain.v1.json")

		# そのままエンコーダ生成
		encoder = draft.to_encoder()
		result = encoder.encode(data)
		```

		---

		## 2. Output Controller — LLM に DCP を出力させる

		### 問題

		LLM は DCP を正確に出力できない（軽量モデルテスト: 正しいフィールド順序 = 0%）。
		大型モデルでも不安定。しかし LLM が DCP データを出力すべき場面がある。

		### 人間との類推

		```
		人間: 自由タイピング → フォーム/GUI が構造を強制 → 構造化データ
		LLM: 自由テキスト生成 → 出力コントローラが配置 → DCP データ
		```

		### 「encoder ではなく validator」との関係

		```
		否定したもの（NL→DCP 常駐 encoder）:
		LLM → "auth の jwt を変更した" → encoder が意味推論 → DCP
		問題: encoder が意味を解釈する。推論エラーの温床。

		提案するもの（意味決定と構造化の分離）:
		LLM → {action:"replace", domain:"auth", ...} → controller が配置 → DCP
		LLM が意味を決定。controller は並べるだけ。推論なし。
		```

		controller は意味を推論しない。LLM が key-value で意味を出し、controller がスキーマに従って positional array に配置する。MCP tool use の JSON Schema 引数強制と同じ発想。

		### 設計

		```
		入力: LLM の出力（key-value object, tool use 引数, 構造化テキスト）
		+ スキーマ ID

		処理:
		1. スキーマをロード → フィールド順序を取得
		2. 入力の key をスキーマフィールドにマッチ
		3. 順序通りに配置 → positional array
		4. バリデーション（型チェック、enum チェック）

		出力: DCP positional array
		```

		### API 案

		```python
		from dcp_rag.core.controller import OutputController

		ctrl = OutputController(schema="knowledge:v1")

		# key-value dict → DCP array
		row = ctrl.place({"action": "replace", "domain": "auth", "detail": "jwt migration", "confidence": 0.9})
		# → ["replace", "auth", "jwt migration", 0.9]

		# 不足フィールドは None
		row = ctrl.place({"action": "flag", "domain": "payment"})
		# → ["flag", "payment", None, None] or cutdown

		# 余分なキーは無視（安全）
		row = ctrl.place({"action": "add", "domain": "auth", "detail": "...", "confidence": 0.8, "extra": "ignored"})
		# → ["add", "auth", "...", 0.8]

		# バリデーション
		result = ctrl.place({"action": "invalid_value", ...})
		# → ValidationError: action must be one of [add, replace, flag, remove]
		```

		### MCP tool use との統合

		```
		engram_push の native フィールド:
		現状: LLM が直接 positional array を書く → エラーが多い
		改善: LLM が key-value で出す → OutputController が配置 → native に格納

		tool schema:
		native: { action: "replace", domain: "auth", ... } // LLM はこちらを書く
		→ controller → ["replace", "auth", ...] // システムが変換
		```

		---

		## 3. 統合 — Generator + Controller

		```
		Phase 1: スキーマ生成
		既存データサンプル → SchemaGenerator → DcpSchema + FieldMapping

		Phase 2: データ変換（システム側）
		既存データ → DcpEncoder (スキーマ駆動) → DCP Data

		Phase 3: LLM 出力（AI 側）
		LLM key-value 出力 → OutputController → DCP Data

		全て同一のスキーマから駆動される。スキーマが SSOT。
		```

		---

		## 4. Gateway — スキーマの運用者

		### スキーマは辞書、ゲートウェイは司書

		スキーマ単体は静的な定義。ゲートウェイがスキーマを運用する:

		```
		スキーマ = データの定義（静的）
		- フィールド定義、型定義、バリデーションルール
		- JSON ファイルに閉じている

		ゲートウェイ = スキーマの運用者（動的）
		- 誰に、どの密度で、いつ渡すか（shadow level 判断）
		- エージェントの準拠率を観測して密度を調整
		- スキーマの展開要求（$S? → full 返却）に応答
		- バリデーション結果をエージェントプロファイルに反映
		```

		### Gateway の構成

		```
		Gateway
		├── SchemaRegistry（全スキーマ定義を保持）
		│ ├── hotmemo:v1
		│ ├── knowledge:v1
		│ └── rag-chunk-meta:v1
		│
		├── AgentProfile（エージェントごとの観測データ）
		│ ├── errorRate, hintStage
		│ └── → shadow_level を決定
		│
		├── Encoder（スキーマ駆動、レジストリから取得）
		│ └── → shadow_level を受けて変換するだけ。判断しない。
		│
		└── Validator（LLM 出力の準拠チェック）
		└── → スキーマをレジストリから参照
		```

		### Shadow Level と Encoder の責務分離

		```
		密度の判断: Gateway（AgentProfile の errorRate を観測）
		密度の実行: Encoder（shadow_level を引数で受け取る）
		```

		Encoder は「言われた通りに変換する」。shadow level の決定権は外にある。

		### $S ヘッダ各要素の必要性

		```
		$S ヘッダ: ["$S","rag-chunk-meta:v1#hash",5,"source","page","section","score","chunk_index"]

		"$S" → システムのパーサ用。LLM には不要。
		"rag-chunk-meta:v1" → 複数スキーマを同時に扱う時の識別子。
		"#hash" → セッション内でスキーマ定義を再取得せずに参照する。
		"5" → フィールド数。パーサ用。LLM はデータを見ればわかる。
		field names → LLM がデータを解釈するのに必要。唯一の本質。
		```

		### Shadow Level 定義 — 能力 × 処理複雑度の2軸

		field names が全ての基本線。プロトコル情報は複雑な処理に耐える AI のためのオプション。

		```
		単一スキーマ処理複数スキーマ同時処理
		────────────── ──────────────────
		高性能 AI field names $S + ID + hash + fields
		(abbreviated で十分)

		中性能 field names field names + schema ID

		軽量 LLM field names のみ複数スキーマを同時に
		扱わせるべきでない
		```

		### Encoder shadow_level

		```
		encoder.encode(chunks, shadow_level=0)
		# L0: fields only — フィールド名 + データ行のみ
		# [source, page, section, score, chunk_index]
		# ["docs/auth.md",12,"JWT Config",0.92,3]

		encoder.encode(chunks, shadow_level=1)
		# L1: with schema ID — 複数スキーマ識別用
		# ["$S","rag-chunk-meta:v1","source","page","section","score","chunk_index"]
		# ["docs/auth.md",12,"JWT Config",0.92,3]

		encoder.encode(chunks, shadow_level=2)
		# L2: full protocol — $S + ID + hash + field count + fields
		# ["$S","rag-chunk-meta:v1#hash",5,"source","page","section","score","chunk_index"]
		# ["docs/auth.md",12,"JWT Config",0.92,3]

		encoder.encode(chunks, shadow_level=3)
		# L3: full schema definition（初回/教育用）
		# { id, fields, types, examples... }
		# ["docs/auth.md",12,"JWT Config",0.92,3]

		encoder.encode(chunks, shadow_level=4)
		# L4: NL fallback（最終手段）
		# Source: docs/auth.md, Page: 12, Section: JWT Config, Score: 0.92, Chunk: 3
		```

		### 動線

		```
		データ入力時:
		データ → Gateway → AgentProfile 参照 → shadow_level 決定
		→ SchemaRegistry からスキーマ取得
		→ Encoder に shadow_level 付きで渡す
		→ DCP 出力 → エージェントへ

		エージェント応答時:
		応答 → Gateway → Validator でスキーマ準拠チェック
		→ 結果を AgentProfile に反映
		→ 次回の shadow_level に影響（フィードバックループ）
		```

		---

		## 5. 全体像 — スキーマから全てが駆動される

		```
		SchemaGenerator
		既存データサンプル → DcpSchema + FieldMapping 生成
		│
		▼
		SchemaRegistry（Gateway が保持）
		│
		├──→ Encoder（データ → DCP 変換）
		│ shadow_level は Gateway が決定
		│
		├──→ OutputController（LLM 出力 → DCP 配置）
		│ LLM が意味を決定、controller が並べるだけ
		│
		├──→ Validator（LLM 出力のチェック）
		│ 結果を AgentProfile にフィードバック
		│
		└──→ AgentProfile（観測データ蓄積）
		→ 次回の shadow_level を決定
		```

		スキーマが SSOT。Generator で作り、Registry で保持し、Encoder / Controller / Validator が参照する。Gateway がこれらを束ねて、エージェントごとに最適な密度で運用する。

		---

		## 実装優先度

		```
		1. SchemaGenerator.from_samples() — 型推定、フィールド順序、自動マッピング
		2. OutputController.place() — key-value → positional array 配置
		3. Encoder shadow_level 対応 — L0〜L3 の密度切り替え
		4. 自動バインド (FieldMapping) — 同名フィールドのマッピング不要化
		5. MCP 統合例 — engram_push での利用パターン
		```

+44

-0

dist/types.d.ts

		@@ -9,2 +9,37 @@ /** Type definition for a single schema field. */
		}
		/**
		* Validation shadow — independent verification lens attached to a schema.
		* Shadows are disposable: removing one does not affect the body or other shadows.
		* Bound to schemaId; if the schema ID changes, this shadow is invalid.
		*/
		export interface ValidationShadow {
		/** Schema this shadow is bound to. Must match DcpSchemaDef.id exactly. */
		schemaId: string;
		/** Per-field constraints, keyed by field name. Unrecognized fields are passed through. */
		fields?: Record<string, ValidationShadowField>;
		}
		/** Per-field constraint set for a validation shadow. */
		export interface ValidationShadowField {
		/** Regex pattern the value must match (string fields). */
		pattern?: string;
		/** Maximum string length. */
		maxLength?: number;
		/** Minimum string length. */
		minLength?: number;
		}
		/**
		* Routing shadow — declarative distribution control attached to a schema.
		* Declares who receives data and under what conditions.
		* Bound to schemaId; system reads and executes, does not own the logic.
		*/
		export interface RoutingShadow {
		/** Schema this shadow is bound to. Must match DcpSchemaDef.id exactly. */
		schemaId: string;
		/** Minimum agent density level required (L0–L4). */
		minLevel?: number;
		/** Group-level access control. */
		access?: string[];
		/** Field-value conditions for inclusion. */
		filter?: Record<string, unknown[]>;
		}
		/** DCP schema definition (the $dcp JSON document). */
		@@ -20,2 +55,11 @@ export interface DcpSchemaDef {
		nestSchemas?: Record<string, NestSchemaDef>;
		/**
		* Shadows attached to this schema definition.
		* Each shadow is independently disposable — attach one, some, or all.
		* All shadows are bound to this schema's id; id change = shadows invalid.
		*/
		shadows?: {
		validation?: ValidationShadow;
		routing?: RoutingShadow;
		};
		}
		@@ -22,0 +66,0 @@ /** Sub-schema + mapping pair for a nested array-of-objects field. */

+1

-2

docs/mcp-server-contract.md

		@@ -187,3 +187,2 @@ # MCP Server DCP Contract
		- [DCP Specification](https://dcp-docs.pages.dev/dcp/specification) — full protocol design
		- [dcp-wrap](https://github.com/hiatamaworkshop/dcp-wrap) — encoder/decoder library + CLI
		- [PicoClaw integration](./picoclaw-integration.md) — framework-specific hook setup
		- [dcp-wrap](https://github.com/hiatamaworkshop/dcp-wrap) — encoder/decoder library + CLI

+13

-6

package.json

		{
		"name": "dcp-wrap",
		"version": "0.3.0",
		"version": "0.3.1",
		"description": "Convert JSON to DCP positional-array format for AI agents — 40-70% token reduction",
		@@ -9,4 +9,3 @@ "type": "module",
		"bin": {
		"dcp-wrap": "dist/cli.js",
		"dcp-picoclaw-hook": "dist/picoclaw-hook.js"
		"dcp-wrap": "dist/cli.js"
		},
		@@ -26,3 +25,11 @@ "exports": {
		"files": [
		"dist",
		"dist/*.js",
		"dist/*.js.map",
		"dist/*.d.ts",
		"!dist/*.test.js",
		"!dist/*.test.js.map",
		"!dist/*.test.d.ts",
		"!dist/picoclaw-hook.js",
		"!dist/picoclaw-hook.js.map",
		"!dist/picoclaw-hook.d.ts",
		"README.md",
		@@ -48,6 +55,6 @@ "docs"
		"positional-array",
		"picoclaw",
		"hook"
		"schema",
		"compression"
		],
		"license": "Apache-2.0"
		}

+0

-6

README.md

		@@ -247,10 +247,4 @@ # dcp-wrap

		## PicoClaw Integration

		dcp-wrap ships with an out-of-process hook for [PicoClaw](https://github.com/sipeed/picoclaw) that DCP-encodes tool results before they reach the LLM. No core modification needed — just configure a process hook.

		See [docs/picoclaw-integration.md](docs/picoclaw-integration.md) for setup guide, Docker instructions, and gotchas.

		## License

		Apache-2.0

-1

dist/decoder.test.d.ts

export {};

-144

dist/decoder.test.js

		import { describe, it } from "node:test";
		import assert from "node:assert/strict";
		import { DcpSchema } from "./schema.js";
		import { DcpDecoder } from "./decoder.js";
		const reportDef = {
		$dcp: "schema",
		id: "ctrl-report:v1",
		description: "Task completion report controller",
		fields: ["action", "target", "detail", "cost"],
		fieldCount: 4,
		types: {
		action: { type: "string", enum: ["done", "error", "skip", "partial"] },
		target: { type: "string" },
		detail: { type: ["string", "null"] },
		cost: { type: ["number", "null"] },
		},
		};
		const reportTemplates = {
		done: "✓ {{target}} — {{detail}} ({{cost}}ms)",
		error: "✗ {{target}} — {{detail}}",
		skip: "⊘ {{target}} skipped",
		partial: "◐ {{target}} — {{detail}}",
		default: "{{action}} {{target}} {{detail}}",
		};
		const logDef = {
		$dcp: "schema",
		id: "ctrl-log:v1",
		description: "Event log controller",
		fields: ["ts", "level", "source", "msg"],
		fieldCount: 4,
		types: {
		ts: { type: "string" },
		level: { type: "string", enum: ["info", "warn", "error", "debug"] },
		source: { type: "string" },
		msg: { type: "string" },
		},
		};
		const logTemplates = {
		error: "[{{ts}}] ERROR {{source}}: {{msg}}",
		warn: "[{{ts}}] WARN {{source}}: {{msg}}",
		info: "[{{ts}}] INFO {{source}}: {{msg}}",
		default: "[{{ts}}] {{level}} {{source}}: {{msg}}",
		};
		describe("DcpDecoder", () => {
		describe("decode", () => {
		it("decodes report:done with template", () => {
		const schema = new DcpSchema(reportDef);
		const decoder = new DcpDecoder(schema, reportTemplates);
		const result = decoder.decode(["done", "/v1/auth", "200 ok", 42]);
		assert.deepEqual(result.keyValues, {
		action: "done",
		target: "/v1/auth",
		detail: "200 ok",
		cost: 42,
		});
		assert.equal(result.text, "✓ /v1/auth — 200 ok (42ms)");
		});
		it("decodes report:error", () => {
		const schema = new DcpSchema(reportDef);
		const decoder = new DcpDecoder(schema, reportTemplates);
		const result = decoder.decode(["error", "/v1/orders", "timeout", 0]);
		assert.equal(result.text, "✗ /v1/orders — timeout");
		});
		it("decodes report:skip with nulls", () => {
		const schema = new DcpSchema(reportDef);
		const decoder = new DcpDecoder(schema, reportTemplates);
		const result = decoder.decode(["skip", "cleanup", null, null]);
		assert.equal(result.text, "⊘ cleanup skipped");
		});
		it("decodes log row with level-specific template", () => {
		const schema = new DcpSchema(logDef);
		const decoder = new DcpDecoder(schema, logTemplates, "level");
		const result = decoder.decode([
		"2026-03-28T14:30:00Z",
		"error",
		"db-writer",
		"connection refused",
		]);
		assert.equal(result.text, "[2026-03-28T14:30:00Z] ERROR db-writer: connection refused");
		});
		it("falls back to field:value when no templates", () => {
		const schema = new DcpSchema(reportDef);
		const decoder = new DcpDecoder(schema);
		const result = decoder.decode(["done", "/v1/auth", "ok", 42]);
		assert.equal(result.text, "action: done \| target: /v1/auth \| detail: ok \| cost: 42");
		});
		});
		describe("decodeRows", () => {
		it("decodes multiple rows", () => {
		const schema = new DcpSchema(reportDef);
		const decoder = new DcpDecoder(schema, reportTemplates);
		const results = decoder.decodeRows([
		["done", "/v1/auth", "200 ok", 42],
		["error", "/v1/orders", "timeout", 0],
		]);
		assert.equal(results.length, 2);
		assert.ok(results[0].text.includes("/v1/auth"));
		assert.ok(results[1].text.includes("/v1/orders"));
		});
		});
		describe("decodeRaw", () => {
		it("parses header + rows from raw DCP string", () => {
		const schema = new DcpSchema(reportDef);
		const decoder = new DcpDecoder(schema, reportTemplates);
		const raw = [
		'["$S","ctrl-report:v1","action","target","detail","cost"]',
		'["done","/v1/auth","200 ok",42]',
		'["error","/v1/orders","timeout",0]',
		].join("\n");
		const { header, results } = decoder.decodeRaw(raw);
		assert.equal(header[0], "$S");
		assert.equal(results.length, 2);
		assert.ok(results[0].text.includes("✓"));
		assert.ok(results[1].text.includes("✗"));
		});
		it("handles rows without header", () => {
		const schema = new DcpSchema(reportDef);
		const decoder = new DcpDecoder(schema, reportTemplates);
		const raw = '["done","/v1/auth","ok",42]';
		const { header, results } = decoder.decodeRaw(raw);
		assert.equal(header.length, 0);
		assert.equal(results.length, 1);
		});
		});
		describe("with DcpSchema.validateRow", () => {
		it("validate + decode roundtrip", () => {
		const schema = new DcpSchema(reportDef);
		const decoder = new DcpDecoder(schema, reportTemplates);
		const row = ["done", "/v1/auth", "200 ok", 42];
		const errors = schema.validateRow(row);
		assert.equal(errors.length, 0);
		const result = decoder.decode(row);
		assert.equal(result.text, "✓ /v1/auth — 200 ok (42ms)");
		});
		it("validation catches bad enum", () => {
		const schema = new DcpSchema(reportDef);
		const row = ["invalid", "/v1/auth", "ok", 42];
		const errors = schema.validateRow(row);
		assert.ok(errors.length > 0);
		assert.ok(errors[0].includes("not in enum"));
		});
		});
		});
		//# sourceMappingURL=decoder.test.js.map

-1

dist/decoder.test.js.map

-1

dist/encoder.test.d.ts

export {};

-194

dist/encoder.test.js

		import { describe, it } from "node:test";
		import assert from "node:assert/strict";
		import { SchemaGenerator } from "./generator.js";
		import { DcpSchema } from "./schema.js";
		import { DcpEncoder } from "./encoder.js";
		import { FieldMapping } from "./mapping.js";
		const userSamples = [
		{
		id: "u001",
		name: "Alice Johnson",
		email: "alice@example.com",
		role: "admin",
		score: 95,
		active: true,
		profile: {
		bio: "Platform engineer, 10 years experience",
		avatar_url: "https://cdn.example.com/avatars/alice.png",
		preferences: {
		theme: "dark",
		locale: "en-US",
		notifications: { email: true, push: false, sms: false },
		},
		},
		teams: [
		{ id: "t01", name: "Infrastructure", role: "lead" },
		{ id: "t02", name: "Security", role: "member" },
		],
		recent_activity: [
		{
		action: "deploy",
		target: "api-v2",
		timestamp: "2026-03-28T14:30:00Z",
		metadata: { env: "production", version: "2.1.0" },
		},
		{
		action: "review",
		target: "PR#412",
		timestamp: "2026-03-27T09:15:00Z",
		metadata: { verdict: "approved" },
		},
		],
		},
		{
		id: "u002",
		name: "Bob Smith",
		email: "bob@example.com",
		role: "user",
		score: 72,
		active: true,
		profile: {
		bio: "Junior dev, learning the ropes",
		avatar_url: "https://cdn.example.com/avatars/bob.png",
		preferences: {
		theme: "light",
		locale: "ja-JP",
		notifications: { email: true, push: true, sms: false },
		},
		},
		teams: [{ id: "t03", name: "Frontend", role: "member" }],
		recent_activity: [
		{
		action: "commit",
		target: "feature/navbar",
		timestamp: "2026-03-28T16:00:00Z",
		metadata: { files_changed: 3 },
		},
		],
		},
		{
		id: "u003",
		name: "Charlie Brown",
		email: "charlie@example.com",
		role: "user",
		score: 88,
		active: false,
		profile: {
		bio: "On sabbatical",
		avatar_url: null,
		preferences: {
		theme: "dark",
		locale: "en-GB",
		notifications: { email: false, push: false, sms: false },
		},
		},
		teams: [],
		recent_activity: [],
		},
		];
		describe("Nested DCP Encoding", () => {
		it("generates nestSchemas on DcpSchemaDef", () => {
		const gen = new SchemaGenerator();
		const draft = gen.fromSamples(userSamples, { domain: "search_users" });
		assert.ok(draft.schema.nestSchemas, "schema should have nestSchemas");
		assert.ok(draft.schema.nestSchemas["teams"], "teams nestSchema should exist");
		assert.ok(draft.schema.nestSchemas["recent_activity"], "recent_activity nestSchema should exist");
		const teamsSub = draft.schema.nestSchemas["teams"];
		assert.equal(teamsSub.schema.id, "search_users.teams:v1");
		assert.ok(teamsSub.schema.fields.length > 0);
		assert.ok(teamsSub.mapping.paths);
		});
		it("encodes array-of-objects with $R references (no preamble)", () => {
		const gen = new SchemaGenerator();
		const draft = gen.fromSamples(userSamples, { domain: "search_users" });
		const schema = new DcpSchema(draft.schema);
		const mapping = new FieldMapping(draft.mapping);
		const encoder = new DcpEncoder(schema, mapping);
		const batch = encoder.encode(userSamples);
		const output = DcpEncoder.toString(batch);
		const lines = output.split("\n");
		// First line is the main header (no preamble)
		const mainHeader = JSON.parse(lines[0]);
		assert.equal(mainHeader[0], "$S");
		assert.equal(mainHeader[1], "search_users:v1");
		// Alice's row
		const aliceRow = JSON.parse(lines[1]);
		const teamsIdx = mainHeader.indexOf("teams") - 2;
		const teamsVal = aliceRow[teamsIdx];
		// teams: ["$R", "search_users.teams:v1", [row1], [row2]]
		assert.ok(Array.isArray(teamsVal), "teams should be array");
		assert.equal(teamsVal[0], "$R", "nested field starts with $R");
		assert.ok(teamsVal[1].startsWith("search_users.teams:"), "$R references sub-schema ID");
		assert.equal(teamsVal.length, 4, "$R + schemaId + 2 team rows");
		assert.ok(Array.isArray(teamsVal[2]), "team row 1 is an array");
		});
		it("handles empty arrays as $R with no rows", () => {
		const gen = new SchemaGenerator();
		const draft = gen.fromSamples(userSamples, { domain: "search_users" });
		const schema = new DcpSchema(draft.schema);
		const mapping = new FieldMapping(draft.mapping);
		const encoder = new DcpEncoder(schema, mapping);
		const batch = encoder.encode(userSamples);
		const lines = DcpEncoder.toString(batch).split("\n");
		const mainHeader = JSON.parse(lines[0]);
		// Charlie's row (3rd data row)
		const charlieRow = JSON.parse(lines[3]);
		const teamsIdx = mainHeader.indexOf("teams") - 2;
		// Empty → ["$R", "search_users.teams:v1"]
		const teamsVal = charlieRow[teamsIdx];
		assert.ok(Array.isArray(teamsVal), "empty teams should be array");
		assert.equal(teamsVal[0], "$R", "starts with $R");
		assert.equal(teamsVal.length, 2, "$R + schemaId only, no rows");
		});
		it("nestSchemas persist in schema JSON (serializable)", () => {
		const gen = new SchemaGenerator();
		const draft = gen.fromSamples(userSamples, { domain: "search_users" });
		// Round-trip: serialize → parse → DcpSchema
		const json = JSON.stringify(draft.schema);
		const parsed = JSON.parse(json);
		const restored = new DcpSchema(parsed);
		assert.ok(restored.def.nestSchemas, "nestSchemas survives round-trip");
		assert.ok(restored.def.nestSchemas["teams"]);
		// Encode with restored schema
		const mapping = new FieldMapping(draft.mapping);
		const encoder = new DcpEncoder(restored, mapping);
		const batch = encoder.encode(userSamples);
		const lines = DcpEncoder.toString(batch).split("\n");
		const row = JSON.parse(lines[1]);
		const header = JSON.parse(lines[0]);
		const teamsIdx = header.indexOf("teams") - 2;
		assert.equal(row[teamsIdx][0], "$R", "$R works after schema round-trip");
		});
		it("produces smaller output than JSON", () => {
		const gen = new SchemaGenerator();
		const draft = gen.fromSamples(userSamples, { domain: "search_users" });
		const schema = new DcpSchema(draft.schema);
		const mapping = new FieldMapping(draft.mapping);
		const encoder = new DcpEncoder(schema, mapping);
		const batch = encoder.encode(userSamples);
		const dcpOutput = DcpEncoder.toString(batch);
		const jsonOutput = JSON.stringify(userSamples);
		const reduction = (1 - dcpOutput.length / jsonOutput.length) * 100;
		console.log(`JSON: ${jsonOutput.length} chars, DCP: ${dcpOutput.length} chars, reduction: ${reduction.toFixed(1)}%`);
		assert.ok(reduction > 10, `expected >10% reduction, got ${reduction.toFixed(1)}%`);
		});
		it("scales: 30+ records exceed 30% reduction", () => {
		const gen = new SchemaGenerator();
		const draft = gen.fromSamples(userSamples, { domain: "search_users" });
		const schema = new DcpSchema(draft.schema);
		const mapping = new FieldMapping(draft.mapping);
		const encoder = new DcpEncoder(schema, mapping);
		const scaled = [];
		for (let i = 0; i < 30; i++) {
		const s = JSON.parse(JSON.stringify(userSamples[i % 3]));
		s.id = `u${String(i).padStart(3, "0")}`;
		scaled.push(s);
		}
		const batch = encoder.encode(scaled);
		const dcpOutput = DcpEncoder.toString(batch);
		const jsonOutput = JSON.stringify(scaled);
		const reduction = (1 - dcpOutput.length / jsonOutput.length) * 100;
		assert.ok(reduction > 30, `30 records: expected >30% reduction, got ${reduction.toFixed(1)}%`);
		});
		});
		//# sourceMappingURL=encoder.test.js.map

-1

dist/encoder.test.js.map

-19

dist/picoclaw-hook.d.ts

		#!/usr/bin/env node
		/**
		* PicoClaw DCP Hook — out-of-process JSON-RPC over stdio.
		*
		* Two-hook pattern:
		* before_tool — injects queryType=agent for MCP tools with DCP output mode
		* after_tool — DCP-encodes JSON tool results (fallback for tools without native DCP)
		*
		* Config via PICOCLAW_DCP_TOOLS env (JSON):
		* {
		* "mcp_engram_engram_pull": { "id": "engram-recall:v1", "fields": [...] },
		* "web_fetch": "auto"
		* }
		*
		* Config via PICOCLAW_DCP_AGENT_TOOLS env (comma-separated):
		* Tools where before_tool injects queryType=agent.
		* Example: "mcp_engram_engram_pull,mcp_engram_engram_ls"
		*/
		export {};

-189

dist/picoclaw-hook.js

		#!/usr/bin/env node
		/**
		* PicoClaw DCP Hook — out-of-process JSON-RPC over stdio.
		*
		* Two-hook pattern:
		* before_tool — injects queryType=agent for MCP tools with DCP output mode
		* after_tool — DCP-encodes JSON tool results (fallback for tools without native DCP)
		*
		* Config via PICOCLAW_DCP_TOOLS env (JSON):
		* {
		* "mcp_engram_engram_pull": { "id": "engram-recall:v1", "fields": [...] },
		* "web_fetch": "auto"
		* }
		*
		* Config via PICOCLAW_DCP_AGENT_TOOLS env (comma-separated):
		* Tools where before_tool injects queryType=agent.
		* Example: "mcp_engram_engram_pull,mcp_engram_engram_ls"
		*/
		import { createInterface } from "node:readline";
		import { dcpEncode, SchemaGenerator } from "./index.js";
		// --- State ---
		let toolsConfig = {};
		let agentQueryTools = new Set();
		const autoSchemaCache = new Map();
		// --- Config loading ---
		function loadConfig() {
		const envTools = process.env.PICOCLAW_DCP_TOOLS;
		if (envTools) {
		try {
		toolsConfig = JSON.parse(envTools);
		}
		catch {
		log(`Failed to parse PICOCLAW_DCP_TOOLS`);
		}
		}
		const envAgent = process.env.PICOCLAW_DCP_AGENT_TOOLS;
		if (envAgent) {
		agentQueryTools = new Set(envAgent.split(",").map((s) => s.trim()).filter(Boolean));
		}
		}
		// --- Logging ---
		function log(msg) {
		process.stderr.write(`[dcp-hook] ${msg}\n`);
		}
		// --- JSON-RPC ---
		function sendResponse(id, result) {
		const msg = { jsonrpc: "2.0", id, result };
		process.stdout.write(JSON.stringify(msg) + "\n");
		}
		function sendError(id, code, message) {
		const msg = { jsonrpc: "2.0", id, error: { code, message } };
		process.stdout.write(JSON.stringify(msg) + "\n");
		}
		// --- DCP encoding ---
		function tryParseJSON(text) {
		try {
		const parsed = JSON.parse(text);
		if (Array.isArray(parsed)) {
		if (parsed.length > 0 && typeof parsed[0] === "object" && parsed[0] !== null) {
		return parsed;
		}
		return null;
		}
		if (typeof parsed === "object" && parsed !== null) {
		return [parsed];
		}
		return null;
		}
		catch {
		return null;
		}
		}
		function encodeToolResult(toolName, forLlm) {
		const config = toolsConfig[toolName];
		if (!config)
		return null;
		const records = tryParseJSON(forLlm);
		if (!records)
		return null;
		if (config === "auto") {
		return encodeAuto(toolName, records);
		}
		const schema = { id: config.id, fields: config.fields };
		return dcpEncode(records, schema);
		}
		function encodeAuto(toolName, records) {
		let schema = autoSchemaCache.get(toolName);
		if (!schema) {
		const gen = new SchemaGenerator();
		const draft = gen.fromSamples(records, {
		domain: toolName,
		maxDepth: 3,
		maxFields: 20,
		});
		schema = { id: draft.schema.id, fields: draft.schema.fields };
		autoSchemaCache.set(toolName, schema);
		log(`auto-schema ${toolName}: ${schema.fields.join(",")}`);
		}
		return dcpEncode(records, schema);
		}
		// --- Hook handlers ---
		function handleHello(_params) {
		return { ok: true, name: "dcp-encoder" };
		}
		function handleBeforeTool(params) {
		const payload = params;
		if (agentQueryTools.has(payload.tool)) {
		return {
		action: "modify",
		call: {
		...payload,
		arguments: { ...payload.arguments, queryType: "agent" },
		},
		};
		}
		return { action: "continue" };
		}
		function handleAfterTool(params) {
		const payload = params;
		const toolName = payload.tool;
		const result = payload.result;
		if (!result \|\| result.is_error \|\| !result.for_llm) {
		return { action: "continue" };
		}
		const encoded = encodeToolResult(toolName, result.for_llm);
		if (!encoded) {
		return { action: "continue" };
		}
		const originalLen = result.for_llm.length;
		const encodedLen = encoded.length;
		const pct = ((1 - encodedLen / originalLen) * 100).toFixed(0);
		log(`${toolName}: ${originalLen}→${encodedLen} (${pct}%)`);
		return {
		action: "modify",
		result: {
		...payload,
		result: { ...result, for_llm: encoded },
		},
		};
		}
		function handleRequest(method, params) {
		switch (method) {
		case "hook.hello":
		return handleHello((params ?? {}));
		case "hook.before_tool":
		return handleBeforeTool(params);
		case "hook.after_tool":
		return handleAfterTool(params);
		case "hook.before_llm":
		case "hook.after_llm":
		case "hook.approve_tool":
		return { action: "continue" };
		default:
		throw new Error(`method not found: ${method}`);
		}
		}
		// --- Main loop ---
		function main() {
		loadConfig();
		const dcpTools = Object.keys(toolsConfig);
		const agentTools = [...agentQueryTools];
		log(`dcp=${dcpTools.join(",") \|\| "none"} agent=${agentTools.join(",") \|\| "none"}`);
		const rl = createInterface({ input: process.stdin });
		rl.on("line", (line) => {
		if (!line.trim())
		return;
		let msg;
		try {
		msg = JSON.parse(line);
		}
		catch {
		return;
		}
		if (!msg.id)
		return;
		try {
		const result = handleRequest(msg.method ?? "", msg.params);
		sendResponse(msg.id, result);
		}
		catch (err) {
		sendError(msg.id, -32000, err.message);
		}
		});
		rl.on("close", () => {
		process.exit(0);
		});
		}
		main();
		//# sourceMappingURL=picoclaw-hook.js.map

-1

dist/picoclaw-hook.js.map

-1

dist/picoclaw-hook.test.d.ts

export {};

-200

dist/picoclaw-hook.test.js

		import { describe, it } from "node:test";
		import assert from "node:assert/strict";
		import { spawn } from "node:child_process";
		import { resolve, dirname } from "node:path";
		import { fileURLToPath } from "node:url";
		const __dirname = dirname(fileURLToPath(import.meta.url));
		const hookPath = resolve(__dirname, "../dist/picoclaw-hook.js");
		function startHook(cfg) {
		const env = { ...process.env };
		if (cfg?.tools) {
		env.PICOCLAW_DCP_TOOLS = JSON.stringify(cfg.tools);
		}
		if (cfg?.agentTools) {
		env.PICOCLAW_DCP_AGENT_TOOLS = cfg.agentTools.join(",");
		}
		return spawn("node", [hookPath], {
		stdio: ["pipe", "pipe", "pipe"],
		env,
		});
		}
		function rpc(proc, id, method, params) {
		return new Promise((resolve, reject) => {
		const timeout = setTimeout(() => reject(new Error(`timeout for ${method}`)), 5000);
		const onData = (chunk) => {
		const lines = chunk.toString().split("\n").filter(Boolean);
		for (const line of lines) {
		try {
		const msg = JSON.parse(line);
		if (msg.id === id) {
		clearTimeout(timeout);
		proc.stdout.off("data", onData);
		if (msg.error)
		reject(new Error(msg.error.message));
		else
		resolve(msg.result);
		}
		}
		catch { /* ignore parse errors */ }
		}
		};
		proc.stdout.on("data", onData);
		proc.stdin.write(JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n");
		});
		}
		describe("picoclaw-hook", () => {
		it("responds to hook.hello", async () => {
		const proc = startHook();
		try {
		const result = await rpc(proc, 1, "hook.hello", { name: "test", version: 1, modes: ["tool"] });
		assert.deepStrictEqual(result, { ok: true, name: "dcp-encoder" });
		}
		finally {
		proc.kill();
		}
		});
		it("passes through unconfigured tools", async () => {
		const proc = startHook({ tools: {} });
		try {
		await rpc(proc, 1, "hook.hello", {});
		const result = await rpc(proc, 2, "hook.after_tool", {
		meta: { session_key: "s1" },
		tool: "unknown_tool",
		result: { for_llm: '{"foo":"bar"}', silent: false, is_error: false },
		});
		assert.deepStrictEqual(result, { action: "continue" });
		}
		finally {
		proc.kill();
		}
		});
		it("DCP-encodes configured tool results", async () => {
		const proc = startHook({
		tools: {
		mcp_engram_pull: {
		id: "engram-recall:v1",
		fields: ["id", "relevance", "summary", "tags"],
		},
		},
		});
		try {
		await rpc(proc, 1, "hook.hello", {});
		const toolOutput = JSON.stringify([
		{ id: "abc-123", relevance: 0.95, summary: "Test node", tags: ["howto", "gateway"], hitCount: 3, weight: 1.2 },
		{ id: "def-456", relevance: 0.72, summary: "Another node", tags: ["why"], hitCount: 1, weight: 0.5 },
		]);
		const result = (await rpc(proc, 2, "hook.after_tool", {
		meta: { session_key: "s1" },
		tool: "mcp_engram_pull",
		result: { for_llm: toolOutput, silent: false, is_error: false },
		}));
		assert.equal(result.action, "modify");
		const encoded = result.result.result.for_llm;
		assert.ok(encoded.includes("$S"));
		assert.ok(encoded.includes("engram-recall:v1"));
		assert.ok(encoded.length < toolOutput.length);
		const lines = encoded.split("\n");
		assert.equal(lines.length, 3, "header + 2 data rows");
		const header = JSON.parse(lines[0]);
		assert.equal(header[0], "$S");
		assert.equal(header[1], "engram-recall:v1");
		}
		finally {
		proc.kill();
		}
		});
		it("passes through error results without encoding", async () => {
		const proc = startHook({
		tools: { mcp_engram_pull: { id: "test:v1", fields: ["id", "summary"] } },
		});
		try {
		await rpc(proc, 1, "hook.hello", {});
		const result = await rpc(proc, 2, "hook.after_tool", {
		meta: {},
		tool: "mcp_engram_pull",
		result: { for_llm: "Error: connection refused", is_error: true },
		});
		assert.deepStrictEqual(result, { action: "continue" });
		}
		finally {
		proc.kill();
		}
		});
		it("passes through non-JSON tool output", async () => {
		const proc = startHook({
		tools: { mcp_engram_pull: { id: "test:v1", fields: ["id", "summary"] } },
		});
		try {
		await rpc(proc, 1, "hook.hello", {});
		const result = await rpc(proc, 2, "hook.after_tool", {
		meta: {},
		tool: "mcp_engram_pull",
		result: { for_llm: "This is just plain text output", is_error: false },
		});
		assert.deepStrictEqual(result, { action: "continue" });
		}
		finally {
		proc.kill();
		}
		});
		it("auto-generates schema for 'auto' tools", async () => {
		const proc = startHook({ tools: { web_search: "auto" } });
		try {
		await rpc(proc, 1, "hook.hello", {});
		const toolOutput = JSON.stringify([
		{ title: "Result 1", url: "https://example.com/1", snippet: "First result snippet", score: 0.9 },
		{ title: "Result 2", url: "https://example.com/2", snippet: "Second result snippet", score: 0.7 },
		]);
		const result = (await rpc(proc, 2, "hook.after_tool", {
		meta: {},
		tool: "web_search",
		result: { for_llm: toolOutput, is_error: false },
		}));
		assert.equal(result.action, "modify");
		const encoded = result.result.result.for_llm;
		assert.ok(encoded.includes("$S"));
		assert.ok(encoded.length < toolOutput.length);
		}
		finally {
		proc.kill();
		}
		});
		it("before_tool injects queryType=agent for configured tools", async () => {
		const proc = startHook({
		agentTools: ["mcp_engram_engram_pull", "mcp_engram_engram_ls"],
		});
		try {
		await rpc(proc, 1, "hook.hello", {});
		const result = (await rpc(proc, 2, "hook.before_tool", {
		meta: { session_key: "s1" },
		tool: "mcp_engram_engram_pull",
		arguments: { query: "DCP", limit: 10 },
		}));
		assert.equal(result.action, "modify");
		assert.equal(result.call?.arguments?.queryType, "agent");
		assert.equal(result.call?.arguments?.query, "DCP");
		assert.equal(result.call?.arguments?.limit, 10);
		}
		finally {
		proc.kill();
		}
		});
		it("before_tool passes through non-agent tools", async () => {
		const proc = startHook({
		agentTools: ["mcp_engram_engram_pull"],
		});
		try {
		await rpc(proc, 1, "hook.hello", {});
		const result = await rpc(proc, 2, "hook.before_tool", {
		meta: {},
		tool: "web_search",
		arguments: { query: "test" },
		});
		assert.deepStrictEqual(result, { action: "continue" });
		}
		finally {
		proc.kill();
		}
		});
		});
		//# sourceMappingURL=picoclaw-hook.test.js.map

-1

dist/picoclaw-hook.test.js.map

-548

docs/picoclaw-integration.md

		# DCP Integration with PicoClaw

		Reduce LLM token consumption by 40-60% on structured tool output — without modifying PicoClaw's core.

		PicoClaw's [hook system](https://github.com/sipeed/picoclaw) provides `after_tool` interception points. dcp-wrap runs as an out-of-process hook that intercepts JSON tool results and converts them to DCP positional arrays before they reach the LLM.

		```
		Tool execution → JSON result
		→ after_tool hook (dcp-wrap, Node.js)
		→ DCP encode: {"id":"abc","score":0.9,"tags":["fix"]} → ["abc",0.9,"fix"]
		→ LLM receives compact DCP instead of verbose JSON
		```

		## Prerequisites

		- PicoClaw v0.2.4+
		- Node.js 18+ (installed in PicoClaw's environment)
		- dcp-wrap (`npm install dcp-wrap`)

		## Quick Start

		### 1. Install dcp-wrap

		```bash
		npm install dcp-wrap
		```

		### 2. Add the hook to PicoClaw config

		In your `config.json`, add the `hooks` section. The hook runs as an external process communicating via JSON-RPC over stdio.

		```json
		{
		"version": 1,
		"hooks": {
		"enabled": true,
		"processes": {
		"dcp_encoder": {
		"enabled": true,
		"priority": 50,
		"transport": "stdio",
		"command": ["node", "./node_modules/dcp-wrap/dist/picoclaw-hook.js"],
		"intercept": ["after_tool"],
		"env": {
		"PICOCLAW_DCP_TOOLS": "{\"my_api_tool\":{\"id\":\"api-response:v1\",\"fields\":[\"endpoint\",\"method\",\"status\",\"latency_ms\"]}}"
		}
		}
		}
		}
		}
		```

		### 3. Restart PicoClaw

		The hook starts automatically on the first user message (hooks are lazily initialized).

		Check the logs for:
		```
		Process hook stderr \| hook=dcp_encoder \| stderr="[dcp-hook] Started. Tools configured: my_api_tool"
		```

		## Configuring Tools

		The `PICOCLAW_DCP_TOOLS` environment variable maps tool names to DCP schemas. Two modes:

		### Explicit schema (recommended for known tools)

		Define the schema ID and field list. Fields are extracted by name from the JSON output.

		```json
		{
		"mcp_engram_pull": {
		"id": "engram-recall:v1",
		"fields": ["id", "relevance", "summary", "tags", "hitCount", "weight", "status"]
		}
		}
		```

		### Auto schema (for unknown/varying tools)

		Set `"auto"` and dcp-wrap will infer the schema from the first batch of results.

		```json
		{
		"web_fetch": "auto"
		}
		```

		Auto-generated schemas are cached for the hook process lifetime. Good for exploration; switch to explicit schemas once you know the output shape.

		### Mixed configuration

		```json
		{
		"mcp_engram_pull": {
		"id": "engram-recall:v1",
		"fields": ["id", "relevance", "summary", "tags", "hitCount", "weight", "status"]
		},
		"mcp_engram_ls": {
		"id": "engram-scan:v1",
		"fields": ["id", "summary", "tags", "hitCount", "weight", "status"]
		},
		"web_fetch": "auto"
		}
		```

		Unlisted tools pass through unchanged.

		## What gets encoded

		The hook intercepts `result.for_llm` — the string that PicoClaw sends to the LLM as tool output. Encoding happens only when:

		1. The tool is listed in `PICOCLAW_DCP_TOOLS`
		2. The result is not an error (`is_error: false`)
		3. The result parses as JSON (object or array of objects)

		If any condition fails, the result passes through unchanged. The hook never breaks tool output.

		## Where DCP helps most

		DCP reduces tokens on structured, multi-record data:

		\| Tool output type \| DCP effect \| Why \|
		\|---\|---\|---\|
		\| Array of JSON objects (API results, search results, database rows) \| 40-60% reduction \| Repeated keys eliminated, positional encoding \|
		\| Single JSON object with large text field \| ~0% reduction \| Text dominates, schema overhead > savings \|
		\| Plain text \| Passthrough \| Not JSON, nothing to encode \|

		Best candidates: MCP tool results, API responses, database queries, structured search results.

		## Docker Setup

		PicoClaw's official Docker image (`sipeed/picoclaw:latest`) is Alpine-based with no Node.js. Add it:

		```dockerfile
		FROM docker.io/sipeed/picoclaw:latest

		USER root
		RUN apk add --no-cache nodejs npm

		# Option A: npm install (when dcp-wrap is published)
		# WORKDIR /opt/dcp-hook
		# RUN npm install dcp-wrap

		# Option B: copy built dist (local development)
		WORKDIR /opt/dcp-hook/node_modules/dcp-wrap
		COPY dcp-wrap-dist/ ./dist/
		COPY dcp-wrap-package.json ./package.json

		WORKDIR /root
		ENTRYPOINT ["picoclaw"]
		CMD ["gateway"]
		```

		Update the hook command path in config:
		```json
		"command": ["node", "/opt/dcp-hook/node_modules/dcp-wrap/dist/picoclaw-hook.js"]
		```

		Mount config via volume:
		```yaml
		services:
		picoclaw-gateway:
		build: .
		volumes:
		- ./data:/root/.picoclaw
		extra_hosts:
		- "host.docker.internal:host-gateway"
		ports:
		- "127.0.0.1:18800:18790"
		```

		## Gotchas

		### Config must have `"version": 1`

		PicoClaw's config migration runs when `version` is missing (treated as v0). The v0-to-v1 migration re-serializes the Go struct with `omitempty`, which silently drops `hooks.processes` if it wasn't recognized during migration.

		Always start your config with:
		```json
		{
		"version": 1,
		...
		}
		```

		### Hooks initialize lazily

		Don't expect hook logs at gateway startup. The `hook.hello` handshake happens on the first user message that triggers a turn. If you only see startup logs with no hook activity, send a message first.

		### `intercept: ["after_tool"]` also sends `before_tool`

		PicoClaw maps both `before_tool` and `after_tool` to a single `InterceptTool` flag. The hook receives both RPCs. dcp-wrap handles this correctly (returns `{"action": "continue"}` for `before_tool`).

		### Plain text tool output

		Some built-in tools (e.g., DuckDuckGo `web_search`) return plain text, not JSON. The hook passes these through unchanged. This is correct behavior — DCP encodes structure, not prose.

		## How it works internally

		The hook process communicates with PicoClaw via [JSON-RPC over stdio](https://github.com/sipeed/picoclaw):

		```
		PicoClaw dcp-wrap hook (Node.js)
		│ │
		├──hook.hello──────────────────────▶│
		│◀─────────────{ok:true}────────────┤
		│ │
		│ (user sends message, LLM calls tool)
		│ │
		├──hook.before_tool────────────────▶│
		│◀─────────{action:"continue"}──────┤
		│ │
		│ (tool executes, produces result) │
		│ │
		├──hook.after_tool─────────────────▶│
		│ {tool:"mcp_query", │
		│ result:{for_llm:"[{...},...]"}} │
		│ │
		│ (hook encodes for_llm via DCP) │
		│ │
		│◀──{action:"modify",───────────────┤
		│ result:{for_llm:"[$S,...]\n..."}}│
		│ │
		│ (LLM receives DCP-encoded output) │
		```

		### RPC payloads

		after_tool request (PicoClaw sends):
		```json
		{
		"jsonrpc": "2.0",
		"id": 7,
		"method": "hook.after_tool",
		"params": {
		"meta": {"session_key": "..."},
		"tool": "mcp_engram_pull",
		"arguments": {"query": "docker port conflict"},
		"result": {
		"for_llm": "[{\"id\":\"abc\",\"relevance\":0.95,...}]",
		"is_error": false
		},
		"duration": 234000000
		}
		}
		```

		after_tool response (hook returns, when encoding):
		```json
		{
		"jsonrpc": "2.0",
		"id": 7,
		"result": {
		"action": "modify",
		"result": {
		"meta": {"session_key": "..."},
		"tool": "mcp_engram_pull",
		"result": {
		"for_llm": "[\"$S\",\"engram-recall:v1\",\"id\",\"relevance\",\"summary\",\"tags\"]\n[\"abc\",0.95,\"docker port fix\",\"docker,gotcha\"]"
		}
		}
		}
		}
		```

		## Why PicoClaw, not OpenClaw

		We evaluated both frameworks for DCP integration. PicoClaw is the clear choice.

		### OpenClaw: skill/prompt-level DCP constraints don't work

		OpenClaw's architecture makes hook-level interception impractical:

		- Built-in prompts override skill prompts. DCP formatting instructions added via skills or custom prompts are silently overridden by OpenClaw's internal system prompt. The LLM never sees the DCP constraint.
		- No output hooks. OpenClaw has no `after_tool` equivalent. There is no way to intercept tool results before they reach the LLM. This is tracked as [OpenClaw #12914](https://github.com/openclaw/openclaw) but unimplemented as of March 2026.
		- Plugin architecture is input-only. OpenClaw plugins can modify the system prompt but cannot intercept or transform tool output.

		Bottom line: without output hooks, DCP encoding at the tool result boundary is impossible in OpenClaw.

		### PicoClaw: 4 modifiable hooks = complete DCP pipeline

		PicoClaw's hook system was designed for exactly this kind of interception:

		\| Hook \| DCP role \| Status \|
		\|---\|---\|---\|
		\| `after_tool` \| Encode tool JSON → DCP positional arrays \| Implemented \|
		\| `before_llm` \| Inject output controller ("respond as DCP") \| Planned \|
		\| `after_llm` \| Cap non-conforming output + decode for messaging \| Planned \|
		\| `before_tool` \| (Optional) Tool argument optimization \| Not needed \|

		Key advantages:
		- Out-of-process hooks via JSON-RPC over stdio — any language works, no Go port needed
		- Modifiable responses — hooks can rewrite tool results, not just observe them
		- Per-tool selectivity — DCP encode only configured tools, everything else passes through
		- Edge device focus — PicoClaw targets Raspberry Pi and resource-constrained hardware where token cost is a real constraint, not theoretical

		### Practical comparison

		\| \| OpenClaw \| PicoClaw \|
		\|---\|---\|---\|
		\| Hook system \| No output hooks \| 4 modifiable hooks \|
		\| Tool result interception \| Impossible \| `after_tool` with modify \|
		\| DCP encoding \| Not feasible \| Working (dcp-wrap hook) \|
		\| LLM output control \| Prompt-level only (overridden) \| `before_llm` injection \|
		\| External process hooks \| Not supported \| JSON-RPC over stdio \|
		\| Token cost sensitivity \| Cloud-focused \| Edge-device-focused \|

		## Rate limits and tool count

		PicoClaw sends all tool definitions to the LLM on every turn. With MCP servers, tool count can grow quickly:

		\| Configuration \| Tool count \| ~Input tokens per turn \|
		\|---\|---\|---\|
		\| Default (built-in only) \| 14 \| ~8K \|
		\| + 1 MCP server (6 tools) \| 20 \| ~15K \|
		\| + skills, multiple MCP servers \| 30+ \| ~25K+ \|

		On Anthropic's free/low-tier plans (50K input tokens/min), a single multi-iteration turn with 20+ tools can hit rate limits. Mitigations:

		1. Disable unused tools — Set `"enabled": false` for tools you don't need (exec, read_file, write_file, spawn, subagent, skills)
		2. Clear session history — Delete `data/sessions/` to reset accumulated context
		3. Limit iterations — Set `max_tool_iterations` lower (e.g., 5)
		4. Use a higher-tier API plan — More headroom for agentic loops

		This is actually where `before_llm` DCP encoding of ToolDefinition[] would have the highest impact — compressing 20+ tool schemas that ship on every single turn.

		## Real-world results: engram MCP integration

		### The problem with naive after_tool encoding

		The initial approach — intercept JSON tool results in `after_tool` and DCP-encode them — hit a fundamental issue: most tool output is not JSON.

		\| Tool \| Output format \| DCP encodable? \|
		\|---\|---\|---\|
		\| web_search (DuckDuckGo) \| Plain text (`"Results for: ..."`) \| No \|
		\| web_fetch \| Single JSON object, `text` field dominates \| ~0% reduction \|
		\| read_file, exec, list_dir \| Plain text \| No \|
		\| cron (list) \| Plain text (`"Scheduled jobs:\n- ..."`) \| No \|
		\| MCP tools (engram_pull) \| Depends on output mode \| Yes, with the right approach \|

		Even engram's MCP server returns human-readable text by default:
		```
		Found 10 results for "DCP" (cross-project):
		[1] DCP formatter placement: OUT-side...
		hits=2 weight=-2.58 status=recent relevance=0.345
		tags: why, dcp, formatter, architecture
		id: a7b5dce9-...
		```

		This is 5649 chars for 10 results. Not JSON, so the after_tool hook passes it through unchanged.

		### The solution: before_tool parameter injection

		engram's MCP server already supports a `queryType` parameter:
		- `queryType: "human"` (default) — verbose natural language
		- `queryType: "agent"` — DCP positional arrays with `$S` header

		The problem: PicoClaw's LLM doesn't know to pass `queryType: "agent"`. It uses whatever parameters it decides on.

		The fix: use `before_tool` to inject `queryType: "agent"` before the MCP call executes.

		```typescript
		// In picoclaw-hook.ts
		const AGENT_QUERY_TOOLS = new Set(["mcp_engram_engram_pull", "mcp_engram_engram_ls"]);

		function handleBeforeTool(params: unknown): unknown {
		const payload = params as ToolCallPayload;
		if (AGENT_QUERY_TOOLS.has(payload.tool)) {
		return {
		action: "modify",
		call: {
		...payload,
		arguments: { ...payload.arguments, queryType: "agent" },
		},
		};
		}
		return { action: "continue" };
		}
		```

		This is transparent to the LLM — it calls `engram_pull` normally, the hook injects the parameter, engram returns DCP, and the LLM reads compact positional arrays.

		### Measured results

		```
		before_tool: injecting queryType=agent for mcp_engram_engram_pull
		Tool call: mcp_engram_engram_pull({"crossProject":true,"limit":10,"query":"DCP","queryType":"agent"})
		Tool execution completed \| result_length=1697 \| tool=mcp_engram_engram_pull
		```

		\| \| Human format \| DCP format \| Reduction \|
		\|---\|---\|---\|---\|
		\| engram_pull (10 results) \| 5649 chars \| 1697 chars \| 70% \|
		\| LLM iterations to answer \| 3 \| 2 \| -33% \|

		The LLM correctly interprets the DCP `$S` header and positional rows, extracting the same information from 70% fewer tokens.

		### Key insight: two-hook pattern

		The effective pattern is not `after_tool` alone, but `before_tool` + `after_tool` working together:

		1. `before_tool`: Inject parameters that tell the MCP server to return compact format
		2. `after_tool`: Available as fallback for tools that don't have a compact mode (auto-encode JSON via SchemaGenerator)

		This avoids the fundamental problem of trying to parse and re-encode text that was never JSON in the first place.

		### Difficulties encountered during integration

		engram MCP dist was stale. The `dcp-format.ts` source existed but `dist/dcp-format.js` did not — the MCP server had never been rebuilt after adding DCP output support. The hook injected `queryType: "agent"` correctly, the MCP server received it, but the import of `formatRecallDcp` failed silently and fell through to the human format codepath. Always rebuild MCP servers before copying dist into Docker.

		MCP tool naming convention. PicoClaw prefixes MCP tools with `mcp_{serverName}_{toolName}`. For server `engram` and tool `engram_pull`, the full name is `mcp_engram_engram_pull` — not `mcp_engram_pull`. This affects both the DCP_TOOLS config and the AGENT_QUERY_TOOLS set.

		Environment variable naming. engram's MCP server uses `GATEWAY_URL`, not `ENGRAM_GATEWAY_URL`. The first attempt used the wrong name, causing `Cannot reach http://localhost:3100` errors inside the container (the default fallback).

		Rate limits with many tools. PicoClaw sends all tool definitions on every LLM turn. With 6 MCP tools + 14 built-in tools = 20 tools, a multi-iteration turn can exceed Anthropic's 50K input tokens/min limit. Disabling unused tools (exec, file I/O, skills, spawn) reduced the count from 21 to 11 and resolved the issue.

		## For MCP server authors: why your server should speak DCP

		This integration proved one thing clearly: DCP cannot be bolted on from outside. A hook sitting between the tool and the LLM can only work with what the tool gives it. If the tool returns plain text, there is nothing to compress.

		### The plain text problem

		Most tools — web_search, exec, read_file, cron, list_dir — return plain text for a good reason: maximum compatibility. Any LLM can read text. No schema knowledge required.

		But this "compatibility" has a cost. When an MCP tool returns 10 structured records as formatted text:

		```
		[1] DCP formatter placement: OUT-side...
		hits=2 weight=-2.58 status=recent relevance=0.345
		tags: why, dcp, formatter, architecture
		id: a7b5dce9-...
		```

		Every field label (`hits=`, `weight=`, `status=`, `tags:`, `id:`) is repeated per record. For 10 records, that's 10x the overhead. The LLM reads all of it, pays for all of it, and extracts the same information that a positional array conveys in a fraction of the tokens.

		### What MCP servers should do

		Add a `queryType` parameter (or equivalent) to your tool schema:

		```typescript
		server.tool("my_tool", {
		query: z.string(),
		queryType: z.enum(["human", "agent"]).optional()
		.describe("'agent' returns DCP compact format. Default: 'human'."),
		}, async ({ query, queryType }) => {
		const results = await fetchResults(query);

		if (queryType === "agent") {
		// DCP positional arrays — 70% fewer tokens
		return { content: [{ type: "text", text: dcpEncode(results, schema) }] };
		}

		// Human-readable text — default, compatible with everything
		return { content: [{ type: "text", text: formatHuman(results) }] };
		});
		```

		This is the contract. The MCP tool declares it can speak DCP. The consumer (LLM, hook, agent framework) decides whether to ask for it.

		### Why this matters

		Without this contract:
		- Hooks can only observe text, not compress it
		- Every agent framework must implement its own parsing for every tool's text format
		- Token costs scale linearly with verbosity

		With this contract:
		- A `before_tool` hook injects `queryType: "agent"` once
		- The MCP server returns DCP natively — no parsing, no re-encoding
		- 70% token reduction, measured and proven
		- The human format remains the default — nothing breaks for consumers that don't know DCP

		The MCP protocol already provides the mechanism: typed parameters via tool schemas. DCP doesn't require a new protocol — it requires MCP servers to offer a compact output mode and consumers to ask for it.

		### The pattern for any MCP server

		1. Keep human format as default. Backward compatible. Text works everywhere.
		2. Add `queryType: "agent"` parameter. Declare the compact mode exists.
		3. Return DCP when asked. Use [dcp-wrap](https://github.com/hiatamaworkshop/dcp-wrap) or format positional arrays directly.
		4. Let hooks handle the switching. Agent frameworks inject the parameter automatically — the LLM never needs to learn about it.

		## Scheduled tasks and DCP: where it applies

		PicoClaw has two schedulers: cron (user-defined jobs) and heartbeat (periodic `HEARTBEAT.md` check). Whether DCP applies depends on the execution path.

		### Execution paths

		\| Scheduler \| Mode \| Flow \| DCP applies? \|
		\|---\|---\|---\|---\|
		\| Cron \| `deliver=false` (default) \| Message → LLM turn → LLM calls tools → response \| Yes — tools go through hooks \|
		\| Cron \| `deliver=true` \| Message → direct to Telegram \| No — LLM not involved \|
		\| Cron \| `command` set \| Shell exec → output to Telegram \| No — LLM not involved \|
		\| Heartbeat \| — \| HEARTBEAT.md → LLM turn → LLM calls tools → response \| Yes — same as cron deliver=false \|

		The key: cron `deliver=false` and heartbeat both start an LLM turn via `ProcessDirectWithChannel`. The LLM decides which tools to call. Those tool calls go through `before_tool` (parameter injection) and `after_tool` (encoding fallback) — the same DCP pipeline as interactive messages.

		### What this means in practice

		A heartbeat task like "Check engram for new knowledge about DCP" triggers:

		```
		Heartbeat tick (every 30 min)
		→ LLM reads HEARTBEAT.md task list
		→ LLM calls mcp_engram_engram_pull(query="DCP")
		→ before_tool injects queryType=agent ← DCP kicks in here
		→ engram returns 1697 chars (not 5649) ← 70% saved
		→ LLM summarizes and sends to Telegram
		```

		Every 30 minutes, 70% fewer tokens per engram query. Over a day, that compounds.

		### Input-side DCP: the before_llm frontier

		The current implementation handles tool output (after_tool) and tool parameters (before_tool). But the largest token cost is on the input side — what gets sent to the LLM on every single turn:

		\| Input component \| Sent every turn \| Approximate tokens \| DCP potential \|
		\|---\|---\|---\|---\|
		\| Tool definitions (12+ tools) \| Yes \| ~4K-8K \| High — repeated structured schemas \|
		\| Conversation history \| Yes \| Grows over time \| High — message[] with repeated structure \|
		\| System prompt \| Yes (cached by LLM) \| ~2K \| Low — already prefix-cached \|
		\| User message \| Yes \| Small \| None — natural language \|

		The `before_llm` hook can intercept the full `LLMHookRequest` including `messages[]` and `tools[]`. Compressing tool definitions from verbose JSON Schema to DCP positional format could save 50%+ on every turn — but this requires the LLM to understand DCP tool schemas, which is unverified territory.

		This is the next frontier. The tool output side is solved. The input side is where the remaining cost lives.

		### Guidance for task authors

		When writing `HEARTBEAT.md` tasks or cron job messages, structure them so the LLM calls MCP tools (which benefit from DCP) rather than built-in text tools:

		```markdown
		## Good — triggers MCP tool with DCP benefit
		- Check engram for recent knowledge about deployment issues
		- Search engram for error patterns from this week

		## Less effective — triggers text-output tools
		- Run `df -h` and report disk usage
		- Fetch https://status.example.com and summarize
		```

		The former triggers `engram_pull` → DCP encoding → 70% savings. The latter triggers `exec` or `web_fetch` → plain text → no DCP benefit.

		## Next steps

		- [DCP Specification](https://dcp-docs.pages.dev/dcp/specification) — full protocol design
		- [dcp-wrap README](../README.md) — CLI and programmatic API
		- [Schema-Driven Encoder](https://dcp-docs.pages.dev/dcp/schema-driven-encoder) — how encoding works

dcp-wrap - npm Package Compare versions

New alerts

Fixed alerts

Improved metrics

Worsened metrics