New:Socket for Asana Is Now Available.Learn more
Get Started

dsh-vision-recognizer

Package Overview
Dependencies
Maintainers
1
Versions
5
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

dsh-vision-recognizer - npm Package Compare versions

Comparing version
0.1.3
to
0.2.0
+7
-0
CHANGELOG.md
# Changelog
## 0.2.0 — 2026-08-21
- Add capability-aware image routing: models that declare native image input receive original image blocks directly, while text-only or unknown-capability models use the configured vision transcription fallback.
- Delegate through the public rc8 LLM runtime APIs with target-provider rebinding, preserving wrapper metadata and route-sensitive provider configuration.
- Probe local Ollama lazily only when transcription is required; native multimodal calls no longer depend on vision configuration.
- Add integration coverage for both native pass-through and text-only transcription paths.
## 0.1.3 — 2026-08-18
- Migrate the plugin to the canonical DSH plugin standard, including contract checks, lifecycle integration coverage, artifact verification, isolated install smoke testing, and Node 22/24 CI.
+4
-4

@@ -24,3 +24,3 @@ window.__ModuleLoader__.load({ id: "dsh-vision-recognizer", factory: (require) => {

title: '识图 API 配置',
intro: '选择视觉模型供应商并配置 API Key。附加图片会被自动转译为文字,对话仍由 DeepSeek 作答。',
intro: '当前对话模型原生支持图片时直接传图;否则使用下列视觉模型转译为文字。',
providerLabel: '供应商',

@@ -43,3 +43,3 @@ providerHint: '选择图片转译使用的视觉模型供应商。可在下方覆盖模型与接口地址。',

markerLabel: '转译标记',
markerHint: '转译文本的前缀标记,DeepSeek 靠它区分「这是图片转译的内容」。默认 [图片转译]。',
markerHint: '仅纯文本模型回退时使用:给转译文本添加前缀,默认 [图片转译]。',
autoOllamaLabel: '自动探测本地 Ollama',

@@ -62,3 +62,3 @@ autoOllamaHint: '启动时探测本机 http://localhost:11434 的 Ollama,检测到则作为兜底——图片不出本机、无需 Key。',

title: 'Vision API configuration',
intro: 'Pick a vision provider and configure its API key. Attached images are transcribed to text; DeepSeek still answers the conversation.',
intro: 'Native multimodal conversation models receive images directly; otherwise the vision model below transcribes them to text.',
providerLabel: 'Provider',

@@ -81,3 +81,3 @@ providerHint: 'Vision provider used to transcribe attached images. Model and endpoint can be overridden below.',

markerLabel: 'Transcription marker',
markerHint: 'Text prefix prepended to each transcription, so DeepSeek can tell image transcriptions apart. Default [图片转译].',
markerHint: 'Used only for text-only fallback: prefix prepended to each transcription. Default [图片转译].',
autoOllamaLabel: 'Auto-detect local Ollama',

@@ -84,0 +84,0 @@ autoOllamaHint: 'Probe local Ollama at http://localhost:11434 at startup and use it as a fallback — images never leave your machine and no key is needed.',

/**
* dsh-vision-recognizer: keep DeepSeek as the conversation brain and get image
* understanding anyway, with a multi-provider vision transcription layer.
* dsh-vision-recognizer: adaptive image routing for a wrapped conversation
* model, with a multi-provider vision transcription fallback.
*
* Registers a NEW provider route (`vision-recognizer` by default) that wraps
* the real DeepSeek adapter:
* - `resolveModel` declares `inputModalities` including "image", so the Web
* attachment preflight and the read_image capability gate admit images;
* - `stream` transcribes every image block in the request to text through a
* configurable vision model (15+ providers — OpenAI-compatible and
* Anthropic Messages) and then delegates the text-only conversation to the
* real DeepSeek adapter.
* the configured target provider:
* - `resolveModel` declares image input so attachment preflight admits images;
* - `stream` resolves the exact target model's declared modalities;
* - native multimodal targets receive original image blocks unchanged;
* - text-only or unknown-capability targets receive text produced by the
* configured vision model (15+ providers, OpenAI-compatible or Anthropic).
*

@@ -34,3 +33,3 @@ * Configuration lives in Settings → Plugins → 识图 and is persisted to

const DEFAULT_PROVIDER = 'vision-recognizer';
/** Default inner route whose adapter is wrapped (the text-only DeepSeek line). */
/** Default target route wrapped by the adaptive provider. */
const DEFAULT_INNER_PROVIDER = 'deepseek-official';

@@ -242,7 +241,12 @@ /** Local Ollama endpoint probed when autoLocalOllama is enabled. */

const inner = ctx.llm.registration(state.config.innerProvider ?? DEFAULT_INNER_PROVIDER)?.adapter;
if (inner === undefined) {
ctx.logger.error(`dsh-vision-recognizer: no adapter registered for "${state.config.innerProvider ?? DEFAULT_INNER_PROVIDER}"; proxy route disabled`);
const providerId = state.config.providerId ?? DEFAULT_PROVIDER;
const innerProvider = state.config.innerProvider ?? DEFAULT_INNER_PROVIDER;
if (innerProvider === providerId) {
ctx.logger.error(`dsh-vision-recognizer: wrapper route "${providerId}" cannot target itself; proxy route disabled`);
return;
}
if (!ctx.llm.listProviders().some((provider) => provider.id === innerProvider)) {
ctx.logger.error(`dsh-vision-recognizer: no adapter registered for "${innerProvider}"; proxy route disabled`);
return;
}

@@ -262,37 +266,63 @@ // Transcription re-resolves from live state on every stream so Settings

// Local Ollama probe: non-blocking, memoized; prepended to the chain when
// a local Ollama is running (zero-config local path).
const ollamaProbe = (state.config.autoLocalOllama ?? true)
? core.detectLocalOllama(fetch, OLLAMA_BASE_URL, OLLAMA_PROBE_TIMEOUT_MS)
: Promise.resolve(null);
// Probe local Ollama only when a text-only target actually needs the
// transcription fallback. Native multimodal calls never touch this path.
let ollamaProbe;
const resolveLocalOllama = () => {
if (!(state.config.autoLocalOllama ?? true)) return Promise.resolve(null);
if (ollamaProbe === undefined) {
ollamaProbe = core.detectLocalOllama(fetch, OLLAMA_BASE_URL, OLLAMA_PROBE_TIMEOUT_MS);
ollamaProbe.then((local) => {
if (local !== null) {
ctx.logger.info(`dsh-vision-recognizer: local Ollama detected at ${local.baseURL} (model ${local.model}) — prepended to the fallback chain`);
}
}).catch(() => {});
}
return ollamaProbe;
};
const primary = buildAttempts()[0];
ctx.logger.info(`dsh-vision-recognizer: route "${state.config.providerId ?? DEFAULT_PROVIDER}" wraps "${state.config.innerProvider ?? DEFAULT_INNER_PROVIDER}" · provider "${primary.provider}" · model "${primary.model}" @ ${primary.baseURL || '(unset — set in Settings → Plugins)'} · timeout ${primary.timeoutMs}ms · maxTokens ${primary.maxTokens} · autoLocalOllama ${state.config.autoLocalOllama ?? true}`);
ctx.logger.info('dsh-vision-recognizer: PRIVACY NOTICE — image bytes are sent over HTTPS to the configured vision endpoint for transcription. Images leave your machine unless the endpoint is local (e.g. Ollama).');
ctx.logger.info(`dsh-vision-recognizer: adaptive route "${providerId}" wraps "${innerProvider}" · text-only fallback "${primary.model}" @ ${primary.baseURL || '(unset — set in Settings → Plugins)'} · timeout ${primary.timeoutMs}ms · maxTokens ${primary.maxTokens} · autoLocalOllama ${state.config.autoLocalOllama ?? true}`);
ctx.logger.info('dsh-vision-recognizer: PRIVACY NOTICE — native multimodal models receive images directly; only the text-only fallback sends image bytes to the configured vision endpoint.');
ollamaProbe.then((local) => {
if (local !== null) {
ctx.logger.info(`dsh-vision-recognizer: local Ollama detected at ${local.baseURL} (model ${local.model}) — prepended to the fallback chain`);
}
}).catch(() => {});
const proxy = {
providerInfo: (provider) => ({ id: provider, name: 'DeepSeek + 识图' }),
providerRetryPolicy: (provider) => inner.providerRetryPolicy?.(provider) ?? undefined,
listModels: (provider) => inner.listModels(provider),
providerInfo: (provider) => ({ id: provider, name: 'DeepSeek + 智能识图' }),
providerRetryPolicy: () => ctx.llm.providerRetryPolicy(innerProvider),
listModels: async (provider) => {
const models = await ctx.llm.listModels(innerProvider);
return models.map((model) => ({ ...model, provider }));
},
resolveModel: async (provider, model, signal) => {
const info = await inner.resolveModel(provider, model, signal);
// The key trick: claim image input so attachment preflight and the
// read_image gate admit images; transcription happens in stream().
return { ...info, inputModalities: ['text', 'image'] };
const info = await ctx.llm.resolveModelInfo(innerProvider, model, signal);
// The wrapper always admits images: declared native multimodal
// targets receive them unchanged, while every other target gets the
// configured vision transcription fallback.
return { ...info, provider, inputModalities: ['text', 'image'] };
},
stream: async function* (options) {
const local = await ollamaProbe;
const baseAttempts = buildAttempts();
const attempts = local === null
? baseAttempts
: [{ provider: 'ollama', baseURL: local.baseURL, model: local.model, protocol: 'openai', apiKeyEnv: '', apiKey: '', maxTokens: baseAttempts[0].maxTokens, timeoutMs: Math.min(baseAttempts[0].timeoutMs, core.ANONYMOUS_TIMEOUT_CAP_MS) }, ...baseAttempts];
const marker = state.config.marker ?? '[图片转译]';
const messages = await core.transcribeMessages(ctx, attempts, marker, options.messages, options.signal, cache, cooldowns);
yield* inner.stream({ ...options, messages });
const hasImages = options.messages.some((message) => core.hasImage(message.content));
let messages = options.messages;
if (hasImages) {
const modelInfo = await ctx.llm.resolveModelInfo(innerProvider, options.model, options.signal);
if (modelInfo.inputModalities?.includes('image') !== true) {
const local = await resolveLocalOllama();
const baseAttempts = buildAttempts();
const attempts = local === null
? baseAttempts
: [{ provider: 'ollama', baseURL: local.baseURL, model: local.model, protocol: 'openai', apiKeyEnv: '', apiKey: '', maxTokens: baseAttempts[0].maxTokens, timeoutMs: Math.min(baseAttempts[0].timeoutMs, core.ANONYMOUS_TIMEOUT_CAP_MS) }, ...baseAttempts];
const marker = state.config.marker ?? '[图片转译]';
messages = await core.transcribeMessages(ctx, attempts, marker, options.messages, options.signal, cache, cooldowns);
}
}
const targetConfig = {
provider: innerProvider,
model: options.model,
...(options.reasoningEffort === undefined ? {} : { reasoningEffort: options.reasoningEffort }),
...(options.temperature === undefined ? {} : { temperature: options.temperature }),
...(options.maxTokens === undefined ? {} : { maxTokens: options.maxTokens }),
...(options.stop === undefined ? {} : { stop: options.stop }),
};
const prepared = await ctx.llm.prepareCall(targetConfig, options.signal);
yield* prepared.stream({ ...options, ...prepared.config, messages });
},

@@ -302,3 +332,3 @@ };

ctx.effect(() => {
const dispose = ctx.llm.registerAdapter([state.config.providerId ?? DEFAULT_PROVIDER], proxy);
const dispose = ctx.llm.registerAdapter([providerId], proxy);
return typeof dispose === 'function' ? dispose : undefined;

@@ -305,0 +335,0 @@ }, 'dsh-vision-recognizer: adapter');

{
"name": "dsh-vision-recognizer",
"version": "0.1.3",
"description": "DeepSeek Harness image recognition plugin: keep DeepSeek as the conversation brain while a configurable vision model (15+ providers, OpenAI-compatible + Anthropic) transcribes attached images.",
"version": "0.2.0",
"description": "Adaptive image routing for DeepSeek Harness: pass images directly to native multimodal models and transcribe them only for text-only models.",
"private": false,

@@ -44,3 +44,3 @@ "type": "module",

"node": ">=22.19.0",
"dsh": ">=0.1.0-rc.6"
"dsh": ">=0.1.0-rc.8"
},

@@ -87,3 +87,7 @@ "dsh": {

"prepack": "npm test && npm run verify:plugin"
},
"devDependencies": {
"@deepseek-ai/cordis": "4.0.1",
"@deepseek-ai/dsh-llm": "0.1.0-rc.8"
}
}
+24
-16

@@ -7,9 +7,10 @@ # dsh-vision-recognizer

It registers a new provider route (default `vision-recognizer`, shown as **DeepSeek + 识图** in the model picker) that wraps the real DeepSeek adapter: it declares image input (so the attachment preflight and the `read_image` gate admit images) and, in the request stream, **transcribes every attached image to text through the vision model you select**, then delegates the text-only conversation to DeepSeek. DeepSeek still answers; recognition is an add-on.
It registers an adaptive provider route (default `vision-recognizer`, shown as **DeepSeek + 智能识图** in the model picker) that wraps the configured conversation provider. The wrapper always admits image attachments, then resolves the exact selected model: models declaring native image input receive the original image blocks directly; text-only or unknown-capability models receive text transcribed by the vision model you configure. DeepSeek remains the default wrapped conversation provider.
```
attached image ──▶ vision-recognizer route ──▶ vision-model transcription (OCR + layout + detail)
│ │
▼ ▼
DeepSeek answers ◀── text-only conversation (image replaced by [图片转译] text)
attached image ──▶ vision-recognizer route ──▶ selected model supports image? ── yes ─▶ native image request
no
configured vision transcription ──▶ text-only selected model
```

@@ -19,2 +20,3 @@

- **Adaptive routing**: native multimodal models receive images unchanged; only text-only or unknown-capability models invoke the configured transcription fallback.
- **One-click install**: `dsh plugin --profile web add dsh-vision-recognizer` — no build scripts, no `sharp` approval (no native dependencies at all).

@@ -47,8 +49,10 @@ - **Configure from Settings → Plugins → Vision**: pick a provider, enter an API key, override model / endpoint / token cap / timeout / marker. Saved changes take effect immediately, no restart.

1. Pick **DeepSeek + 识图** in the model selector;
2. Open **Settings → Plugins → Vision**, choose a provider, enter an API key, save;
3. Paste an image into any conversation → you should see the `[图片转译]` marker followed by a DeepSeek answer.
1. Pick **DeepSeek + 智能识图** in the model selector;
2. Open **Settings → Plugins → Vision**, choose the fallback vision provider, enter an API key, save;
3. Paste an image into a conversation. A native multimodal selected model receives it directly; a text-only selected model receives the `[图片转译]` result.
With no key and no local Ollama, a turn fails fast in a few seconds with guidance — that is the intended anti-hang behavior.
With a native multimodal selected model, no fallback key is required. With a text-only model and no key or local Ollama, the turn fails fast with guidance instead of hanging.
> **Scope:** adaptive fallback applies while the **DeepSeek + 智能识图** wrapper route is selected. Selecting another provider route calls that route directly. rc8 does not expose a public decorator hook that can add fallback behavior to every existing provider route.
## Supported providers

@@ -84,13 +88,17 @@

Only stable rc.6 public interfaces are used:
The adaptive wrapper uses rc8 public interfaces only:
- `ctx.llm.registration(innerProvider).adapter` — fetch the wrapped adapter;
- `ctx.llm.registerAdapter([providerId], proxyAdapter)` — register the new route;
- proxying `resolveModel` overrides `inputModalities` to `['text', 'image']`;
- proxying `stream` transcribes image blocks (`{ type: 'image', attachment }`, bytes read via `ctx.get('attachments').readImage(ref)`), then `yield*` forwards the inner adapter's stream;
- the settings UI rides the `settings.plugins.tab` slot plus custom `webServer` routes, persisting config to its own JSON file (independent of the api-proxy settings allowlist).
- `ctx.llm.listModels(innerProvider)` and `resolveModelInfo(innerProvider, model)` inspect the exact target model and rebind its metadata to the wrapper route;
- `ctx.llm.prepareCall(...)` delegates to the configured target provider without depending on private adapter registrations;
- proxy `resolveModel` advertises `['text', 'image']` so the wrapper admits images, while proxy `stream` uses the target's original modality declaration to choose native pass-through or transcription;
- transcription clones request messages and replaces image blocks only in the delegated request; durable session history keeps the original image references;
- the settings UI rides the `settings.plugins.tab` slot plus custom `webServer` routes, persisting config to its own JSON file.
### rc8 limitations
Capability lookup and prepared target dispatch are separate public operations in rc8. A target adapter replaced by HMR in that tiny interval can race the routing decision. Nested target delegation also enters the `llm/stream` waterfall a second time, and DSH may strip provider-private replay metadata when wrapper and target adapters differ. Ordinary text/image history is preserved; provider-specific replay signatures may lose their optimization or fidelity until DSH exposes an atomic delegation handle.
## Privacy
Transcription sends image bytes (base64, HTTPS) to the vision endpoint you configure — **image data leaves your machine** unless the endpoint is local (e.g. Ollama). Nothing beyond the harness's own attachment storage persists any image. For sensitive images, use your own endpoint or a local model, or don't install this plugin.
Native multimodal routing sends image bytes to the selected conversation provider. The text-only fallback instead sends them (base64, normally HTTPS) to the vision endpoint you configure. In either mode, image data leaves your machine unless that endpoint is local. Nothing beyond the harness's own attachment storage persists an image.

@@ -97,0 +105,0 @@ ## License

@@ -7,9 +7,10 @@ # dsh-vision-recognizer

它注册一条新的提供商路由(默认 `vision-recognizer`,模型选择器里显示为 **DeepSeek + 识图**),包装真正的 DeepSeek 适配器:对外声明支持图片输入(附件预检与 `read_image` 门禁放行),并在请求流里**把每张附加图片经你选定的视觉模型转译成文字**,再交给纯文本的 DeepSeek 作答。对话仍由 DeepSeek 完成,识图只是附加能力。
它注册一条自适应提供商路由(默认 `vision-recognizer`,模型选择器里显示为 **DeepSeek + 智能识图**),包装配置的对话提供商。路由始终接纳图片,然后解析当前选中的准确模型:声明原生图片能力的模型直接收到原始图片块;纯文本或能力未知的模型则收到由你配置的视觉模型生成的转译文字。默认被包装的对话提供商仍是 DeepSeek。
```
用户附加图片 ──▶ vision-recognizer 路由 ──▶ 视觉模型转译(OCR + 版式 + 细节)
│ │
▼ ▼
DeepSeek 作答 ◀── 纯文本对话(图片已替换为 [图片转译] 文字)
用户附加图片 ──▶ vision-recognizer 路由 ──▶ 当前模型支持原生图片? ── 是 ─▶ 原生多模态请求
配置的视觉模型转译 ──▶ 纯文本当前模型
```

@@ -19,2 +20,3 @@

- **自适应路由**:原生多模态模型直接收到图片;只有纯文本或能力未知的模型才调用配置的视觉转译回退。
- **一键安装**:`dsh plugin --profile web add dsh-vision-recognizer`,无需构建脚本、无需 approve `sharp`(不依赖任何原生模块)。

@@ -47,8 +49,10 @@ - **在「设置 → 插件 → 识图」里配置**:下拉选择供应商、填 API Key、可覆盖模型 / 接口地址 / token 上限 / 超时 / 标记,保存后**立即生效**,无需重启。

1. 模型选择器选择 **DeepSeek + 识图**;
2. 打开 **设置 → 插件 → 识图**,选供应商、填 API Key,点保存;
3. 向任意对话粘贴图片 → 应看到 `[图片转译]` 标记后 DeepSeek 作答。
1. 模型选择器选择 **DeepSeek + 智能识图**;
2. 打开 **设置 → 插件 → 识图**,配置回退视觉供应商和 API Key;
3. 粘贴图片:当前模型原生支持图片时直接传图;纯文本模型会收到带 `[图片转译]` 的识别结果。
没有 Key 也没有本地 Ollama 时,回合会在数秒内快速失败并给出指引——这是预期的防卡死行为。
当前模型原生支持图片时不需要配置回退 Key;只有纯文本模型且本地没有 Ollama、也没有 Key 时,才会快速失败并提示配置。
> **作用范围:** 自适应回退只在选中 **DeepSeek + 智能识图** 包装路由时生效。若直接选择其他提供商路由,请求会直接走该路由;rc8 还没有公开的 adapter 装饰钩子,插件无法给所有既有路由全局加回退。
## 支持的供应商

@@ -84,13 +88,17 @@

本插件只使用 rc.6 上稳定的公共接口:
自适应包装只使用 rc8 公共接口:
- `ctx.llm.registration(innerProvider).adapter` —— 拿到被包装的适配器;
- `ctx.llm.registerAdapter([providerId], proxyAdapter)` —— 注册新路由;
- 代理 `resolveModel` 把 `inputModalities` 覆盖为 `['text', 'image']`;
- 代理 `stream` 转译图片块(`{ type: 'image', attachment }`,字节经 `ctx.get('attachments').readImage(ref)` 读取),再 `yield*` 原样转发内部适配器流;
- 设置界面走 `settings.plugins.tab` 插槽 + 自定义 `webServer` 路由,配置持久化到自有 JSON 文件(不依赖 api-proxy 的 settings 白名单)。
- `ctx.llm.listModels(innerProvider)` 与 `resolveModelInfo(innerProvider, model)` 检查准确目标模型,并把模型元数据重新绑定到包装路由;
- `ctx.llm.prepareCall(...)` 委派到配置的目标提供商,不依赖私有 adapter registration;
- 代理 `resolveModel` 对外声明 `['text', 'image']` 以接纳图片,代理 `stream` 再根据目标模型原始能力选择原生直传或转译;
- 转译只克隆并修改本次委派的消息,持久会话历史仍保留原始图片引用;
- 设置界面走 `settings.plugins.tab` 插槽 + 自定义 `webServer` 路由,配置持久化到自有 JSON 文件。
### rc8 限制
rc8 的能力查询与目标 `prepareCall` 是两个独立公开操作;如果目标 adapter 恰好在这个极短窗口被 HMR 替换,能力判断可能与最终派发版本不一致。嵌套目标调用还会第二次进入 `llm/stream` waterfall;当包装与目标不是同一个 adapter 时,DSH 可能移除提供商私有的 replay 元数据。普通图文历史不会丢失,但依赖私有 replay 签名的提供商可能暂时失去缓存优化或部分保真度,直到 DSH 提供原子委派句柄。
## 隐私
转译会把图片字节(base64,HTTPS)发送到你配置的视觉端点——**图片数据会离开你的机器**,除非端点指向本地服务(如 Ollama)。除 harness 自身的附件存储外不持久化任何图片。敏感图片请使用自己的端点或本地模型,或不要安装本插件。
原生多模态路由会把图片发送给当前选中的对话提供商;纯文本回退则把图片(base64,通常为 HTTPS)发送给你配置的视觉端点。除非相应端点是本地服务,否则图片数据都会离开你的机器。除 Harness 自身附件存储外,本插件不额外持久化图片。

@@ -97,0 +105,0 @@ ## 许可证