@oncely/ai

@oncely/ai

AI SDK middleware for idempotent LLM calls. Prevent duplicate API calls and save tokens.

The Problem

AI API calls are expensive. When networks fail, users retry, or your app has bugs—you pay twice for the same work. Neither OpenAI, Anthropic, nor the AI SDK provide built-in idempotency.

The Solution

import { wrapLanguageModel, generateText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { idempotencyMiddleware } from '@oncely/ai';

const model = wrapLanguageModel({
  model: openai('gpt-4-turbo'),
  middleware: idempotencyMiddleware({ ttl: '5m' }),
});

// First call - hits the API, costs tokens
const result1 = await generateText({ model, prompt: 'Explain quantum computing' });

// Retry/duplicate - returns cached response instantly, zero tokens
const result2 = await generateText({ model, prompt: 'Explain quantum computing' });

Installation

npm install @oncely/ai @oncely/core ai

For production, add a storage adapter:

npm install @oncely/redis ioredis   # Standard Redis
npm install @oncely/upstash         # Serverless (Upstash, Vercel KV)

Usage

Basic (Memory Storage)

import { wrapLanguageModel } from 'ai';
import { idempotencyMiddleware } from '@oncely/ai';

const model = wrapLanguageModel({
  model: yourModel,
  middleware: idempotencyMiddleware(),
});

Production (Redis)

import { wrapLanguageModel } from 'ai';
import { idempotencyMiddleware } from '@oncely/ai';
import { redis } from '@oncely/redis';

const model = wrapLanguageModel({
  model: openai('gpt-4-turbo'),
  middleware: idempotencyMiddleware({
    storage: redis(),
    ttl: '5m',
  }),
});

Serverless (Upstash)

import { idempotencyMiddleware } from '@oncely/ai';
import { upstash } from '@oncely/upstash';

const model = wrapLanguageModel({
  model: anthropic('claude-3-opus'),
  middleware: idempotencyMiddleware({
    storage: upstash(),
    ttl: '10m',
  }),
});

How Keys Are Generated

By default, the middleware generates a cache key by hashing:

• Model ID
• Prompt/messages
• Temperature, max tokens, and other generation parameters

Same inputs = same key = cached response.

Explicit Keys

Pass an explicit key via providerOptions:

const result = await generateText({
  model,
  prompt: 'Hello',
  providerOptions: {
    oncely: { key: 'user-123-greeting' },
  },
});

Custom Key Function

const model = wrapLanguageModel({
  model: yourModel,
  middleware: idempotencyMiddleware({
    getKey: (params) => {
      // Your custom key logic
      return `custom:${hashObject(params.prompt)}`;
    },
  }),
});

Options

Option	Type	Default	Description
storage	StorageAdapter	MemoryStorage	Storage backend
ttl	string | number	'5m'	Cache duration
getKey	(params) => string	Auto-hash	Custom key generation
includeModelInKey	boolean	true	Include model ID in cache key
onHit	(key, response) => void	—	Callback on cache hit
onMiss	(key) => void	—	Callback on cache miss

Per-Request Options

Override settings per-request via providerOptions.oncely:

const result = await generateText({
  model,
  prompt: 'Hello',
  providerOptions: {
    oncely: {
      key: 'explicit-key', // Override auto-generated key
      ttl: '1h', // Override TTL for this request
      skip: true, // Skip idempotency entirely
    },
  },
});

Works With Any Provider

The middleware works with any AI SDK provider:

• OpenAI (@ai-sdk/openai)
• Anthropic (@ai-sdk/anthropic)
• Google (@ai-sdk/google)
• Mistral (@ai-sdk/mistral)
• Cohere (@ai-sdk/cohere)
• Local models (Ollama, llama.cpp)
• Any custom provider

Streaming Support

Works with both generateText and streamText:

const result = await streamText({
  model,
  prompt: 'Write a poem',
});

// Cached streams are replayed from storage
for await (const chunk of result.textStream) {
  console.log(chunk);
}

Combining with Other Middleware

import { wrapLanguageModel } from 'ai';
import { idempotencyMiddleware } from '@oncely/ai';

const model = wrapLanguageModel({
  model: openai('gpt-4-turbo'),
  middleware: [
    idempotencyMiddleware({ storage: redis() }),
    loggingMiddleware(),
    rateLimitMiddleware(),
  ],
});

Use Cases

• Chatbots — Prevent duplicate completions on network retries
• Agents — Ensure tool calls execute exactly once
• Batch processing — Resume interrupted jobs without re-running
• Rate limit recovery — Retry without burning extra tokens
• Development — Cache expensive calls during iteration

Related Packages

• @oncely/core — Core library
• @oncely/redis — Redis adapter
• @oncely/upstash — Upstash adapter

License

MIT