@krishgupta/ai-prompt-cache

@krishgupta/ai-prompt-cache

Middleware for the Vercel AI SDK that enables prompt caching across providers. Tested improvements show 50%+ reduction in time-to-first-token (TTFT).

Features

• Multi-provider prompt caching (OpenAI, Anthropic, Bedrock, Gemini)
• Full response caching with streaming replay
• In-flight request coalescing to deduplicate concurrent calls
• Key sharding for high-QPS scenarios
• Observability hooks for tracking cache hits and TTFT

Installation

npm install @krishgupta/ai-prompt-cache

Usage

import { wrapLanguageModel, streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { withPromptCache } from '@krishgupta/ai-prompt-cache';

const model = wrapLanguageModel({
  model: openai('gpt-4o'),
  middleware: withPromptCache({
    select: 'system-head',
    extraKeySalt: 'my-app-v1',
  }),
});

const result = await streamText({
  model,
  messages: [
    { role: 'system', content: 'You are a helpful assistant...' },
    { role: 'user', content: 'Hello!' },
  ],
});

Benchmark Results

Tested with OpenAI gpt-4o and a ~1200 line system prompt:

Request	Mode	TTFT
1	Baseline	2223 ms
2	With Cache	1090 ms

Result: 51% faster TTFT

Server-side metrics showed consistent cache key generation and TTFT dropping from 377ms to 337ms on subsequent cached requests.

Supported Providers

• OpenAI (via promptCacheKey)
• Anthropic (via cacheControl markers)
• AWS Bedrock (via cachePoint)
• Google Gemini (implicit caching)
• OpenAI-compatible APIs

Options

Option	Default	Description
select	system-head	Which prefix to cache. Options: system-head, tools+system, or a custom function
extraKeySalt	undefined	Additional data to include in cache key (useful for RAG chunk IDs)
onCacheResult	undefined	Callback with cache report after request completes
debug	false	Enable debug logging

Response Caching

For full response caching with streaming replay:

import { withPromptCache, withResponseCache, MemoryStore } from '@krishgupta/ai-prompt-cache';

const store = new MemoryStore({ maxSize: 1000 });

const model = wrapLanguageModel({
  model: openai('gpt-4o'),
  middleware: [
    withPromptCache({ select: 'system-head' }),
    withResponseCache({ store, ttlSeconds: 3600 }),
  ],
});

Cache Stores

Built-in stores:

// In-memory LRU cache
import { MemoryStore } from '@krishgupta/ai-prompt-cache';
const store = new MemoryStore({ maxSize: 1000 });

// File-based cache (for development)
import { FileStore } from '@krishgupta/ai-prompt-cache';
const store = new FileStore({ directory: '.cache' });

Custom stores need to implement:

interface CacheStore {
  get(key: string): Promise<string | null>;
  set(key: string, value: string, ttlSeconds?: number): Promise<void>;
}

How It Works

The middleware generates a stable SHA-256 hash of the cacheable prefix (system prompt by default) and passes it to the provider:

• OpenAI: Sets promptCacheKey in provider options
• Anthropic: Adds cacheControl markers to messages
• Bedrock: Inserts cachePoint for Claude models

OpenAI caches prompts with 1024+ tokens and reuses them in 128-token increments. Anthropic requires similar minimum thresholds depending on the model.

Provider Requirements

Provider	Minimum Tokens
OpenAI	1024
Anthropic Claude Sonnet	1024
Anthropic Claude Haiku	2048

License

MIT