New:Socket for Asana Is Now Available.Learn more
Get Started

token-optimizer-mcp

Package Overview
Dependencies
Maintainers
1
Versions
29
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

token-optimizer-mcp

Intelligent context window optimization for Claude Code - store content externally via caching and compression, freeing up your context window for what matters

Source
npmnpm
Version
2.13.0
Version published
Weekly downloads
119
-46.4%
Maintainers
1
Weekly downloads
 
Created
Source

Token Optimizer MCP

Intelligent token optimization through caching and compression for Claude Code and Claude Desktop

Overview

Token Optimizer MCP is a Model Context Protocol (MCP) server that reduces context window usage through intelligent caching and compression. By storing compressed content externally in SQLite, the server removes tokens from your context window while keeping them accessible. The server provides tools to compress text, cache results, and analyze token usage - helping you maximize your available context window.

Key Features

  • Context Window Optimization: Store content externally to free up context window space
  • High Compression: Brotli compression (2-4x typical, up to 82x for repetitive content)
  • Persistent Caching: SQLite-based cache that persists across sessions
  • Accurate Token Counting: Uses tiktoken for precise token measurements
  • Smart Analysis: Analyze text to determine optimal caching strategy
  • Zero External Dependencies: Completely offline operation
  • Production Ready: Built with TypeScript for reliability

Implemented Features

Core Modules

  • Cache Engine - SQLite-based persistent storage with automatic cleanup
  • Token Counter - Accurate token counting using tiktoken (GPT-4 tokenizer)
  • Compression Engine - Brotli compression for optimal token efficiency

Available MCP Tools

  • optimize_text - Compress and cache text to reduce token usage
  • get_cached - Retrieve previously cached and optimized text
  • count_tokens - Count tokens in text using tiktoken
  • compress_text - Compress text using Brotli compression
  • decompress_text - Decompress base64-encoded Brotli-compressed text
  • get_cache_stats - Get cache statistics including hit rate and compression ratio
  • clear_cache - Clear all cached data
  • analyze_optimization - Analyze text and get optimization recommendations

Technology Stack

  • Runtime: Node.js 20+
  • Language: TypeScript
  • Database: SQLite (better-sqlite3)
  • Token Counting: tiktoken
  • Compression: Brotli (built-in)
  • Caching: LRU Cache
  • Protocol: MCP SDK (@modelcontextprotocol/sdk)

Installation

npm install
npm run build

Configuration

For Claude Code

The server is already configured in .mcp.json at the project root. To use it:

  • Restart Claude Code (the server will auto-load)
  • The token-optimizer tools will appear in your available MCP tools

For Claude Desktop

The server is configured in claude_desktop_config.json. To verify:

  • Check that the configuration file at %APPDATA%\Roaming\Claude\claude_desktop_config.json includes:
{
  "mcpServers": {
    "token-optimizer": {
      "command": "node",
      "args": [
        "C:\\Users\\yolan\\source\\repos\\token-optimizer-mcp\\dist\\server\\index.js"
      ]
    }
  }
}
  • Restart Claude Desktop
  • The token-optimizer tools will be available in all conversations

Usage Examples

Optimize and Cache Text

// Use the optimize_text tool to cache content externally
optimize_text({
  text: "Your large text content here...",
  key: "my-cache-key",
  quality: 11  // 0-11, higher = better compression
})

// Result - compressed data is stored in SQLite, NOT returned in context
{
  "success": true,
  "key": "my-cache-key",
  "originalTokens": 1500,
  "compressedTokens": 450,        // Tokens IF it were in context (not relevant)
  "tokensSaved": 1050,             // Context window savings (what matters)
  "percentSaved": 70.5,            // Based on compression + external storage
  "originalSize": 6000,
  "compressedSize": 1800,          // Stored in SQLite
  "cached": true
}

// Your context window now contains only the cache key (~50 tokens)
// instead of the original 1500 tokens - 96.7% reduction in context usage!

Retrieve Cached Text

// Use the get_cached tool
get_cached({ key: "my-cache-key" })

// Result:
{
  "success": true,
  "key": "my-cache-key",
  "text": "Your original text content...",
  "fromCache": true
}

Count Tokens

// Use the count_tokens tool
count_tokens({ text: "Your text here" })

// Result:
{
  "tokens": 150,
  "characters": 500
}

Analyze Optimization Potential

// Use the analyze_optimization tool
analyze_optimization({ text: "Your text here" })

// Result:
{
  "tokens": {
    "current": 1500,
    "afterCompression": 450,
    "saved": 1050,
    "percentSaved": 70
  },
  "size": {
    "current": 6000,
    "compressed": 1800,
    "ratio": 3.33,
    "percentSaved": 70
  },
  "recommendations": {
    "shouldCompress": true,
    "reason": "Compression will provide significant token savings"
  }
}

Get Cache Statistics

// Use the get_cache_stats tool
get_cache_stats({})

// Result:
{
  "totalEntries": 15,
  "totalSize": 45000,
  "hits": 42,
  "misses": 8,
  "hitRate": 84.0,
  "avgCompressionRatio": 3.2,
  "totalTokensSaved": 12500
}

How Token Optimization Works

Understanding Context Window Savings vs Compression Ratio

It's important to understand the difference between compression ratio and context window savings:

Compression Ratio

  • Measures how much the original data is reduced in size (e.g., 10KB → 2KB = 5x compression)
  • Brotli achieves 2-4x typical compression, up to 82x for highly repetitive content
  • Does NOT directly translate to token savings in the compressed form

Context Window Savings (The Real Benefit)

When you cache content using this MCP server:

  • Original text is compressed with Brotli (up to 82x compression)
  • Compressed data is stored externally in SQLite database
  • 100% of original tokens are removed from your context window
  • Only a small cache key remains in context (~50 tokens for key + metadata)

Example: A 10,000 token API response is cached:

  • Before: 10,000 tokens in your context window
  • After: ~50 tokens (cache key + metadata)
  • Savings: 9,950 tokens removed from context (99.5% reduction)

Why Base64 Encoding Increases Token Count

When you compress text without caching (using compress_text), the compressed data must be encoded as Base64 to be transmitted as text:

  • Base64 encoding adds ~33% overhead to the compressed size
  • This often results in MORE tokens than the original (unless compression ratio >4x)
  • Solution: Use optimize_text which caches the compressed data externally

When Token Optimization Works Best

High Value Use Cases:

  • Caching large API responses that are referenced multiple times
  • Storing repetitive configuration or data files
  • Caching large code files that need to be referenced repeatedly
  • Archiving conversation history while keeping it accessible

Lower Value Use Cases:

  • Small text snippets (<500 characters) - overhead exceeds savings
  • One-time use content - no benefit from caching
  • Content with low compression ratio - external storage still helps

Token Optimization Workflow

Original Text (10,000 tokens)
        ↓
  Brotli Compress (82x ratio)
        ↓
Store in SQLite (~122 bytes)
        ↓
Return Cache Key (~50 tokens)
        ↓
RESULT: 9,950 tokens removed from context window

The key insight: The value is in external storage, not compression alone. Even with modest compression ratios, moving data out of your context window provides massive savings.

Development

# Build the project
npm run build

# Run in development mode (watch)
npm run dev

# Run tests (when implemented)
npm test

# Run benchmarks (when implemented)
npm run benchmark

How It Works

  • Compression: Uses Brotli compression (quality 11) to reduce text size
  • Token Counting: Uses tiktoken with GPT-4 tokenizer for accurate counts
  • Caching: Stores compressed text in SQLite database for persistence
  • Cache Management: Automatic cleanup of old entries to prevent unbounded growth

Performance

  • Compression Ratio: Typically 2-4x size reduction (up to 82x for highly repetitive content)
  • Context Window Savings: Up to 100% for cached content (removed from context window)
  • Cache Hit Rate: >80% in typical usage
  • Overhead: <10ms for cache operations
  • Compression Speed: ~1ms per KB of text

Limitations

  • Small Text: Best for text >500 characters (cache overhead on small snippets)
  • Base64 Overhead: Compressed-only output (without caching) may use MORE tokens due to Base64 encoding
  • Cache Storage: Cache size limited to prevent disk usage issues (automatic cleanup after 7 days)
  • Compression Tradeoff: Quality setting affects speed vs ratio (default quality 11 is optimal)
  • Token Counting: Uses GPT-4 tokenizer (approximation for Claude, but close enough for optimization decisions)
  • One-Time Content: No benefit for content that won't be referenced again (caching provides the value)

License

ISC

Author

Built for optimal Claude Code token efficiency.

Keywords

mcp

FAQs

Package last updated on 25 Oct 2025

Related posts