@composio/embeddings

@composio/embeddings

TurboPuffer schema definition for the Composio tool lookup system.

Overview

This package defines the TurboPuffer schema for storing and searching tools using:

• Vector embeddings: 3072-dimensional vectors from OpenAI text-embedding-3-large
• BM25 full-text search: On formatted tool descriptions
• Filterable attributes: Toolkit names, tool names, display names

All types extend the official @turbopuffer/turbopuffer SDK types.

Exports

• TOOL_SCHEMA: TurboPuffer schema configuration using AttributeSchema type
• TOOL_DISTANCE_METRIC: Distance metric for vector similarity (cosine_distance)
• ToolDocumentSchema: TypeScript type for tool documents
• ToolRow: Query result type extending TurboPuffer's Row

Re-exported TurboPuffer Types

• Row, Vector, DistanceMetric
• AttributeSchema, AttributeSchemaConfig
• NamespaceQueryParams, NamespaceQueryResponse
• NamespaceWriteParams, NamespaceWriteResponse
• Turbopuffer client

Tool Document Schema

TOOL_SCHEMA is the single source of truth. All TypeScript types are derived from it.

import type { ToolDocumentSchema } from '@composio/embeddings'
import { TOOL_SCHEMA } from '@composio/embeddings'

// Schema definition (source of truth)
TOOL_SCHEMA = {
  text: { type: 'string', full_text_search: true },
  tool_name: { type: 'string', filterable: true },
  display_name: { type: 'string', filterable: true },
  description: { type: 'string' },
  toolkit_name: { type: 'string', filterable: true },
  created_at: { type: 'string' },
}

// TypeScript type (automatically derived from TOOL_SCHEMA)
type ToolDocumentSchema = {
  id: string // Random 10-char identifier
  vector: Vector // 3072-dimensional embedding
  text: string // Formatted text (BM25 searchable)
  tool_name: string // Tool slug (filterable)
  display_name: string // Human-readable name (filterable)
  description: string // Tool description
  toolkit_name: string // App/toolkit name (filterable)
  created_at: string // ISO timestamp
}

Usage

import {
  TOOL_DISTANCE_METRIC,
  TOOL_SCHEMA,
  Turbopuffer,
} from '@composio/embeddings'

// Use schema when writing to TurboPuffer
const tpuf = new Turbopuffer({ apiKey: process.env.TURBOPUFFER_API_TOKEN })
const namespace = tpuf.namespace('tool-lookup-v3')

await namespace.write({
  upsert_rows: [...tools],
  schema: TOOL_SCHEMA,
  distance_metric: TOOL_DISTANCE_METRIC,
})

Design Context & Resources

Slack Discussions

• TurboPuffer implementation - Jayesh's update on TurboPuffer and Weaviate vector store implementations
• Weaviate integration - Testing Weaviate implementation for vector store interface

Notion Documentation

• Embedding Model Comparison for Workflow Cache System - Comparison of embedding models for the workflow cache
• Tool embedding (non-cached path) - Experiment on tool embedding non-cached search path (Rahul Tarak, P1)

Related Documentation

• See CLAUDE.md in this directory for AI assistant context
• See /apps/apollo for tool search API implementation