@thinkhive/sdk

ThinkHive SDK v4.2.2

The official JavaScript/TypeScript SDK for ThinkHive - AI Agent Observability Platform.

Features

• OpenTelemetry-Based Tracing: Built on OTLP for seamless integration with existing observability tools
• Run-Centric Architecture: Atomic unit of work tracking with claims, calibration, and linking
• Facts vs Inferences: Claims API for separating verified facts from inferences
• Deterministic Ticket Linking: 7 methods for linking runs to support tickets (Zendesk, Intercom, Salesforce, Freshdesk)
• Calibrated Predictions: Brier scores and ECE for prediction accuracy
• Configurable ROI Engine: Business impact calculation with customizable cost, deflection, and attribution models
• Auto-Instrumentation: Works with LangChain, OpenAI, Anthropic, and more
• Multi-Format Support: Normalizes traces from 25+ observability platforms

Installation

npm install @thinkhive/sdk

Quick Start

Initialization

import { init, runs, shutdown } from '@thinkhive/sdk';

init({
  apiKey: 'thk_your_api_key',
  serviceName: 'my-ai-agent',
  autoInstrument: true,
  frameworks: ['langchain', 'openai'],
});

// Create a run (atomic unit of work)
const run = await runs.create({
  agentId: 'weather-agent',
  conversationMessages: [
    { role: 'user', content: 'What is the weather in San Francisco?' },
    { role: 'assistant', content: 'The weather in San Francisco is currently 65F and sunny.' }
  ],
  outcome: 'success',
});

console.log(`Run ID: ${run.id}`);
await shutdown();

API Reference

Core APIs (V3)

API	Import	Description
runs	@thinkhive/sdk	Create and manage runs (atomic work units)
claims	@thinkhive/sdk	Manage facts/inferences for runs
calibration	@thinkhive/sdk	Track prediction accuracy (Brier scores)
linking	@thinkhive/sdk	Connect runs to support tickets (7 methods)
customerContext	@thinkhive/sdk	Time-series customer snapshots
roiAnalytics	@thinkhive/sdk	Configurable ROI calculation engine
agents	@thinkhive/sdk	Agent CRUD management
apiKeys	@thinkhive/sdk	API key management with scoped permissions

Assessment APIs

API	Description
humanReview	Human-in-the-loop review queues
nondeterminism	Multi-sample reliability testing
evalHealth	Metric health monitoring
deterministicGraders	Rule-based grading
conversationEval	Multi-turn conversation grading
transcriptPatterns	Pattern detection in transcripts

Operations & Monitoring APIs

API	Description
evalRuns	Create, manage, and query evaluation runs
signals	Behavioral signal configuration, stats, and trends
notifications	Alert rules and notification management
documents	Agent RAG document upload and management
shadowTests	Shadow test creation and execution
sessions	Trace session grouping and querying
drift	Model/behavior drift detection
llmCosts	LLM usage cost tracking and optimization

Legacy APIs (V2)

API	Description
issues	Clustered failure patterns
analyzer	User-selected trace analysis with cost estimation
businessMetrics	Industry-driven metrics with historical tracking
qualityMetrics	RAG scoring and hallucination detection

Runs API

import { runs, createRunWithContext } from '@thinkhive/sdk';

// Create a run
const run = await runs.create({
  agentId: 'support-agent',
  conversationMessages: [
    { role: 'user', content: 'Help me cancel my order' },
    { role: 'assistant', content: 'I can help you cancel your order...' }
  ],
  outcome: 'resolved',
  durationMs: 1500,
});

// List runs with filters
const result = await runs.list({
  agentId: 'support-agent',
  outcome: 'resolved',
  startedAfter: '2025-01-01',
  limit: 50,
});

// Get a specific run
const existing = await runs.get('run_abc123');

// Get run statistics
const stats = await runs.stats('support-agent');

Claims API (Facts vs Inferences)

import { claims, isFact, isInference, getHighConfidenceClaims } from '@thinkhive/sdk';

// List claims for a run
const claimList = await claims.list(runId);

// Filter by type
const facts = claimList.filter(isFact);
const inferences = claimList.filter(isInference);

// Get high confidence claims
const confident = getHighConfidenceClaims(claimList, 0.9);

// Group by category
const byCategory = groupClaimsByCategory(claimList);

Calibration API

import { calibration, calculateBrierScore, isWellCalibrated } from '@thinkhive/sdk';

// Get calibration status for a prediction type
const status = await calibration.status('agent-123', 'churn_risk');

// Get all calibration metrics
const metrics = await calibration.allMetrics('agent-123');

// Single prediction
const score = calculateBrierScore(0.8, true); // 0.04

// Batch calculation
const batchScore = calculateBrierScore([
  { predicted: 0.8, actual: 1 },
  { predicted: 0.3, actual: 0 },
]);

if (isWellCalibrated(score)) {
  console.log('Agent predictions are well calibrated');
}

Ticket Linking (V3)

Deterministic linking between runs and support tickets using 7 methods with explicit confidence scores.

import {
  linking,
  generateZendeskMarker,
  linkRunToTicket,
  linkRunToZendeskTicket,
  getBestLinkMethod,
  LINK_METHOD_CONFIDENCE,
} from '@thinkhive/sdk';

// SDK explicit linking (confidence: 1.0)
const link = await linking.create({
  runId: 'run_abc123',
  ticketId: 'ticket_xyz',
  method: 'sdk_explicit',
});

// Zendesk marker linking
const zdLink = await linking.create({
  runId: 'run_abc123',
  externalTicketId: '12345',
  platform: 'zendesk',
  method: 'zendesk_marker',
});

// Convenience functions
await linkRunToTicket('run_abc123', 'ticket_xyz');
await linkRunToZendeskTicket('run_abc123', '12345');

// Get links for a run
const links = await linking.getForRun('run_abc123');

// Get links for a ticket
const ticketLinks = await linking.getForTicket('ticket_xyz');

// Verify a link
await linking.verify('link_id', { verified: true, notes: 'Confirmed by agent' });

// Auto-link using all available evidence
const autoLinked = await linking.autoLink('run_abc123');

// Get link statistics
const stats = await linking.stats();
// { totalLinks, byMethod, avgConfidence, verifiedCount, unverifiedCount }

// Generate a Zendesk marker to embed in responses
const marker = generateZendeskMarker('run_abc123');
// Returns: '[THID:run_abc123]'

// Server-side marker generation with format options
const serverMarker = await linking.generateMarker({
  traceId: 'trace_123',
  format: 'html_comment',
});

// Delete a link
await linking.delete('link_id');

Link Methods and Confidence

Method	Confidence	Description
sdk_explicit	1.0	Direct SDK call with ticket ID
zendesk_marker	1.0	Embedded THID marker in response
custom_field	1.0	Zendesk custom field
manual	1.0	Human-assigned
middleware_stamp	0.98	Middleware-injected trace ID
session_match	0.95	Session ID correlation
email_time_window	0.6	Email + 15min window

ROI Analytics (V3)

Configurable ROI calculation engine with customizable cost models, deflection definitions, and attribution rules.

Configuration

import { roiAnalytics } from '@thinkhive/sdk';

// Get active ROI config (auto-provisions default if none exists)
const config = await roiAnalytics.getConfig();

// Create custom ROI config
const newConfig = await roiAnalytics.createConfig({
  costConfig: {
    humanAgentCostPerHour: 35,
    aiCostPer1kTokens: 0.01,
    escalationCostMultiplier: 2.5,
    infrastructureMonthlyCost: 200,
  },
  deflectionConfig: {
    definition: 'resolved_without_human',
    outcomesThatCount: ['resolved'],
    minConfidence: 0.8,
    excludeEscalated: true,
  },
  resolutionConfig: {
    definition: 'customer_confirmed',
    requiresSatisfaction: false,
    minSatisfactionScore: null,
    noRecontactWindowHours: 24,
  },
  attributionConfig: {
    timeSavedValuePerHour: 50,
    churnPreventionAttribution: 0.2,
    upsellAttribution: 0.1,
    useActualARR: true,
    defaultARRIfUnknown: 10000,
  },
});

// Update config (creates new version, keeps history)
const updated = await roiAnalytics.updateConfig({
  costConfig: { humanAgentCostPerHour: 40 },
});

// List config version history
const versions = await roiAnalytics.configVersions({ limit: 10 });

Calculate ROI

// Calculate ROI for a date range
const result = await roiAnalytics.calculateV3({
  startDate: '2025-01-01',
  endDate: '2025-02-01',
  includeBreakdown: true,
  includeConfidenceIntervals: false,
});

console.log(`Net ROI: $${result.summary.netROI}`);
console.log(`ROI %: ${result.summary.roiPercentage}%`);
console.log(`Deflection Rate: ${result.metrics.deflectionRate}`);
console.log(`Total Runs: ${result.metrics.totalRuns}`);

// Filter by agent
const agentROI = await roiAnalytics.calculateV3({
  agentId: 'support-agent',
  startDate: '2025-01-01',
  endDate: '2025-02-01',
});

ROI Trends

// Get ROI trend over time
const trend = await roiAnalytics.trendV3({
  granularity: 'day', // 'day' | 'week' | 'month'
  startDate: '2025-01-01',
  endDate: '2025-02-01',
});

for (const bucket of trend.trend) {
  console.log(`${bucket.periodStart}: ${bucket.runCount} runs, $${bucket.netROI} ROI`);
}

if (trend.comparison) {
  console.log(`Improving: ${trend.comparison.isImproving}`);
  console.log(`Change: ${trend.comparison.changePercent}%`);
}

Legacy V1 ROI (still available)

// V1 summary endpoint
const summary = await roiAnalytics.summary({
  startDate: '2025-01-01',
  endDate: '2025-01-31',
});

// V1 trends
const trends = await roiAnalytics.trends({
  startDate: '2025-01-01',
  endDate: '2025-01-31',
});

// V1 per-trace ROI calculation
const impact = await roiAnalytics.calculate({
  traceId: 'trace_abc123',
});

// V1 industry configs
const industries = await roiAnalytics.industries();

// V1 correlation analysis
const correlations = await roiAnalytics.correlations();

Manual Tracing

import { traceLLM, traceRetrieval, traceTool, traceChain } from '@thinkhive/sdk';

// Trace an LLM call
const response = await traceLLM({
  name: 'generate-response',
  modelName: 'gpt-4',
  provider: 'openai',
  input: { prompt: 'Hello!' }
}, async () => {
  return await openai.chat.completions.create({...});
});

// Trace a retrieval operation
const docs = await traceRetrieval({
  name: 'search-knowledge-base',
  query: 'refund policy',
  topK: 5
}, async () => {
  return await vectorStore.similaritySearch(query, 5);
});

// Trace a tool call
const result = await traceTool({
  name: 'lookup-order',
  toolName: 'order_lookup',
  parameters: { orderId: '12345' }
}, async () => {
  return await lookupOrder('12345');
});

// Trace a chain/workflow
const output = await traceChain({
  name: 'support-workflow',
  input: { query: 'help with refund' }
}, async () => {
  const docs = await retrieveContext(query);
  return await generateResponse(docs);
});

Analyzer API

import { analyzer } from '@thinkhive/sdk';

// Estimate cost before running analysis
const estimate = await analyzer.estimateCost({
  traceIds: ['trace-1', 'trace-2'],
  tier: 'standard',
});
console.log(`Estimated cost: $${estimate.estimatedCost}`);

// Analyze specific traces
const analysis = await analyzer.analyze({
  traceIds: ['trace-1', 'trace-2'],
  tier: 'standard',
  includeRootCause: true,
  includeLayers: true,
});

// Analyze traces by time window with smart sampling
const windowAnalysis = await analyzer.analyzeWindow({
  agentId: 'support-agent',
  startDate: new Date('2025-01-01'),
  endDate: new Date('2025-01-31'),
  filters: { outcomes: ['failure'], minSeverity: 'medium' },
  sampling: { strategy: 'smart', samplePercent: 10 },
});

// Get aggregated insights
const summary = await analyzer.summarize({
  agentId: 'support-agent',
  startDate: new Date('2025-01-01'),
  endDate: new Date('2025-01-31'),
});

Issues API

import { issues } from '@thinkhive/sdk';

const issueList = await issues.list('support-agent', { status: 'open', limit: 10 });
const issue = await issues.get('issue-123');
const fixes = await issues.getFixes('issue-123');

API Keys

import { apiKeys, hasPermission, canAccessAgent } from '@thinkhive/sdk';

const result = await apiKeys.create({
  name: 'CI Pipeline Key',
  permissions: { read: true, write: true, delete: false },
  scopeType: 'agent',
  allowedAgentIds: ['agent-prod-001'],
  environment: 'production',
  expiresAt: new Date(Date.now() + 90 * 24 * 60 * 60 * 1000),
});

if (hasPermission(key, 'write')) { /* can write */ }
if (canAccessAgent(key, 'agent-123')) { /* can access */ }

Business Metrics

import { businessMetrics, isMetricReady, getStatusMessage } from '@thinkhive/sdk';

const metric = await businessMetrics.current('agent-123', 'Deflection Rate');
const history = await businessMetrics.history('agent-123', 'Deflection Rate', {
  startDate: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000),
  granularity: 'daily',
});

await businessMetrics.record('agent-123', {
  metricName: 'CSAT/NPS',
  value: 4.5,
  unit: 'score',
  periodStart: '2025-01-01T00:00:00Z',
  periodEnd: '2025-01-07T23:59:59Z',
  source: 'survey_system',
});

Auto-Instrumentation

import { init } from '@thinkhive/sdk';

init({
  apiKey: 'thk_your_api_key',
  serviceName: 'my-ai-agent',
  autoInstrument: true,
  frameworks: ['langchain', 'openai', 'anthropic'],
});
// All LangChain, OpenAI, and Anthropic calls are now automatically traced

Sub-package Imports

// OpenAI instrumentation
import { instrumentOpenAIClient } from '@thinkhive/sdk/instrumentation/openai';

// LangChain instrumentation
import { ThinkHiveCallbackHandler } from '@thinkhive/sdk/instrumentation/langchain';

// Ticket linking (standalone)
import { linking, generateZendeskMarker } from '@thinkhive/sdk/integrations/ticket-linking';

// Customer context (standalone)
import { customerContext, captureCustomerContext } from '@thinkhive/sdk/integrations/customer-context';

Environment Variables

Variable	Description
THINKHIVE_API_KEY	Your ThinkHive API key
THINKHIVE_ENDPOINT	Custom API endpoint (default: https://app.thinkhive.ai)
THINKHIVE_SERVICE_NAME	Service name for traces (optional)
THINKHIVE_AGENT_ID	Default agent ID (optional)

Eval Runs

import { evalRuns } from '@thinkhive/sdk';

// Create an evaluation run
const run = await evalRuns.create('agent-123', { confidenceLevel: 'medium' });

// Get results
const results = await evalRuns.getResults(run.id, { limit: 50 });

// Estimate cost before running
const cost = await evalRuns.estimateCost('agent-123', { confidenceLevel: 'high' });

// List recent runs
const runs = await evalRuns.list({ agentId: 'agent-123', limit: 10 });

Signals

import { signals } from '@thinkhive/sdk';

// List all signals
const allSignals = await signals.list();

// Create a custom behavioral signal
await signals.create('Escalation Request', 'negative', {
  type: 'keywords',
  keywords: ['speak to manager', 'escalate'],
});

// Get signal stats and trends
const stats = await signals.getStats({ agentId: 'agent-123' });
const trends = await signals.getTrends({ granularity: 'daily' });

Notifications

import { notifications } from '@thinkhive/sdk';

// Create an alert rule
await notifications.createRule({
  agentId: 'agent-123',
  name: 'High failure rate alert',
  eventType: 'failure_spike',
  condition: { threshold: 0.3 },
  channel: 'email',
  target: '[e-mail]',
});

// List notifications
const alerts = await notifications.list('agent-123', true);
// or: await notifications.listNotifications('agent-123', true);

Documents (RAG)

import { documents } from '@thinkhive/sdk';

// Upload a document
await documents.upload('agent-123', 'faq.txt', 'text/plain', 1024);

// List agent documents
const docs = await documents.list('agent-123');

// Delete a document
await documents.remove('agent-123', 'doc-456');

Shadow Tests

import { shadowTests } from '@thinkhive/sdk';

// Create a shadow test
await shadowTests.create({
  fixId: 'fix-456',
  agentId: 'agent-123',
  testName: 'Refund policy test',
  inputData: { message: 'How do I get a refund?' },
  expectedOutput: 'You can request a refund within 30 days.',
});

// List tests for an agent
const tests = await shadowTests.list('agent-123');

Sessions

import { sessions } from '@thinkhive/sdk';

// List conversation sessions
const allSessions = await sessions.list('agent-123', { limit: 20 });

// Get all traces in a session
const traces = await sessions.getTraces('session-789', 'agent-123');

Drift Detection

import { drift, hasDrift, getDriftSeverity } from '@thinkhive/sdk';

// Detect drift for an agent
const report = await drift.detect('agent-123');
if (hasDrift(report)) {
  console.log(`Drift severity: ${getDriftSeverity(report)}`);
}

// Detect drift across all agents
const allDrift = await drift.detectAll();

LLM Costs

import { llmCosts, formatCost } from '@thinkhive/sdk';

// Get cost summary
const summary = await llmCosts.summary({ period: '30d' });
// or: await llmCosts.getSummary({ period: '30d' });
console.log(`Total cost: ${formatCost(summary.totalCost)}`);

// Get per-agent breakdown
const breakdown = await llmCosts.getBreakdown('agent-123');

// Get optimization savings
const savings = await llmCosts.getSavings();

Error Handling

import { ThinkHiveError, RateLimitError } from '@thinkhive/sdk';

try {
  await runs.create({ /* ... */ });
} catch (error) {
  if (error instanceof RateLimitError) {
    console.log(`Retry after ${error.retryAfter}ms`);
  }
}

// Both forms work:
throw new RateLimitError(60000);
throw new RateLimitError('Custom message', 60000);

Upgrading

v4.2.0 → v4.2.2

Improvements in v4.2.2:

• RateLimitError now accepts both new RateLimitError(retryAfterMs) and new RateLimitError('message', retryAfterMs) signatures
• Claims helpers (isFact, isInference, isComputed) now accept both {claimType: 'observed'} (API format) and {type: 'fact'} (shorthand)
• groupClaimsByType returns both API names (observed/inferred/computed) and shorthand names (fact/inference)
• calculateBrierScore now accepts both single prediction (probability, outcome) and batch ([{predicted, actual}]) forms
• calculateECE now accepts pre-computed bucket objects [{averageConfidence, accuracy, count}] in addition to raw predictions
• isWellCalibrated accepts both a raw number and {brierScore: number} object
• calculatePassAtK now supports both (passRate, k) and (n, c, k) calling conventions
• isReliableEvaluation accepts both a CriterionAnalysis object and a raw reliability score
• Rule helpers (createRegexRule, createContainsRule, createLengthRule, createJsonSchemaRule) now return named rule objects with {type, name, config} structure
• allRulesPassed and getFailedRules accept any {passed: boolean} objects, not just RuleResult[]
• aggregateWorst / aggregateAverage now accept simple number[] arrays in addition to TurnEvaluation[]
• Method aliases added for cross-SDK consistency:
  • signals.delete() (alias for remove())
  • notifications.list() (alias for listNotifications())
  • llmCosts.summary() (alias for getSummary())
  • qualityMetrics.evaluate() (alias for evaluateRag())
  • guardrails.evaluate() (alias for scan())
  • customerContext.capture() (alias for captureSnapshot())

v4.1.0 → v4.2.0

New in v4.2.0:

• evalRuns — create, list, and manage evaluation runs programmatically
• signals — behavioral signal CRUD with stats and trends
• notifications — alert rule configuration and notification management
• documents — agent RAG document upload/list/delete
• shadowTests — shadow test creation and management
• sessions — trace session grouping and querying
• drift — model/behavior drift detection with helpers
• llmCosts — LLM usage cost tracking and optimization savings

v4.0 → v4.1.0

• roiAnalytics now includes V3 methods: getConfig(), createConfig(), updateConfig(), configVersions(), calculateV3(), trendV3()
• linking now includes: autoLink(), stats(), generateMarker()
• GET /roi/config auto-provisions a default configuration if none exists
• ROI date range limit increased to 92 days

License

MIT License - see LICENSE for details.