@helicone/export

Helicone Data Export Tool

A robust command-line tool to export request/response data from Helicone's API. This tool allows you to fetch and export your Helicone request history in various formats, with advanced features for reliability and monitoring.

✨ Key Features

Core Features

• 📦 Export data in multiple formats (JSON, JSONL, CSV)
• 📅 Date range filtering
• 🔄 Automatic pagination handling
• 📄 Full request/response body inclusion (optional)
• 🚫 Automatic filtering of large streamed_data fields

Advanced Features (NEW!)

• 💾 Auto-recovery from crashes - Checkpoint system saves progress automatically
• 🔁 Retry logic with exponential backoff - Handles transient failures gracefully
• 🛑 Graceful shutdown - Ctrl+C saves progress for later resume
• 📊 Progress tracking - Real-time progress bar with ETA
• 🔍 Multiple log levels - quiet, normal, or verbose output
• ✅ Pre-flight validation - Checks API key, permissions, and disk space
• ⚡ Configurable batch sizes and retry attempts
• 🔒 Overwrite protection - Prompts before overwriting existing files
• 🏷️ Property filtering - Filter exports by custom properties

Prerequisites

• Node.js v18.0.0 or later (for native fetch API)
• A Helicone API key

Installation

Option 1: NPX (Recommended - No Installation Required)

Use the package directly without installing:

npx @helicone/export [options]

Option 2: Global Installation

Install once and use anywhere:

npm install -g @helicone/export
helicone-export [options]

Option 3: Local Project Installation

Add to your project:

npm install @helicone/export
npx helicone-export [options]

Option 4: Development (From Source)

Clone and build from source:

git clone https://github.com/Helicone/helicone.git
cd helicone/examples/export/typescript
npm install
npm run build
node dist/cli.js [options]

Setup

Set your Helicone API key as an environment variable:

export HELICONE_API_KEY="your-helicone-api-key"

Or prefix each command:

HELICONE_API_KEY="your-key" npx @helicone/export [options]

Usage

npx @helicone/export [options]

Or if globally installed:

helicone-export [options]

Core Options

Option	Description	Default
--start-date <date>	Start date (YYYY-MM-DD or ISO string)	30 days ago
--end-date <date>	End date (YYYY-MM-DD or ISO string)	now
--limit <number>	Maximum number of records to fetch	unlimited
--format <format>	Output format: json, jsonl, or csv	jsonl
--include-body	Include full request/response bodies	false
--output, -o <path>	Custom output file path	output.{format}
--property, -p <key=value>	Filter by property (can use multiple times)	-
--help, -h	Show help message and exit	-

Advanced Options

Option	Description	Default
--log-level <level>	Log level: quiet, normal, or verbose	normal
--max-retries <number>	Maximum retry attempts for failed requests	5
--batch-size <number>	Batch size for API requests	1000
--clean-state	Remove checkpoint and start fresh export	-
--resume	Explicitly resume from checkpoint	-

Examples

Basic Usage

• Export last 30 days of data (default behavior):

npx @helicone/export

• Export specific date range in CSV format:

npx @helicone/export --start-date 2024-01-01 --end-date 2024-02-01 --format csv

• Export with full request/response bodies:

npx @helicone/export --limit 100 --include-body

• Custom output file:

npx @helicone/export --output my-export.jsonl

• Filter by property (e.g., only export LlamaCoder requests):

npx @helicone/export --property appname=LlamaCoder

• Multiple property filters:

npx @helicone/export --property appname=LlamaCoder --property environment=production

Advanced Usage

• Quiet mode for automation:

npx @helicone/export --log-level quiet --limit 10000

• Verbose logging for debugging:

npx @helicone/export --log-level verbose --max-retries 10

• Large export with custom batch size:

npx @helicone/export --limit 50000 --batch-size 500

• Clean state and start fresh:

npx @helicone/export --clean-state

• Filter by property with other options:

npx @helicone/export --property appname=LlamaCoder --format csv --limit 5000 --include-body

Recovery Scenarios

• After a crash (automatic resume prompt):

npx @helicone/export
# Will detect checkpoint and ask: "Resume from checkpoint? (y/n)"

• Force resume from checkpoint:

npx @helicone/export --resume

• Cancel and save progress (during export):

Press Ctrl+C during export
# Progress is saved automatically
# Run the same command again to resume

Output Formats

JSONL (Default)

Each line is a complete JSON object representing one record. Best for large datasets and streaming processing.

JSON

A single JSON array containing all records. Includes pretty-printing for better readability.

CSV

Tabular format with the following columns:

• response_id
• response_created_at
• response_status
• request_created_at
• request_body
• request_properties
• request_user_id
• model
• prompt_tokens
• completion_tokens
• latency
• cost

How It Works

Auto-Recovery System

The tool automatically saves checkpoints after each batch of records:

• Checkpoint file (.helicone-export-state.json) tracks:

  • Current offset in the export
  • Total records processed
  • Output file path
  • Export configuration

• On restart, the tool:

  • Detects existing checkpoint
  • Validates it matches current configuration
  • Prompts user to resume or start fresh

• On crash/interrupt:

  • Checkpoint is saved before exit
  • Output file is properly closed
  • No data loss occurs

Retry Logic

When API requests fail, the tool automatically retries with exponential backoff:

• Attempt 1: Wait 1 second
• Attempt 2: Wait 2 seconds
• Attempt 3: Wait 4 seconds
• Attempt 4: Wait 8 seconds
• Attempt 5: Wait 16 seconds

Special handling for rate limits (429):

• Respects Retry-After header if present
• Otherwise uses exponential backoff

Progress Tracking

Three log levels available:

• quiet: Only start/complete/error messages
• normal: Progress bar with ETA and records/sec
• verbose: Detailed logs of each API call and retry attempt

Example progress bar:

[==================>           ] 62% (6,234/10,000) ETA: 3m 45s | 12.3 rec/s

Rate Limiting

The tool implements intelligent rate limiting:

• Processes records in configurable batches (default 1000)
• Fetches signed bodies in chunks of 10
• Adds delays between chunks to avoid API limits
• Automatically handles 429 rate limit responses

Error Handling

Comprehensive error handling:

• ✅ Pre-flight validation (API key, permissions, disk space)
• ✅ Validates command-line arguments
• ✅ Handles API errors with retry logic
• ✅ Distinguishes retryable vs fatal errors
• ✅ Provides clear, actionable error messages
• ✅ Ensures proper cleanup of file streams and signal handlers

Architecture

The code is structured into specialized classes:

• CheckpointManager: Handles state persistence and recovery
• ProgressTracker: Manages logging and progress display
• HeliconeClient: API client with retry logic
• ExportWriter: Handles file writing for different formats

Benefits:

• Strong TypeScript typing
• Separation of concerns
• Easy to test and maintain
• Extensible for new features

License

MIT