@pyai/sdk

@pyai/sdk

Official TypeScript/JavaScript SDK for PyAI, the all-in-one voice AI platform: lightning-fast speech-to-text, ultra-realistic text-to-speech, end-to-end realtime voice agents, and automatic call compliance. Zero dependencies; runs in the browser and Node 18+.

PyAI products

• Hear, Lightning-fast, telephony-native speech-to-text. Whisper-compatible transcription tuned for real phone-call audio, with live streaming partials so your app reacts mid-sentence, plus async batch transcription for big archives. POST /v1/audio/transcriptions
• Speak, Ultra-realistic text-to-speech that starts speaking in tens of milliseconds. Stream lifelike, expressive voices, choose from 144 stock voices, or clone any voice instantly, for free. POST /v1/audio/speech
• Omni (flagship), One API for a complete, end-to-end voice AI agent. A single WebSocket where your agent listens, thinks, and speaks, grounded in your knowledge bases and tools, with human-like turn-taking and instant barge-in, no STT, LLM, or TTS to stitch together yourself. wss://api.pyai.com/v1/omni
• Trace (flagship), The compliance API that keeps your AI agents safe. Trace automatically checks every call for HIPAA, TCPA, and PII risks (plus your own brand-voice rules), flags the exact rule broken, redacts sensitive data, and seals each call with a tamper-evident audit trail, so a risky conversation never slips through. GET /v1/trace/interactions
• Cue, reserved turn-detection and grounding fields on Hear streaming. Grounding is not active on the serving route yet.
• AMD, Answering-machine detection that tells your dialer who or what answered, human, voicemail, IVR, iPhone/Google screening, dead number, fax, in a fraction of Twilio's dead-air dwell, with the reason. A one-line-TwiML Twilio Media Streams drop-in; billed per answered call (first 5,000/month free). wss://api.pyai.com/v1/amd/stream
• Telephony, Instant managed phone numbers for your voice agents. Provision a US number and route live calls straight into an Omni agent, no carrier contracts, no telephony glue. POST /v1/telephony/numbers

The contract is https://api.pyai.com/openapi.json. This SDK wraps it ergonomically with typed errors, automatic retries, and a realtime helper.

Install

npm install @pyai/sdk

Quickstart

import PyAI from "@pyai/sdk";

const pyai = new PyAI({ apiKey: process.env.PYAI_API_KEY! });

// Text-to-speech
const audio = await pyai.audio.speech({ input: "Hello from PyAI.", voice: "stock_emma_en_gb" });
await Bun.write?.("hello.wav", audio); // or fs.writeFile in Node

// Stream PCM bytes as they arrive. Configure your player for mono PCM16LE
// at 24 kHz. MP3 and Opus are buffered server-side, even with speechStream().
const stream = await pyai.audio.speechStream({
  input: "Hello from PyAI.", voice: "stock_emma_en_gb",
  response_format: "pcm", sample_rate: 24000,
});
for await (const chunk of stream) writeToSpeakerOrResponse(chunk);

// Voices
const { data: voices } = await pyai.voices.list({ gender: "female" });

// Async transcription (safe retry with an idempotency key)
const job = await pyai.transcriptionJobs.create(
  { audio_url: "https://example.com/call.wav", diarize: true },
  { idempotencyKey: crypto.randomUUID() },
);
const done = await pyai.transcriptionJobs.get(job.job_id);

Use with MCP (AI coding agents)

Building this SDK with an AI coding agent (Cursor, Claude Code, Codex)? Add the PyAI MCP server (@pyai/mcp) so the agent can mint a free key and call PyAI as tools, no endpoint guessing, no human setup step:

// .cursor/mcp.json  ·  or:  claude mcp add pyai -- npx -y @pyai/mcp
{ "mcpServers": { "pyai": { "command": "npx", "args": ["-y", "@pyai/mcp"] } } }

With no key set, the server exposes create_sandbox_key, calls it, and adopts the minted key for the session, then get_started, list_voices, synthesize_speech, and the transcription tools work immediately. Full setup + a runnable client: the mcp-quickstart example.

Realtime (Omni)

omni.connect() opens an agentic-voice session and hides the wire protocol, including its frame-key asymmetry (your control frames are keyed on type, the server's frames are keyed on event). It sends a type-keyed configure the instant the socket opens and routes server frames to typed callbacks, so you can't trip the #1 Omni integration bug (a hand-rolled {"event":"configure"} is acked but silently dropped, giving you a connected session with zero turns):

Requires SDK 0.8.0 for --template omni. The runnable starter uses ESM, Node 20.19+ with an explicit ws transport, or Node 22+. MCP 0.5.0 requires Node 22+. The SDK's REST client continues to support Node 18+.

npm install @pyai/sdk@0.8.0
npx pyai init voice-demo --template omni
cd voice-demo
npm install

Initialization is offline; installing dependencies and running main.mjs are separate steps. For an existing checkout of this example, run npm install in its directory instead.

Inject PYAI_API_KEY through your environment, then supply a 24 kHz PCM16 mono WAV (at most 20 seconds) saying “Please look up the office opening time.”

node main.mjs caller.wav
# Optional interruption clip, sent while reply audio is queued:
node main.mjs caller.wav interruption.wav

The generated project contains the complete runnable source, including its Node 20 WebSocket import, WAV reader, one paced input stream, read-only tool, and bounded capture. It waits for configuration and greeting playback to drain. It sends caller PCM or silence in each slot, never overlapping silence timers.

Running it consumes Omni and Hear usage under the injected key. It saves a private report and WAV files. The report separates received audio from an answer recovered by Hear from captured bytes; synthesis text alone cannot pass. The playback sink is simulated. Interruption clears that queue; physical speaker playback remains a separate test. Capture uses two seconds of quiet after the queue drains, because the protocol has no reply-end marker.

This section and the CLI project are generated from the same tested example. Release readiness: https://pyai.com/agents/bot-release.json

Live 0x02 transcript bodies are plain UTF-8 caller-text deltas, not JSON. onTranscript receives the normalized { event:"transcript", role:"user", text, final:false, mode:"delta" } shape. Coalesce successive deltas for the current caller turn. Bounded direct JSON bodies remain accepted for older bridges.

Since version 0.5.1, the engine's four-field assistant synthesis advisory on 0x03 also reaches onTranscript with role: "assistant", final: true, and mode: "replace". This is text submitted for synthesis; it does not prove that playback completed. Caller transcripts continue to require the 0x02 carrier.

rate configures caller input. A rate: 16000 session still receives 24 kHz agent audio; rate: 8000 receives 8 kHz. Read hello.audio_out before playback. Omni has no commit frame—keep streaming silence during caller pauses. Use one paced input stream: send caller PCM when available, otherwise silence. Pause any separate silence timer while microphone or fixture frames are being sent; never interleave extra silence with active input.

For a client-executed lookup, declare side_effect: "read" in its tool definition:

const officeHoursTool = {
  name: "lookup_office_hours",
  description: "Read the office opening time.",
  side_effect: "read",
  parameters: { type: "object", properties: {} },
};
// Include officeHoursTool in configure.tools. In onToolCall, return the
// actual lookup result using session.toolResult(frame.call_id, { result }).

An omitted side_effect is treated as an action. Action results need a positive completion acknowledgement, for example { ok: true, receipt_id: actualReceiptId }, returned inside toolResult's result. Send that only after the operation has completed; a queued request or transport acknowledgement is insufficient. On failure, return { error: "Operation failed" } instead of claiming success.

From the browser, mint an ephemeral token server-side with pyai.omni.createSession({ allowedOrigins }) and pass it as token so the page never holds a secret key:

const omni = pyai.omni.connect({ token: session.token, configure: { voice_id, persona } });

Omni connects only to wss://api.pyai.com/v1/omni and is zero-state, no agent to create. sessionLabel is an optional opaque tag (never required). Need the raw socket? Use pyai.realtimeURL({ sessionLabel }) with pyai.realtimeSubprotocol() (or pyai.connectRealtime()). The raw URL helper accepts canonical format, rate, and api_key query parameters; retired connect aliases, model selectors, and token query names throw instead of being translated.

Streaming speech-to-text (Hear)

transcriptions.stream() hides the WebSocket frame protocol behind callbacks. It opens wss://api.pyai.com/v1/audio/transcriptions/stream?protocol=pyai-hear-v1 (key carried as the WS subprotocol, so it works in the browser), routes the wire frames to onConfigAck/onPartial/onFinal/onError, and gives you sendAudio, configureEndpointing(), commit(), and close():

const hear = pyai.audio.transcriptions.stream({
  sampleRate: 16000,
  endpointingMs: 800, // minimum trailing pause; may wait up to max(800, 1500) ms
  vocabulary: ["Nguyen", "SKU-99"],
  onConfigAck: (ack) => {
    if (ack.warnings.length) throw new Error(JSON.stringify(ack.warnings));
  },
  onPartial: (f) => console.log("…", f.text),
  onFinal: (f) => console.log("✓", f.text, f.endpoint_reason),
  onError: (e) => console.error(e),
});

micChunks.on("data", (pcm16) => hear.sendAudio(pcm16)); // keep sending silence through pauses
hear.configureEndpointing(950);                         // update without reconnecting
vad.on("end", () => hear.commit());                     // optional forced final
// hear.close() also flushes a final for any buffered audio

Streaming uses up to five sanitized vocabulary terms. To store organization suggestions, use a key with hear:configure and set explicit activation profiles first:

await pyai.hear.vocabulary.set({
  terms: ["Nguyen", "SKU-99"],
  enabledFor: ["batch", "hear_stream"],
});

Request-level terms come first. Stored suggestions fill remaining slots up to five. The effective list is fixed when a stream opens or a batch job is created. Organization Hear terms are not used by Omni and are not populated automatically from CRM or dialer data. A managed Agent can opt in with its own list:

const agent = await pyai.agents.create({
  name: "Front desk",
  vocabulary: ["Nguyen", "Acme Dental", "SKU-99"],
});

await pyai.agents.update(agent.agent_id, { vocabulary: [] });

The Agent list is sanitized to at most five effective terms and fixed when a new session starts. An empty list turns the feature off.

Frame types, WS close codes, and error codes are exported as named constants so you never hardcode a magic string:

import { HearFrameType, WSCloseCode, ErrorCode } from "@pyai/sdk";
HearFrameType.SpeechFinal; // "speech_final"
WSCloseCode.OverCapacity;  // 4429
ErrorCode.CreditExhausted; // "credit_exhausted"

Set grounding: true to turn the stream into Cue (turn detection + KB context): the SDK sends the grounding config on open and final/speech_final frames then carry a grounding array of top KB passages.

In Node, pass a WebSocket implementation if there's no global one: transcriptions.stream({ webSocket: (await import("ws")).WebSocket }).

Speak audio formats (incl. telephony G.711)

audio.speech encodes server-side into any of eight formats via response_format, the audio comes back already in the shape you need, so telephony callers can drop the hand-rolled resampler + μ-law encoder entirely:

// Twilio/SIP-ready in one param: raw 8 kHz mono μ-law, no client-side DSP.
const ulaw = await pyai.audio.speech({
  input: "Your appointment is confirmed.",
  voice: "stock_emma_en_gb",
  response_format: "g711_ulaw", // -> audio/basic, forced 8 kHz
});
// base64-encode `ulaw` straight into a Twilio media frame.

response_format	sample rates (Hz)	Content-Type
wav (default)	8000 / 16000 / 24000 / 48000	audio/wav
mp3	8000 / 16000 / 24000 / 48000	audio/mpeg
opus	8000 / 16000 / 24000 / 48000	audio/ogg
aac	8000 / 16000 / 24000 / 48000	audio/aac
flac	8000 / 16000 / 24000 / 48000	audio/flac
pcm (raw int16 LE, no header)	8000 / 16000 / 24000 / 48000	audio/pcm
g711_ulaw	8000 (forced)	audio/basic
g711_alaw	8000 (forced)	audio/basic

sample_rate is optional, omit it for the engine's native 24 kHz (g711_* is always 8 kHz). The set is typed (SpeechFormat) and exported as SPEECH_FORMATS / SPEECH_SAMPLE_RATES for dropdowns and validation. Any other value is a 400 unsupported_format; omit response_format for the default wav.

See examples/speak-telephony-formats for the full before/after: ~120 lines of resampler + μ-law replaced by one param, with Node (@pyai/twilio), Python, and raw-curl snippets.

AMD (answering-machine detection)

Already on Twilio? The usual path is one line of TwiML pointing the call's media at PyAI, no SDK needed. The answered_by_twilio field maps to Twilio's exact AnsweredBy enum, so your routing logic doesn't change:

<Response><Start>
  <Stream url="wss://api.pyai.com/v1/amd/stream">
    <Parameter name="api_key" value="YOUR_PYAI_KEY"/>
    <Parameter name="aggressiveness" value="0.25"/>
    <Parameter name="webhook" value="https://you/amd-events"/>
  </Stream>
</Start>
  <!-- Standalone test: replace this pause with your existing call flow. -->
  <Pause length="30"/>
</Response>

AMD listens only: use <Start><Stream> so subsequent TwiML can run. <Connect><Stream> blocks that call flow until the socket closes. Keep a following TwiML verb: without one Twilio disconnects the call. The pause above is only for a standalone test; replace it with your existing call flow. Read the AMD result from the webhook.

For a three-second elapsed decision budget, add <Parameter name="decision_timeout_ms" value="3000"/> inside <Stream>; use 5000 for five seconds (supported range: 1000–15000 ms). An earlier decision returns immediately. At the cutoff, inconclusive evidence returns unknown with rule_id: "decision_timeout"; shorter budgets can increase unknowns. The clock starts at the accepted authenticated start, and webhook transport takes additional time. Omitting the parameter preserves existing behavior. decision_ms measures processed audio; decision_elapsed_ms measures elapsed result-preparation time when a timeout is set.

Additional string parameters such as lead_id return under custom_parameters in socket results, both callbacks, and stored call details. See the AMD guide for limits, timing fields, and fallback handling. These options belong in TwiML or start.customParameters, not account configuration.

(The key rides a <Parameter> because Twilio strips query strings from the <Stream> URL; PyAI verifies it from the start frame before processing any audio.)

From code, set the operating point and read decisions back:

// One aggressiveness dial: near 0 = human-safe, near 1 = fire "machine" fast.
await pyai.amd.config.set({ aggressiveness: 0.25, webhookUrl: "https://you/amd-events" });

const { data: decisions } = await pyai.amd.calls.list({ sessionLabel: "sales" });
const decision = await pyai.amd.calls.get("C_123");
// decision.answered_by = "human" | "voicemail" | "screening" | "sit_invalid" | ...
// decision.answered_by_twilio = "human" | "machine_start" | ...  (Twilio parity)
// decision.reason = "machine phrase: 'leave a message' @1.2s"

// Server-side helper if you fork the media yourself (Twilio Media Streams wire):
const stream = pyai.amd.stream({
  aggressiveness: 0.25,
  onDecision: (d) => console.log(d.answered_by, d.decision_ms, d.reason),
});

Billed per answered call, first 5,000 answered calls/month free, then $0.004/call; free when bundled with PyAI telephony/Omni.

More APIs: clones, telephony, trace

// Voice clones (Speak)
const { data: clones } = await pyai.clones.list();
const clone = await pyai.clones.create({ name: "Brand VO", file: refAudioBlob });
await pyai.clones.delete(clone.id);

// Managed phone numbers (Telephony)
const { data: avail } = await pyai.telephony.numbers.available({ areaCode: "415" });
const num = await pyai.telephony.numbers.buy({ phone_number: avail[0]!.phone_number, agent_id: "agent_123" });
await pyai.telephony.numbers.assign(num.id, "agent_123");
await pyai.telephony.numbers.release(num.id);

// Compliance (Trace)
const { data: calls } = await pyai.trace.interactions.list({ verdict: "FAIL" });
const detail = await pyai.trace.interactions.get(calls[0]!.id);
await pyai.trace.config.set({ agent_id: "agent_123", enabled: true });
const exposure = await pyai.trace.exposure(30);

// Per-call eval scorecard (timeline + quality metrics). These are additive and
// forward-compatible, present once the engine emits them, so reading them is
// always safe (the timeline reader returns [] until then).
const timeline = await pyai.trace.callTimeline(detail.id); // TraceTimelineTurn[]
const quality = detail.quality_metrics;                    // { wer?, ttfb_ms?, turn_p95_ms?, vaqi?, … }

Reproducible runs (evals)

audio.speech and audio.transcriptions.create take optional seed and temperature for deterministic eval runs. They're forward-compatible, honored once the engine supports them and otherwise ignored, so it's always safe to send:

await pyai.audio.speech({ input: "Hello", voice: "stock_emma_en_gb", seed: 42, temperature: 0 });
await pyai.audio.transcriptions.create({ file: wavBlob, seed: 42 });

Errors

Failures throw PyAIError with a stable code (branch on it, not the message):

import { PyAIError } from "@pyai/sdk";

try {
  await pyai.audio.speech({ input: "hi" });
} catch (err) {
  if (err instanceof PyAIError && err.code === "credit_exhausted") {
    // out of prepaid credit, add credit or use a sandbox key
  }
}

Common codes: unauthorized, forbidden, credit_exhausted, rate_limit_exceeded, concurrency_limit_exceeded, idempotency_conflict. 429/5xx are retried automatically (honoring Retry-After); tune with new PyAI({ apiKey, maxRetries }).

CLI (pyai)

The package provides a pyai executable for engineers, CI, and coding agents. Install version 0.5.0 with npm install -g @pyai/sdk@0.5.0, then use pyai login for browser sign-in. Environment API keys work for unattended automation.

pyai speak "Your appointment is confirmed." -o confirmation.wav
pyai hear confirmation.wav --text-only
pyai login -p work
pyai whoami -j
pyai schema agents create -j
pyai agents create --data @agent.json --dry-run -j
pyai recipes speak
pyai init voice-project --template agent

The detailed CLI handbook covers installation, profiles, speech, transcription, Dub submission through download, Cast, resource configuration, JSON and stdin contracts, exit codes, and troubleshooting. For coding agents, use the raw integration guide and live OpenAPI contract.

pyai doctor checks the key, catalogs, and a Speak-to-Hear round trip; pyai smoke checks catalogs and synthesis. Both make real API calls.

Develop

npm install
npm test         # node --test, fetch injected (no network)
npm run build    # emits dist/ (incl. the pyai CLI bin)

Network recovery

REST automatically retries GET/HEAD only: transient connection failures with a recognized cause, and HTTP 429/500/502/503/504. maxRetries defaults to 2; retryWindowMs defaults to 10000. Backoff uses jitter and respects both numeric and HTTP-date Retry-After. A delay outside the remaining window returns the failure instead of retrying early. Original transport exceptions retain their cause. An unclassified browser TypeError is surfaced without retry.

The window bounds admission of retries, not an in-flight request or response body. Pass a caller-owned signal (for example AbortSignal.timeout(10000) where supported) to bound the REST operation and cancel backoff. A signal belongs to its deadline: create a new client/signal for a later independent operation. Custom fetch implementations must honor the signal. Response-body failures after successful headers are surfaced; the SDK does not replay a partially read body.

Writes, including speech and job creation, are no longer automatically retried on server errors. The server may already have acted even when no response audio arrived. Where the endpoint documents idempotency, applications can explicitly retry the identical operation with the same supported idempotency key. An arbitrary header does not make every endpoint safe to replay.

For an explicit Omni restoration attempt, use omni.connect({ resumeToken, resumeTimeoutMs: 5000, ... }) with the single-use token from the previous session_started. The SDK waits for session_started.resumed === true before sending automatic configuration or delivering media/tools. Sending audio or controls earlier throws. Rejection or a missing acknowledgement closes with resume_failed; it cannot silently become a new conversation. onOpen only means the socket opened. After successful restoration, follow the normal configured readiness rules before sending microphone audio. Fresh connections are unchanged. The same guard applies to query.resume_token.

This is an explicit, bounded restoration attempt, not automatic reconnection. Do not loop a consumed token, replay unacknowledged audio, or replay tool actions. Store rotated tokens privately; do not log connection URLs containing them. Clear queued playback after a disconnect and tell the caller when speech was lost. Conversation-state restoration does not guarantee recovery of audio sent immediately before a disconnect. Python currently provides URL helpers rather than this WebSocket lifecycle guard.