@pyai/twilio

@pyai/twilio

Turn a phone number into a production voice agent, in one line of code.

@pyai/twilio bridges a live Twilio call to a PyAI Omni voice agent. Point a Twilio number at a tiny server, hand the Media Streams WebSocket to OmniAgent.bridge(...), and your caller is instantly talking to a real listen → think → speak agent, with sub-500 ms turn-taking, natural barge-in, DTMF, and live transfer-to-human. You write zero audio or DSP code.

import { OmniAgent } from "@pyai/twilio";

OmniAgent.bridge(twilioWebSocket, {
  apiKey: process.env.PYAI_API_KEY!,
  voice: "stock_emma_en_gb",
  persona: "You are a warm, concise support agent for Acme.",
  knowledge: async (q) => myVectorSearch(q), // optional per‑turn grounding
  // sessionLabel: "support-line", // optional opaque tag echoed to your kb_endpoint
});

That single call is a complete phone agent.

Why Omni, one SDK for the whole voice agent

Most voice stacks are a fragile chain of four vendors: speech-to-text → an LLM → text-to-speech → a telephony bridge, each with its own latency, billing, and failure mode. Omni collapses all of it into one realtime engine behind one WebSocket and one API key:

• End-to-end, not glued together. Transcription, the reasoning brain, retrieval, and synthesis run as a single speech-to-speech model, so you ship an agent, not an integration project.
• Built for the phone. ~390 ms median turn-taking with real barge-in, tuned for 8 kHz call audio. Conversations feel human, not walkie-talkie.
• Grounded and capable. Bind your own knowledge per turn and (roadmap) call your tools, so the agent answers from your content and takes real actions.
• One predictable bill. Omni is all-in, one rate for the whole voice loop, no four-vendor passthrough, no surprise invoice. See current rates at pyai.com/pricing.
• Open and portable. Standard WebSocket, opaque API key, OpenAI-compatible surfaces. Nothing proprietary to lock you in.

@pyai/twilio is the last mile: it makes Omni answer a real phone number.

🎁 Get $50 in free credit when you sign up

Create a key at console.pyai.com, email only, no credit card, no sales call. That's plenty of live Omni calls to build and test with, free. Your key works on every surface instantly.

What the bridge handles for you

The bridge owns the entire audio path so you never touch a codec or a resampler:

• Codec & rate. Twilio speaks G.711 mu‑law @ 8 kHz; Omni speaks PCM16. The bridge transcodes both ways and resamples with a proper anti‑aliased polyphase filter (not naive decimation), 8 kHz ⇄ 24 kHz is a clean 3:1, 8 kHz ⇄ 16 kHz is 2:1. Run Omni at omniRate: 8000 to skip resampling entirely.
• Handshake. Opens the Omni WebSocket with subprotocol auth and sends the post‑handshake configure frame (voice, persona, optional knowledge endpoint).
• Barge‑in. When the caller talks over the agent, Omni's barge‑in event is relayed to Twilio as a clear, so the agent stops mid‑word.
• DTMF. Caller key presses are forwarded to the agent.
• Knowledge. Your optional knowledge(query) callback is invoked on each finalized caller turn; whatever you return is pushed to the agent as grounding.
• Call control. The bridge demuxes the agent's engine-native verbs, transfer_to_human, end_call, send_dtmf, play_hold, collect. end_call hangs up cleanly; pass twilioControl (your Twilio REST creds) and the bridge also performs the transfer (and a guaranteed hangup) by callSid. Without it, each verb surfaces via an on* callback for you to act on.

Install

npm install @pyai/twilio

Requires Node ≥ 22 (uses the ws WebSocket client; everything else is built‑in). You'll need a PyAI key (pyai_live_… or a free pyai_test_… sandbox key), grab one with $50 free credit, and a Twilio number with Media Streams.

1. TwiML: open a bidirectional stream

When Twilio receives a call it fetches TwiML from your webhook. Use <Connect><Stream> (not <Start><Stream>) so the socket is two‑way and the agent can speak back:

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://your-host.example.com/media" />
  </Connect>
</Response>

connectStreamTwiML("wss://your-host/media") builds exactly this string for you. Set the number's A call comes in webhook to https://your-host/voice.

2. A ~10‑line Node server

import Fastify from "fastify";
import websocket from "@fastify/websocket";
import { OmniAgent, connectStreamTwiML } from "@pyai/twilio";

const app = Fastify();
await app.register(websocket);

app.post("/voice", (req, reply) =>
  reply.type("text/xml").send(connectStreamTwiML(`wss://${req.headers.host}/media`)));

app.get("/media", { websocket: true }, (twilioWS) =>
  OmniAgent.bridge(twilioWS, { apiKey: process.env.PYAI_API_KEY }));

await app.listen({ port: 8080, host: "0.0.0.0" });

Tunnel it (ngrok http 8080), point your Twilio number at https://<host>/voice, and call the number. See examples/twilio-omni-voice-agent for a complete, runnable version.

API

OmniAgent.bridge(twilioWS, options) → BridgeHandle

twilioWS is the Twilio Media Streams socket (the Node ws socket your framework hands you). Options:

Option	Type	Notes
apiKey	string	Required. pyai_live_… / pyai_test_…. Opaque, never parsed.
sessionLabel	string	Optional opaque tag echoed to your kb_endpoint so you can branch per call. Nothing to pre-create; omit if unneeded.
voice	string	Voice id (stock / clone / designed).
persona	string	System prompt / role for the agent.
knowledge	(q) => facts | Promise<facts>	Per‑turn grounding callback.
kbEndpoint / kbToken	string	Customer‑hosted endpoint the engine pulls per turn.
omniRate	8000 | 16000 | 24000	Omni session rate. Default 24000. 8000 = no resampling.
baseURL	string	Defaults to https://api.pyai.com.
twilioControl	{ accountSid, authToken }	Optional Twilio REST creds so the bridge performs transfer_to_human + end_call by callSid. Omit to keep it audio-only and handle the verbs yourself.
transferDestination	string	Fallback transfer target if an engine frame doesn't carry one.
onTranscript / onTransfer / onError / onClose	callbacks	Observability + lifecycle.
onSendDtmf / onPlayHold / onCollect / onEndCall	callbacks	Engine-native call-control verbs, surfaced for you to act on.

Returns a handle with close() and the underlying omni client.

The bridge connects only to /v1/omni?format=pcm16&rate=…. sessionLabel is the only connect-time tag; there is no model selector or agent-id alias.

The Omni side is strict native framing: server audio, transcript, and control messages are binary 0x01/0x02/0x03 frames. Control JSON is keyed on event; live 0x02 bodies are plain UTF-8 caller-text deltas. parseOmniTranscriptBody normalizes those deltas and retains bounded direct-JSON support for older bridges. The connect rate is caller input: 16/24 kHz sessions receive 24 kHz agent audio, while 8 kHz sessions receive 8 kHz; hello.audio_out is authoritative.

Performing call control

Pass your Twilio REST credentials and the agent's call-control decisions become real carrier actions, no extra code:

OmniAgent.bridge(twilioWS, {
  apiKey: process.env.PYAI_API_KEY!,
  persona: "You are Acme support. Escalate to a human on request.",
  twilioControl: {
    accountSid: process.env.TWILIO_ACCOUNT_SID!,
    authToken: process.env.TWILIO_AUTH_TOKEN!,
  },
  transferDestination: "[phone]", // fallback if the agent's config has none
});

When the agent calls transfer_to_human, the bridge redirects the live call to the resolved destination; on end_call it hangs the call up. (transfer is a blind redirect in this version; warm/announced transfer is on the roadmap.)

Lower‑level building blocks

Also exported for custom pipelines (and fully unit‑tested):

import {
  muLawEncode, muLawDecode,        // G.711 companding
  Resampler, makeResampler,         // anti-aliased rational resampler
  pcm16ToBytes, bytesToPcm16,       // little-endian PCM16 <-> bytes
  OmniClient, omniWsUrl,            // raw Omni realtime client
  parseTwilioMessage, twilioMedia,  // Twilio frame parse/build
  twilioClear, connectStreamTwiML,
  dialTwiml, redirectCall, hangupCall, // Twilio REST call control
} from "@pyai/twilio";

Notes & limits

• Barge‑in / event names. A couple of Omni event field names aren't yet byte‑pinned across the protocol doc and the Twilio guide; the client accepts both spellings and isolates them in src/omni.ts so they're trivial to update.
• Transfer to a human. With twilioControl set, the bridge performs the redirect for you via Twilio's REST API (using your Twilio credentials, not PyAI's). Without it, the transfer event still surfaces via onTransfer(info) so you can redirect the call yourself. The redirect is a blind transfer (<Dial>); warm/announced transfer (a conference bridge) is on the roadmap.
• send_dtmf on Media Streams. Twilio's Media Streams can't inject outbound DTMF without ending the stream, so the bridge surfaces send_dtmf via onSendDtmf rather than performing it; handle it on a transport that supports mid-call DTMF if you need it.
• Outbound framing. Agent audio is reframed into ~20 ms (160‑byte) mu‑law frames tagged with the streamSid, which is what Twilio's jitter buffer likes.

Use with MCP (AI coding agents)

Building your phone agent with an AI coding agent (Cursor, Claude Code, Codex)? Add the PyAI MCP server (@pyai/mcp) so it can mint a free key and explore PyAI directly:

// .cursor/mcp.json  ·  or:  claude mcp add pyai -- npx -y @pyai/mcp
{ "mcpServers": { "pyai": { "command": "npx", "args": ["-y", "@pyai/mcp"] } } }

The MCP tools cover key minting + REST (TTS/STT/voices); the Twilio↔Omni bridge itself is this SDK. Full setup: the mcp-quickstart example.

Develop

npm install
npm test          # node --test, mock Twilio + mock Omni sockets (no network)
npm run typecheck
npm run build     # emits dist/

MIT licensed.