@convai/web-sdk
Advanced tools
Build web apps with lifelike AI characters. The Convai Web SDK gives you real-time voice, lipsync, emotions, and dynamic context — with first-class support for React and vanilla TypeScript.
Real-time conversational AI characters for the web.
TypeScript-first SDK for embedding Convai AI characters into React and vanilla JS applications. Voice, text, lipsync, emotions, video, and screen share — all in one package.
Breaking for 1.8.0 users: the
<convai-chat>custom element and the./embed/browserentry point (dist/embed/chat-embed-v1.js) are removed. The copy/paste published-chat embed is now<convai-widget>from@convai/web-widget; the CHANGELOG has the before/after snippet.
characters config option (two copies of one character are two independent members), switch who answers with setInteractionTarget(), add or remove members live with updateCharacterRoster(), and let other participants join the same room with joinRoom(). Every transcript line, audio track and readiness event is attributed to a membershipId. Usage below.client.on(event, callback) infers each callback's payload from the exported ConvaiEventMap, so botReady, modelOutput, characterStatus and the rest are checked at compile time. See the events reference.appendMessage() inserts a line into the transcript without sending anything, and toggleUsageUpdates(true) opts into per-turn usageUpdate cost breakdowns (on a session connected with debug: true; otherwise the server has no usage stream).characterVersion config option, and list, compare, release, promote, fork and discard versions through client.characterVersions. Usage below.interactionApiUrl: text in, the reply streamed as text and, with ttsEnabled, as speech (audioData); the one transport that runs under Node. We recommend SSE for chat and realtime for voice — see Choosing a Transport. Usage below.@convai/web-sdk/core, /vanilla, /vanilla/websocket and /lipsync-helpers load in plain Node, SSR and Next.js server components.setCharacterMuted() / isCharacterMuted() silence the character's voice for the listener; audioControls stays the microphone.rawMessage event exposes incoming LiveKit, WebSocket and SSE frames.capabilities
selects v2 correlated client tool calls, the canonical Core-validated
modelOutput envelope, and raw bot-llm-text streaming, each independent
of the others. Core must confirm a v2 capability or the SDK refuses the
connection, so a half-upgraded backend fails at connect rather than
mid-conversation. Omit the block entirely and nothing changes: existing
connections and one-argument text messages stay on the v1 wire contract.
Options table.dist/embed/chat-embed-v1.js for iframe-free embeds. The embed never
receives a publisher API key. Usage below.stateOfMind config option, and change it mid-session with updateEmotion() without triggering a response. Usage below.blendshapeConfig.deliver_chunks_ahead: false.ConvaiWidget adapts to light and dark backgrounds.narrativeTemplateKeys config option, replace them mid-session with updateTemplateKeys(). Usage below.actionResponse is now typed via the exported ConvaiAction / ActionResponseEvent, including the target of parameterized actions. Usage below.useConvaiClient hook, ConvaiWidget, and a framework-agnostic coreclient.on payload typed through ConvaiEventMapnpm install @convai/web-sdk
# or
pnpm add @convai/web-sdk
# or
yarn add @convai/web-sdk
React peer dependencies: react and react-dom ^18 || ^19
Runtime requirement: secure context (https:// or http://localhost) for microphone/camera access.
import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";
export function App() {
const client = useConvaiClient({
apiKey: import.meta.env.VITE_CONVAI_API_KEY,
characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});
return <ConvaiWidget convaiClient={client} />;
}
import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";
const client = new ConvaiClient({
apiKey: import.meta.env.VITE_CONVAI_API_KEY,
characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});
const widget = createConvaiWidget(document.body, { convaiClient: client });
window.addEventListener("beforeunload", () => {
widget.destroy();
void client.disconnect();
});
See the Published text chat guide for React and vanilla integrations, lifecycle behavior, and the complete API reference.
Published chat pages use a short-lived launch grant instead of the publisher's API key. Native Web SDK integration is the recommended embed path; an iframe is not required. Obtain the grant from the public Character API and pass it to the core client:
import { ConvaiClient } from '@convai/web-sdk/core';
import { requestPublishedChatGrant } from '@convai/web-sdk/embed';
const launchToken = await requestPublishedChatGrant(
`${characterApiUrl}/chat-publications/${publicationId}/launch-grants`,
);
const client = new ConvaiClient();
const unsubscribe = client.on('botReady', () => {
unsubscribe();
client.sendUserTextMessage('Hello!');
});
await client.connectWithPublicationGrant(launchToken);
The grant is single-use and is not retained for reconnect. Request a new grant
for each new session. The SDK attaches a browser-generated logical attempt ID
and retries one transient grant exchange with that same ID, so a response lost
after server-side consumption does not strand the token or create a duplicate
session. An application performing its own retry may pass the same
connectAttemptId option with the same launch token. Published chat is
text-only; the server-issued room token does not permit publishing microphone
or camera tracks.
In a browser, the @convai/web-sdk/embed grant helper
(requestPublishedChatGrant) creates its own per-browser requester identifier.
If a server or SSR integration calls the published-chat grant helpers directly,
it must pass a caller-scoped requester identifier; do not reuse a process-wide
identifier across visitors.
When using ConvaiWidget, provide requestPublicationGrant so Reset,
Disconnect, and later reopen operations mint a new grant instead of attempting
to reuse the consumed token:
<ConvaiWidget
convaiClient={client}
publishedChatDescriptor={{
display_name: publication.display_name,
image_url: publication.image_url,
}}
requestPublicationGrant={async () => {
const response = await fetch(`/api/chat/${publicationId}/grant`, {
method: 'POST',
});
if (!response.ok) throw new Error('Unable to relaunch published chat');
return (await response.json()).launch_token;
}}
/>
For copy/paste installation without an application build step, use
@convai/web-widget. Its
<convai-widget> custom element reads the same publication attributes, mints
its own grant per session, and labels itself from the public descriptor:
<script src="https://cdn.jsdelivr.net/npm/@convai/web-widget/v1.js" async></script>
<convai-widget
publication-id="PUBLICATION_UUID"
publication-api-url="https://character-api.example.com"
core-url="https://realtime.example.com"
placement="inline"
style="display:block;width:100%;height:640px">
</convai-widget>
Two details are load-bearing, and each renders nothing when wrong: the script
tag must not be type="module" (the loader reads document.currentScript,
which is null in a module script), and a panel sized by style needs
placement="inline" — the widget otherwise defaults to a floating bubble.
Always pin an exact widget version rather than latest: generated embed
snippets must stay bound to the version validated for their target
environment. Platform-generated snippets emit the deployment's public
Character API and Core endpoints explicitly, so an on-prem or non-production
embed never falls through to another deployment and still requires no
credential.
The SDK used to ship this element itself, as <convai-chat> at
@convai/web-sdk/embed/browser. It re-bundled the whole SDK — 588 KB, 43% of
everything this package published — to render a chat panel the vanilla widget
already renders, so it was removed in favour of the widget package. The
server-safe publication helpers at @convai/web-sdk/embed are unchanged.
Descriptor and grant endpoints must allow credential-free cross-origin GET
and POST requests. The embed never receives a publisher API key, character
configuration, or reusable launch token.
For customer-managed and GovCloud deployments, the browser artifact and its
on-prem image now live with @convai/web-widget.
This package no longer builds deploy/published-chat, because it no longer
ships a browser bundle to serve.
Full documentation is at docs.convai.com.
| Guide | Description |
|---|---|
| Quick Start | First working integration in under 5 minutes |
| Configuration | All ConvaiConfig options |
| React Integration | useConvaiClient, ConvaiWidget, and React-specific patterns |
| Vanilla JS | ConvaiClient, createConvaiWidget, and audio setup |
| Events | Full event reference — botReady, stateChange, messagesChange, interactionCreated, and more |
| Context Management | Dynamic context, updateContext, file upload, session management |
| Emotions | Per-turn emotion detection, provider options, and stateOfMind / updateEmotion() |
| Lipsync | ARKit / MetaHuman blendshape streams and BlendshapeQueue API |
| Actions | Trigger character behaviors and scene actions |
| Memory | Long-term memory scoped to end users |
| Audio & Video | Microphone, camera, screen share controls |
| Error Handling | error, disconnect, serverResponse, retry patterns |
| Auth Tokens | Server-side token exchange for production |
| WebSocket Transport | Alternative transport for WebRTC-constrained environments |
| SSE Transport | Chat transport: text in, text or audio out; the one that runs under Node |
| Choosing a Transport | SSE vs realtime vs WebSocket, the recommended SSE-chat-plus-voice-mode setup, and why |
| Character Versioning | Connect to a draft, latest, or tagged version; list, compare, release and promote versions |
| Multi-Character Rooms | Rosters, joinRoom, interaction targets, live roster updates, per-member attribution |
| Published Text Chat | Grant-based text-only sessions, the copy/paste embed, lifecycle and API reference |
| Link Previews | Rendering links the character sends: previews, custom chat UIs, the scheme allowlist |
| Mute the Character's Voice | setCharacterMuted / isCharacterMuted: silence the character for the listener |
| Raw Transport Messages | The typed rawMessage event across LiveKit, WebSocket and SSE |
| Import path | Contents |
|---|---|
@convai/web-sdk | React hooks, components, and re-exported core types |
@convai/web-sdk/react | Same as default (React-explicit alias) |
@convai/web-sdk/vanilla | ConvaiClient, createConvaiWidget, AudioRenderer |
@convai/web-sdk/core | Framework-agnostic ConvaiClient, managers, and all types |
@convai/web-sdk/embed | Server-safe publication descriptor and grant helpers |
@convai/web-sdk/lipsync-helpers | Blendshape format utilities and queue helpers |
@convai/web-sdk/vanilla/websocket | Opt-in. Registers the WebSocket transport. Import alongside /vanilla when using transport: "websocket". |
The package is ESM. Every entry point above also resolves through the require
and default conditions, so CommonJS tooling — Jest, ts-node, CJS bundlers —
can resolve and mock it:
const { ConvaiClient } = require('@convai/web-sdk/core')
There is no separate CommonJS build; require() loads the ESM files directly,
which Node supports from 22.12 onward. On older Node, import it or let
your bundler handle the ESM.
The default transport is WebRTC (LiveKit). A WebSocket-based transport is available for environments where WebRTC is unavailable or undesirable.
// Add this import once (e.g. in your app entry file)
import "@convai/web-sdk/vanilla/websocket";
import { ConvaiClient } from "@convai/web-sdk/vanilla";
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
transport: "websocket",
});
The WebSocket packages (@pipecat-ai/client-js, @pipecat-ai/websocket-transport) are excluded from your bundle unless you import the /vanilla/websocket subpath — so LiveKit-only apps pay no bundle cost.
See the WebSocket Transport guide for the full feature comparison.
Use the interaction API for chat — typed input, with the reply streamed as text and, optionally, speech:
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
interactionApiUrl: "https://interaction-api-stg.convai.com/v1/interactions",
});
await client.connect();
client.sendUserTextMessage("Hello!");
This sends each message as an SSE POST with Authorization: Bearer .... Set transport: "sse"
explicitly if preferred. With ttsEnabled: true the reply's speech streams as audioData
events on the same response; there is no microphone, video, or lipsync. It never calls
/connect, so connect() makes no network request and the session id arrives on the first
interaction via the characterSessionId event.
We recommend SSE for chat and realtime for voice: start on SSE and move the same conversation to realtime when the user wants to talk. See Choosing a Transport for why, and for building on one transport only.
Because it needs neither WebRTC nor a DOM, this is the only transport that runs outside a browser.
Import @convai/web-sdk/core from Node, SSR, or a serverless function — not the root or /react
entry points, which pull in browser-only dependencies.
The interaction API is currently reachable at staging only; a production host is planned. Full guide: SSE Interaction Transport.
Every character has an editable draft and, once released, immutable tagged versions
(1.0, 1.1, …). A movable latest pointer names the release the runtime uses when a client
connects without a selector. Pick the version to run with characterVersion:
characterVersion | Runs |
|---|---|
| (omitted) | The effective latest — how characters that predate versioning behave |
"draft" | The editable draft, for testing unreleased changes |
"latest" | The promoted latest release, explicitly |
"1.2" / "1.2.3" | An immutable tagged version |
const client = useConvaiClient({
apiKey: "...",
characterId: "...",
characterVersion: "draft",
});
On the wire the selector is joined to the id as <uuid>-draft; client.characterId keeps
returning the bare UUID, and client.characterReference returns the joined form.
Manage versions from the same client — client.characterVersions is available as soon as the
config has an apiKey, before connecting:
const versions = client.characterVersions!;
const { has_unpublished_changes, draft_revision_id } = await versions.list();
const diff = await versions.diff("latest", "draft", { view: "semantic" });
await versions.create("1.1", { makeLatest: true }); // release the draft as a tag
await versions.promote("1.0"); // roll back: latest → 1.0
await versions.discardDraft(draft_revision_id!); // throw away unreleased edits
CharacterVersionManager can also be constructed standalone with an apiKey or a Convai
personal access token; every method rejects with CharacterApiError (status, detail) on a
non-2xx answer. Point characterApiUrl at https://api2-stg.convai.com to author against
staging.
Explicit selectors are resolved by the runtime through the Character REST platform. Staging has
this today; production answers explicit selectors with 503 until it is promoted there.
Full guide: Character Versioning.
Pass characters instead of characterId to open a room with several members.
characters[0] answers first; the same character id may appear more than once
and each entry is an independent member with its own membershipId. Roster
rooms need the LiveKit transport and a nonblank endUserId.
const client = new ConvaiClient({
apiKey: "YOUR_API_KEY",
endUserId: "player-42",
characters: [
{ characterId: GUIDE }, // initial active member
{ characterId: ASSESSOR },
{ characterId: GUIDE }, // a second, independent copy of the guide
],
maxNumParticipants: 2,
});
client.on("characterReady", (member) => console.log("ready", member.membershipId));
client.on("characterStatus", ({ membershipId, status, failureCode }) => {
if (status === "failed") console.warn(membershipId, failureCode);
});
await client.connect();
// Route the next user turns to another member; null closes every gate.
await client.setInteractionTarget(client.characters[1].membershipId);
// Change the roster on a live room.
await client.updateCharacterRoster({ add: [{ characterId: GUIDE }] });
await client.updateCharacterRoster({ remove: [client.characters[0].membershipId] });
// Another participant joins the same room without sending any topology.
await other.joinRoom({ roomSessionId: client.roomSession!.roomSessionId, endUserId: "player-7" });
Key points:
characters and characterId are mutually exclusive, and roster entries take a bare character UUID (no -draft / -1.2 selectors).provisioningStatus describes dispatch only; a member is usable when its own characterReady arrives.characterAudioTrack events and botReady payloads carry membershipId, so two copies of one character never merge.ConvaiRoomError with status, code, requestTraceId and isRetryable; the SDK retries ROSTER_PROVISIONING_IN_PROGRESS for you (rosterProvisioningRetry).actionConfig.tools are single-character only and cannot be combined with a roster.Full reference: Host several characters in one room.
Vision dynamic context is the default WebRTC/LiveKit vision path when enableVideo: true. Camera, screen, canvas, and custom video tracks can feed unified vision context; set visionInputConfig.enabled: false only when you need to keep the video channel while opting out.
import { useConvaiClient } from "@convai/web-sdk";
const client = useConvaiClient({
apiKey: import.meta.env.VITE_CONVAI_API_KEY,
characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
enableVideo: true,
visionInputConfig: {
framesPerTurn: 12,
bufferFrames: 30,
samplingWindows: [
{ count: 6, intervalMs: 300 },
{ count: 6, intervalMs: 1000 },
],
stalenessSeconds: 10,
replacePreviousVisionContext: true,
},
respondModes: {
vision: "silent",
contextUpdate: "auto",
sceneMetadata: "silent",
trigger: "must_respond",
},
});
Publish visual sources with semantic labels so acknowledgments can distinguish webcam, canvas, screen, and custom feeds:
await client.videoControls.enableVideo(); // webcam source
const handle = await client.videoControls.publishCanvas(canvas, {
source: "canvas",
name: "canvas-pov",
fps: 1,
});
const statusId = client.visionStatus();
const triggerId = client.visionTrigger({
respondMode: "silent",
frameIndices: [-1, -1],
});
await client.videoControls.unpublishVisionSource(handle);
visionStatus() and visionTrigger() return an update id. Match it against serverResponse events for outcomes such as frames_available, buffer_empty, attached, no_active_video or vision_not_enabled. Vision needs more than enableVideo: true: the account's unified vision context feature and a vision-capable, non-realtime model. Without them frames publish fine but are never seen, and the SDK warns once per session.
Declare affordances in actionConfig, then handle the actionResponse event.
Actions run in order; a target means it's parameterized (acts on an object/character).
client.on("actionResponse", ({ actions }: ActionResponseEvent) => {
for (const { name, target } of actions) {
target ? runParameterized(name, target) : runSimple(name); // e.g. "Move To"/"Cube" vs "Wave"
}
});
The v2 protocol is an explicit opt-in. Existing connections continue to receive
the v1 actionResponse contract. A v2 client declares tools at connect time,
dispatches only Core-validated items, and returns one correlated terminal
result for every tool_call.
import { ConvaiClient, type ActionResult } from "@convai/web-sdk/core";
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
capabilities: {
actionProtocolVersion: 2,
modelOutputVersion: 2,
},
actionConfig: {
actions: [],
objects: [],
characters: [],
tools: [
{
name: "search_documents",
description: "Search the user's approved document collection",
inputSchema: {
type: "object",
properties: { query: { type: "string" } },
required: ["query"],
},
},
],
},
});
client.on("modelOutput", async (output) => {
for (const item of output.items) {
if (item.type === "message") renderAssistantText(item.content);
if (item.type !== "tool_call") continue;
let terminalResult: ActionResult;
try {
const result = await executeApprovedTool(item.name, item.arguments);
terminalResult = {
id: item.id,
status: "completed",
output: result,
};
} catch (error) {
terminalResult = {
id: item.id,
status: "error",
error: { message: String(error) },
};
}
try {
const ack = await client.sendActionResultAndWait(terminalResult, {
timeoutMs: 10_000,
});
markResultReported(item.id, ack.idempotent);
} catch (deliveryError) {
markResultDeliveryUncertain(item.id, deliveryError);
// If appropriate for your session lifecycle, retry `terminalResult`
// unchanged. Never execute the tool again merely because its ACK was lost.
}
}
});
Model output v2 may provide a separate diagnostics-only raw stream alongside
the unchanged bot-llm-text chat stream. Each chunk carries Core's stable
logical-turn correlation id and, on newer Core versions, an invocation id:
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
capabilities: { modelOutputVersion: 2 },
});
client.on("botLlmTextRaw", ({ text, logical_turn_id, output_id }) => {
appendRawDiagnostics(output_id ?? logical_turn_id, text);
});
The SDK ignores this channel unless Core negotiated modelOutputVersion: 2.
Raw chunks never enter chatMessages, bot output, TTS, speech, or response
lifecycle state. They are retained for diagnostics and must never be executed
or reparsed. The ordinary bot-llm-text text and cadence remain unchanged. On
v2 connections, ChatMessage.outputId matches the raw event's optional
output_id and the final modelOutput.output_id; logicalTurnId/
logical_turn_id still groups every invocation in the user turn. Older v2 Core
omits the invocation id, so applications can fall back to one turn-level raw
aggregate. Never correlate by arrival order. When v2 is selected, the SDK suppresses the legacy actionResponse compatibility
projection so an application cannot execute the same operation twice.
Developer-defined visual output uses data-only extension items. Applications
register a renderer for a specific schema and version; unknown schemas use the
server-provided text fallback or remain unhandled. The SDK never evaluates code
from a model output.
import { ModelOutputExtensionRegistry } from "@convai/web-sdk/core";
const extensions = new ModelOutputExtensionRegistry<HTMLElement>();
extensions.register("com.example/status-card", 1, ({ payload }) =>
renderStatusCard(payload),
);
Personalize Narrative Design section objectives per session. Placeholders like {player_name} in the graph are substituted from one key map — seed it at connect, replace it at runtime.
const client = new ConvaiClient({
apiKey: "...",
characterId: "...", // Narrative Design enabled
narrativeTemplateKeys: { player_name: "Alex", quest_item: "oxygen generator" },
});
// Later — replaces the whole map; set before firing the next trigger
client.updateTemplateKeys({ player_name: "Alex", quest_item: "med kit" });
client.sendTriggerMessage("QuestUpdate");
Requires Narrative Design on the character; updateTemplateKeys is a full replace, not a merge. See the Context Management guide for dashboard setup and troubleshooting.
Set a temporary generation mood that shapes tone and pacing for the next response. This is separate from enableEmotion, which reports the character's detected emotion after a turn.
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
stateOfMind: "anticipation",
});
// Mid-session. Affects the next response; does not trigger one.
client.updateEmotion("joy");
client.updateEmotion(null); // clear — "neutral" and "" clear it too
The server applies Plutchik's eight primary emotions — joy, trust, fear, surprise, sadness, disgust, anger, anticipation — and ignores anything else (the SDK warns). Values are trimmed and lowercased. The runtime value carries into the next connect, including a reconnect; an explicit stateOfMind in the connect config takes precedence.
Requires backend support — live on production as of 2026-08-20. On a backend without it, the server replies with an Unknown message type error on serverResponse. Because /connect silently ignores unknown fields, a successful connect alone does not prove state_of_mind was applied, so watch the ack:
client.on("serverResponse", (r) => {
if (r.event_type === "update-emotion" && r.status !== "success") {
console.warn("state of mind unsupported by this backend:", r.message);
}
});
Licensed under the Apache License 2.0. Copyright 2025 Convai.
FAQs
Build web apps with lifelike AI characters. The Convai Web SDK gives you real-time voice, lipsync, emotions, and dynamic context — with first-class support for React and vanilla TypeScript.
The npm package @convai/web-sdk receives a total of 533 weekly downloads. As such, @convai/web-sdk popularity was classified as not popular.
We found that @convai/web-sdk demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Security News
On Front Lines’ Early Adopters: CISO podcast, Andrew Becherer shares lessons from Datadog, weighs the long-term costs of AI-built tools, and explains how founders can actually gain a CISO’s trust.
Security News
/Product
Socket for Origin scans dependency changes in pull requests and flags security risks based on your team’s policies. Now available in Cursor’s App Marketplace.
Security News
Agent Baseline’s new reference architecture shows how to secure software factories with four boundaries enforced outside the AI agent.