@convai/web-sdk
Advanced tools
Build web apps with lifelike AI characters. The Convai Web SDK gives you real-time voice, lipsync, emotions, and dynamic context — with first-class support for React and vanilla TypeScript.
Real-time conversational AI characters for the web.
TypeScript-first SDK for embedding Convai AI characters into React and vanilla JS applications. Voice, text, lipsync, emotions, video, and screen share — all in one package.
Install the beta with npm install @convai/web-sdk@beta.
characters config option (two copies of one character are two independent members), switch who answers with setInteractionTarget(), add or remove members live with updateCharacterRoster(), and let other participants join the same room with joinRoom(). Every transcript line, audio track and readiness event is attributed to a membershipId. Usage below.capabilities, declare client-executed tools in actionConfig.tools, consume typed modelOutput envelopes, return results with sendActionResultAndWait() for a correlated server acknowledgement, and read the raw provider stream through botLlmTextRaw. visionPromptMode: "disabled" keeps scene context without advertising visual perception. Usage below.connectWithPublicationGrant(), connectionType: "text", and the self-contained <convai-chat> browser bundle. Usage below.client.on(event, callback) infers each callback's payload from the exported ConvaiEventMap, so botReady, modelOutput, characterStatus and the rest are checked at compile time. See the events reference.appendMessage() inserts a line into the transcript without sending anything, and toggleUsageUpdates(true) opts into per-turn usageUpdate cost breakdowns.characterVersion config option, and list, compare, release, promote, fork and discard versions through client.characterVersions. Usage below.interactionApiUrl; the one transport that runs under Node. Usage below.@convai/web-sdk/core, /vanilla, /vanilla/websocket and /lipsync-helpers load in plain Node, SSR and Next.js server components.stateOfMind config option, and change it mid-session with updateEmotion() without triggering a response. Usage below.blendshapeConfig.deliver_chunks_ahead: false.ConvaiWidget adapts to light and dark backgrounds.narrativeTemplateKeys config option, replace them mid-session with updateTemplateKeys(). Usage below.actionResponse is now typed via the exported ConvaiAction / ActionResponseEvent, including the target of parameterized actions. Usage below.useConvaiClient hook, ConvaiWidget, and a framework-agnostic core<convai-chat> elementclient.on payload typed through ConvaiEventMapnpm install @convai/web-sdk
# or
pnpm add @convai/web-sdk
# or
yarn add @convai/web-sdk
React peer dependencies: react and react-dom ^18 || ^19
Runtime requirement: secure context (https:// or http://localhost) for microphone/camera access.
import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";
export function App() {
const client = useConvaiClient({
apiKey: import.meta.env.VITE_CONVAI_API_KEY,
characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});
return <ConvaiWidget convaiClient={client} />;
}
import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";
const client = new ConvaiClient({
apiKey: import.meta.env.VITE_CONVAI_API_KEY,
characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});
const widget = createConvaiWidget(document.body, { convaiClient: client });
window.addEventListener("beforeunload", () => {
widget.destroy();
void client.disconnect();
});
See the Published text chat guide for React and vanilla integrations, lifecycle behavior, and the complete API reference.
Published chat pages use a short-lived launch grant instead of the publisher's API key. Native Web SDK integration is the recommended embed path; an iframe is not required. Obtain the grant from the public Character API and pass it to the core client:
import { ConvaiClient } from '@convai/web-sdk/core';
import { requestPublishedChatGrant } from '@convai/web-sdk/embed';
const launchToken = await requestPublishedChatGrant(
`${characterApiUrl}/chat-publications/${publicationId}/launch-grants`,
);
const client = new ConvaiClient();
const unsubscribe = client.on('botReady', () => {
unsubscribe();
client.sendUserTextMessage('Hello!');
});
await client.connectWithPublicationGrant(launchToken);
The grant is single-use and is not retained for reconnect. Request a new grant
for each new session. The SDK attaches a browser-generated logical attempt ID
and retries one transient grant exchange with that same ID, so a response lost
after server-side consumption does not strand the token or create a duplicate
session. An application performing its own retry may pass the same
connectAttemptId option with the same launch token. Published chat is
text-only; the server-issued room token does not permit publishing microphone
or camera tracks.
The @convai/web-sdk/embed browser element creates its own per-browser
requester identifier. If a server or SSR integration calls its published-chat
grant helpers directly, it must pass a caller-scoped requester identifier; do
not reuse a process-wide identifier across visitors.
When using ConvaiWidget, provide requestPublicationGrant so Reset,
Disconnect, and later reopen operations mint a new grant instead of attempting
to reuse the consumed token:
<ConvaiWidget
convaiClient={client}
publishedChatDescriptor={{
display_name: publication.display_name,
image_url: publication.image_url,
}}
requestPublicationGrant={async () => {
const response = await fetch(`/api/chat/${publicationId}/grant`, {
method: 'POST',
});
if (!response.ok) throw new Error('Unable to relaunch published chat');
return (await response.json()).launch_token;
}}
/>
For copy/paste installation without an application build step, deploy the
versioned browser artifact at dist/embed/chat-embed-v1.js and use its native
custom element:
<script
type="module"
src="https://cdn.jsdelivr.net/npm/@convai/web-sdk@<SDK_VERSION>/dist/embed/chat-embed-v1.js"
></script>
<convai-chat publication-id="PUBLICATION_UUID"></convai-chat>
The element renders directly in the host page with Shadow DOM style isolation,
automatically mints a fresh grant per connection, and emits convai-ready,
convai-disconnect, and convai-error DOM events. The production bundle uses
https://api.convai.com and https://realtime-api.convai.com only as defaults
for a hand-written SaaS integration. Platform-generated snippets always emit
the deployment's public Character API and Core endpoints explicitly, so an
on-prem or non-production embed never falls through to another deployment and
still requires no credential:
<convai-chat
publication-id="PUBLICATION_UUID"
publication-api-url="https://character-api.example.com"
core-url="https://realtime.example.com"
></convai-chat>
Replace <SDK_VERSION> with the exact published package version. Do not use
latest: generated embed snippets must stay bound to the SDK version validated
for their target environment.
Descriptor and grant endpoints must allow credential-free cross-origin GET
and POST requests. The embed never receives a publisher API key, character
configuration, or reusable launch token.
For customer-managed and GovCloud deployments, the repository also builds a
browser-only image from deploy/published-chat/Dockerfile. It serves the exact
released bundle at /sdk/chat-embed-v1.js on port 8080, plus /healthz, and
contains no API key or cloud identity. The manual build-onprem workflow
requires an immutable semver tag matching package.json, refuses to overwrite
an existing Docker Hub tag, and publishes both amd64 and arm64 images. Pin the
resulting digest in the deployment repository; do not make an on-prem browser
depend on jsDelivr, npm availability, GCP workload identity, or the Experience
application image.
Full documentation is at docs.convai.com.
| Guide | Description |
|---|---|
| Quick Start | First working integration in under 5 minutes |
| Configuration | All ConvaiConfig options |
| React Integration | useConvaiClient, ConvaiWidget, and React-specific patterns |
| Vanilla JS | ConvaiClient, createConvaiWidget, and audio setup |
| Events | Full event reference — botReady, stateChange, messagesChange, interactionCreated, and more |
| Context Management | Dynamic context, updateContext, file upload, session management |
| Emotions | Per-turn emotion detection, provider options, and stateOfMind / updateEmotion() |
| Lipsync | ARKit / MetaHuman blendshape streams and BlendshapeQueue API |
| Actions | Trigger character behaviors and scene actions |
| Memory | Long-term memory scoped to end users |
| Audio & Video | Microphone, camera, screen share controls |
| Error Handling | error, disconnect, serverResponse, retry patterns |
| Auth Tokens | Server-side token exchange for production |
| WebSocket Transport | Alternative transport for WebRTC-constrained environments |
| SSE Transport | Text-only interaction transport; the one that runs under Node |
| Character Versioning | Connect to a draft, latest, or tagged version; list, compare, release and promote versions |
| Multi-Character Rooms | Rosters, joinRoom, interaction targets, live roster updates, per-member attribution |
| Published Text Chat | Grant-based text-only sessions, <convai-chat>, lifecycle and API reference |
| Events Reference (local) | Every event with its typed payload, including the v2 and multi-character events |
| Import path | Contents |
|---|---|
@convai/web-sdk | React hooks, components, and re-exported core types |
@convai/web-sdk/react | Same as default (React-explicit alias) |
@convai/web-sdk/vanilla | ConvaiClient, createConvaiWidget, AudioRenderer |
@convai/web-sdk/core | Framework-agnostic ConvaiClient, managers, and all types |
@convai/web-sdk/embed | Server-safe publication descriptor and grant helpers |
@convai/web-sdk/embed/browser | Self-contained browser bundle that registers <convai-chat> |
@convai/web-sdk/lipsync-helpers | Blendshape format utilities and queue helpers |
@convai/web-sdk/vanilla/websocket | Opt-in. Registers the WebSocket transport. Import alongside /vanilla when using transport: "websocket". |
The default transport is WebRTC (LiveKit). A WebSocket-based transport is available for environments where WebRTC is unavailable or undesirable.
// Add this import once (e.g. in your app entry file)
import "@convai/web-sdk/vanilla/websocket";
import { ConvaiClient } from "@convai/web-sdk/vanilla";
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
transport: "websocket",
});
The WebSocket packages (@pipecat-ai/client-js, @pipecat-ai/websocket-transport) are excluded from your bundle unless you import the /vanilla/websocket subpath — so LiveKit-only apps pay no bundle cost.
See the WebSocket Transport guide for the full feature comparison.
Use the interaction API for text-only integrations that need a streamed response:
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
interactionApiUrl: "https://interaction-api-stg.convai.com/v1/interactions",
});
await client.connect();
client.sendUserTextMessage("Hello!");
This sends each message as an SSE POST with Authorization: Bearer .... Set transport: "sse"
explicitly if preferred. SSE carries text only — no voice, video, or lipsync — and it never calls
/connect, so connect() makes no network request and the session id arrives on the first
interaction via the characterSessionId event.
Because it needs neither WebRTC nor a DOM, this is the only transport that runs outside a browser.
Import @convai/web-sdk/core from Node, SSR, or a serverless function — not the root or /react
entry points, which pull in browser-only dependencies.
The interaction API is currently reachable at staging only; a production host is planned. Full guide: SSE Interaction Transport.
Every character has an editable draft and, once released, immutable tagged versions
(1.0, 1.1, …). A movable latest pointer names the release the runtime uses when a client
connects without a selector. Pick the version to run with characterVersion:
characterVersion | Runs |
|---|---|
| (omitted) | The effective latest — how characters that predate versioning behave |
"draft" | The editable draft, for testing unreleased changes |
"latest" | The promoted latest release, explicitly |
"1.2" / "1.2.3" | An immutable tagged version |
const client = useConvaiClient({
apiKey: "...",
characterId: "...",
characterVersion: "draft",
});
On the wire the selector is joined to the id as <uuid>-draft; client.characterId keeps
returning the bare UUID, and client.characterReference returns the joined form.
Manage versions from the same client — client.characterVersions is available as soon as the
config has an apiKey, before connecting:
const versions = client.characterVersions!;
const { has_unpublished_changes, draft_revision_id } = await versions.list();
const diff = await versions.diff("latest", "draft", { view: "semantic" });
await versions.create("1.1", { makeLatest: true }); // release the draft as a tag
await versions.promote("1.0"); // roll back: latest → 1.0
await versions.discardDraft(draft_revision_id!); // throw away unreleased edits
CharacterVersionManager can also be constructed standalone with an apiKey or a Convai
personal access token; every method rejects with CharacterApiError (status, detail) on a
non-2xx answer. Point characterApiUrl at https://api2-stg.convai.com to author against
staging.
Explicit selectors are resolved by the runtime through the Character REST platform. Staging has
this today; production answers explicit selectors with 503 until it is promoted there.
Full guide: Character Versioning.
Pass characters instead of characterId to open a room with several members.
characters[0] answers first; the same character id may appear more than once
and each entry is an independent member with its own membershipId. Roster
rooms need the LiveKit transport and a nonblank endUserId.
const client = new ConvaiClient({
apiKey: "YOUR_API_KEY",
endUserId: "player-42",
characters: [
{ characterId: GUIDE }, // initial active member
{ characterId: ASSESSOR },
{ characterId: GUIDE }, // a second, independent copy of the guide
],
maxNumParticipants: 2,
});
client.on("characterReady", (member) => console.log("ready", member.membershipId));
client.on("characterStatus", ({ membershipId, status, failureCode }) => {
if (status === "failed") console.warn(membershipId, failureCode);
});
await client.connect();
// Route the next user turns to another member; null closes every gate.
await client.setInteractionTarget(client.characters[1].membershipId);
// Change the roster on a live room.
await client.updateCharacterRoster({ add: [{ characterId: GUIDE }] });
await client.updateCharacterRoster({ remove: [client.characters[0].membershipId] });
// Another participant joins the same room without sending any topology.
await other.joinRoom({ roomSessionId: client.roomSession!.roomSessionId, endUserId: "player-7" });
Key points:
characters and characterId are mutually exclusive, and roster entries take a bare character UUID (no -draft / -1.2 selectors).provisioningStatus describes dispatch only; a member is usable when its own characterReady arrives.characterAudioTrack events and botReady payloads carry membershipId, so two copies of one character never merge.ConvaiRoomError with status, code, requestTraceId and isRetryable; the SDK retries ROSTER_PROVISIONING_IN_PROGRESS for you (rosterProvisioningRetry).actionConfig.tools are single-character only and cannot be combined with a roster.Full reference: docs/convai_multi_character.md.
Vision dynamic context is the default WebRTC/LiveKit vision path when enableVideo: true. Camera, screen, canvas, and custom video tracks can feed unified vision context; set visionInputConfig.enabled: false only when you need to keep the video channel while opting out.
import { useConvaiClient } from "@convai/web-sdk";
const client = useConvaiClient({
apiKey: import.meta.env.VITE_CONVAI_API_KEY,
characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
enableVideo: true,
visionInputConfig: {
framesPerTurn: 12,
bufferFrames: 30,
samplingWindows: [
{ count: 6, intervalMs: 300 },
{ count: 6, intervalMs: 1000 },
],
stalenessSeconds: 10,
replacePreviousVisionContext: true,
},
respondModes: {
vision: "silent",
contextUpdate: "auto",
sceneMetadata: "silent",
trigger: "must_respond",
},
});
Publish visual sources with semantic labels so acknowledgments can distinguish webcam, canvas, screen, and custom feeds:
await client.videoControls.enableVideo(); // webcam source
const handle = await client.videoControls.publishCanvas(canvas, {
source: "canvas",
name: "canvas-pov",
fps: 1,
});
const statusId = client.visionStatus();
const triggerId = client.visionTrigger({
respondMode: "silent",
frameIndices: [-1, -1],
});
await client.videoControls.unpublishVisionSource(handle);
visionStatus() and visionTrigger() return an update id. Match it against serverResponse events for outcomes such as frames_available, buffer_empty, attached, or no_active_video.
Declare affordances in actionConfig, then handle the actionResponse event.
Actions run in order; a target means it's parameterized (acts on an object/character).
client.on("actionResponse", ({ actions }: ActionResponseEvent) => {
for (const { name, target } of actions) {
target ? runParameterized(name, target) : runSimple(name); // e.g. "Move To"/"Cube" vs "Wave"
}
});
The v2 protocol is an explicit opt-in. Existing connections continue to receive
the v1 actionResponse contract. A v2 client declares tools at connect time,
dispatches only Core-validated items, and returns one correlated terminal
result for every tool_call.
import { ConvaiClient, type ActionResult } from "@convai/web-sdk/core";
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
capabilities: {
actionProtocolVersion: 2,
modelOutputVersion: 2,
},
actionConfig: {
actions: [],
objects: [],
characters: [],
tools: [
{
name: "search_documents",
description: "Search the user's approved document collection",
inputSchema: {
type: "object",
properties: { query: { type: "string" } },
required: ["query"],
},
},
],
},
});
client.on("modelOutput", async (output) => {
for (const item of output.items) {
if (item.type === "message") renderAssistantText(item.content);
if (item.type !== "tool_call") continue;
let terminalResult: ActionResult;
try {
const result = await executeApprovedTool(item.name, item.arguments);
terminalResult = {
id: item.id,
status: "completed",
output: result,
};
} catch (error) {
terminalResult = {
id: item.id,
status: "error",
error: { message: String(error) },
};
}
try {
const ack = await client.sendActionResultAndWait(terminalResult, {
timeoutMs: 10_000,
});
markResultReported(item.id, ack.idempotent);
} catch (deliveryError) {
markResultDeliveryUncertain(item.id, deliveryError);
// If appropriate for your session lifecycle, retry `terminalResult`
// unchanged. Never execute the tool again merely because its ACK was lost.
}
}
});
Model output v2 may provide a separate diagnostics-only raw stream alongside
the unchanged bot-llm-text chat stream. Each chunk carries Core's stable
logical-turn correlation id and, on newer Core versions, an invocation id:
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
capabilities: { modelOutputVersion: 2 },
});
client.on("botLlmTextRaw", ({ text, logical_turn_id, output_id }) => {
appendRawDiagnostics(output_id ?? logical_turn_id, text);
});
The SDK ignores this channel unless Core negotiated modelOutputVersion: 2.
Raw chunks never enter chatMessages, bot output, TTS, speech, or response
lifecycle state. They are retained for diagnostics and must never be executed
or reparsed. The ordinary bot-llm-text text and cadence remain unchanged. On
v2 connections, ChatMessage.outputId matches the raw event's optional
output_id and the final modelOutput.output_id; logicalTurnId/
logical_turn_id still groups every invocation in the user turn. Older v2 Core
omits the invocation id, so applications can fall back to one turn-level raw
aggregate. Never correlate by arrival order. When v2 is selected, the SDK suppresses the legacy actionResponse compatibility
projection so an application cannot execute the same operation twice.
Developer-defined visual output uses data-only extension items. Applications
register a renderer for a specific schema and version; unknown schemas use the
server-provided text fallback or remain unhandled. The SDK never evaluates code
from a model output.
import { ModelOutputExtensionRegistry } from "@convai/web-sdk/core";
const extensions = new ModelOutputExtensionRegistry<HTMLElement>();
extensions.register("com.example/status-card", 1, ({ payload }) =>
renderStatusCard(payload),
);
Personalize Narrative Design section objectives per session. Placeholders like {player_name} in the graph are substituted from one key map — seed it at connect, replace it at runtime.
const client = new ConvaiClient({
apiKey: "...",
characterId: "...", // Narrative Design enabled
narrativeTemplateKeys: { player_name: "Alex", quest_item: "oxygen generator" },
});
// Later — replaces the whole map; set before firing the next trigger
client.updateTemplateKeys({ player_name: "Alex", quest_item: "med kit" });
client.sendTriggerMessage("QuestUpdate");
Requires Narrative Design on the character; updateTemplateKeys is a full replace, not a merge. See the Context Management guide for dashboard setup and troubleshooting.
Set a temporary generation mood that shapes tone and pacing for the next response. This is separate from enableEmotion, which reports the character's detected emotion after a turn.
const client = new ConvaiClient({
apiKey: "...",
characterId: "...",
stateOfMind: "anticipation",
});
// Mid-session. Affects the next response; does not trigger one.
client.updateEmotion("joy");
client.updateEmotion(null); // clear — "neutral" and "" clear it too
Values are trimmed and lowercased. The runtime value carries into the next connect, including a reconnect; an explicit stateOfMind in the connect config takes precedence.
Requires backend support — live on production as of 2026-08-20. On a backend without it, the server replies with an Unknown message type error on serverResponse. Because /connect silently ignores unknown fields, a successful connect alone does not prove state_of_mind was applied, so watch the ack:
client.on("serverResponse", (r) => {
if (r.event_type === "update-emotion" && r.status !== "success") {
console.warn("state of mind unsupported by this backend:", r.message);
}
});
Licensed under the Apache License 2.0. Copyright 2025 Convai.
FAQs
Build web apps with lifelike AI characters. The Convai Web SDK gives you real-time voice, lipsync, emotions, and dynamic context — with first-class support for React and vanilla TypeScript.
The npm package @convai/web-sdk receives a total of 284 weekly downloads. As such, @convai/web-sdk popularity was classified as not popular.
We found that @convai/web-sdk demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Research
/Security News
A Twitch browser extension on Chrome and Firefox forwards users’ live OAuth session tokens through proxies controlled by a Russian bot service.
Security News
Anthropic found biased reasoning and recklessness drove Claude Mythos 5 to publish malware on PyPI and compromise a security vendor.
Research
/Security News
Malicious Chrome and Firefox extensions target Axiom Trade and Padre users, stealing session tokens and wallet data.