@convai/web-sdk

@convai/web-sdk

Real-time conversational AI characters for the web.

TypeScript-first SDK for embedding Convai AI characters into React and vanilla JS applications. Voice, text, lipsync, emotions, video, and screen share — all in one package.

What's new in 1.8.0 (beta)

Install the beta with npm install @convai/web-sdk@beta.

• Multi-character rooms — connect a roster of characters with the characters config option (two copies of one character are two independent members), switch who answers with setInteractionTarget(), add or remove members live with updateCharacterRoster(), and let other participants join the same room with joinRoom(). Every transcript line, audio track and readiness event is attributed to a membershipId. Usage below.
• Agentic actions and canonical model output (v2) — opt in with capabilities, declare client-executed tools in actionConfig.tools, consume typed modelOutput envelopes, return results with sendActionResultAndWait() for a correlated server acknowledgement, and read the raw provider stream through botLlmTextRaw. visionPromptMode: "disabled" keeps scene context without advertising visual perception. Usage below.
• Published text chat — a credential-free, text-only connection for public chat pages: connectWithPublicationGrant(), connectionType: "text", and the self-contained <convai-chat> browser bundle. Usage below.
• Typed events — client.on(event, callback) infers each callback's payload from the exported ConvaiEventMap, so botReady, modelOutput, characterStatus and the rest are checked at compile time. See the events reference.
• Local transcript lines and usage updates — appendMessage() inserts a line into the transcript without sending anything, and toggleUsageUpdates(true) opts into per-turn usageUpdate cost breakdowns.
• Character versioning — connect to a character's editable draft, its promoted latest release, or an immutable tag with the characterVersion config option, and list, compare, release, promote, fork and discard versions through client.characterVersions. Usage below.
• SSE interaction transport — text-only streamed interactions through the interaction API, selected by interactionApiUrl; the one transport that runs under Node. Usage below.
• Embeddable vanilla widget — the vanilla widget runs inside a shadow root, with a character header and a connecting overlay, so it can be dropped into any page.
• Node-importable dist — @convai/web-sdk/core, /vanilla, /vanilla/websocket and /lipsync-helpers load in plain Node, SSR and Next.js server components.

Previously in 1.7.0

• Character state of mind — set a temporary generation mood with the stateOfMind config option, and change it mid-session with updateEmotion() without triggering a response. Usage below.
• Send-ahead enabled by default — NeuroSync lipsync ahead-delivery is now on by default, reversing the 1.6.0 opt-in. Fall back to the legacy paced path with blendshapeConfig.deliver_chunks_ahead: false.
• Lipsync naturalness pipeline — modular naturalness processing with a tuned MetaHuman profile.
• Adaptive glass widget styling — ConvaiWidget adapts to light and dark backgrounds.

1.6.0

• Narrative Design template keys — personalize one narrative graph per session: seed values at connect with the narrativeTemplateKeys config option, replace them mid-session with updateTemplateKeys(). Usage below.
• Typed parameterized actions — actionResponse is now typed via the exported ConvaiAction / ActionResponseEvent, including the target of parameterized actions. Usage below.
• Vision dynamic context — camera, screen, canvas, and custom tracks feed unified vision context on WebRTC, with hardened WebSocket vision handling. Usage below.

Features

• React & vanilla JS — useConvaiClient hook, ConvaiWidget, and a framework-agnostic core
• Real-time audio/video — full-duplex WebRTC with echo cancellation, camera, and screen share
• Lipsync — ARKit and MetaHuman blendshape streams for facial animation
• Emotions & state of mind — per-turn emotion detection with intensity scale, plus a settable generation mood
• Dynamic context & vision — inject text state, scene metadata, and LiveKit video frames mid-session
• Actions & Narrative Design — typed action decisions with parameterized targets, named triggers, and per-session template keys
• Long-term memory — persistent cross-session memories scoped to each end user
• Multi-character rooms — a roster of characters in one room, per-member attribution, live roster changes
• Agentic actions (v2) — client-executed tools, canonical model output, acknowledged action results, raw provider stream
• Published text chat — grant-based, credential-free text sessions and a drop-in <convai-chat> element
• Typed events — every client.on payload typed through ConvaiEventMap
• Character versioning — connect to a draft, latest, or tagged version, and manage releases from the client
• File upload — send images to the character during a live session
• WebSocket transport — opt-in alternative to WebRTC for constrained networks
• Auth tokens — server-side token exchange for production deployments

Installation

npm install @convai/web-sdk
# or
pnpm add @convai/web-sdk
# or
yarn add @convai/web-sdk

React peer dependencies: react and react-dom ^18 || ^19

Runtime requirement: secure context (https:// or http://localhost) for microphone/camera access.

Quick start

React

import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";

export function App() {
  const client = useConvaiClient({
    apiKey: import.meta.env.VITE_CONVAI_API_KEY,
    characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  });

  return <ConvaiWidget convaiClient={client} />;
}

Vanilla TypeScript

import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});

const widget = createConvaiWidget(document.body, { convaiClient: client });

window.addEventListener("beforeunload", () => {
  widget.destroy();
  void client.disconnect();
});

Published text chat

See the Published text chat guide for React and vanilla integrations, lifecycle behavior, and the complete API reference.

Published chat pages use a short-lived launch grant instead of the publisher's API key. Native Web SDK integration is the recommended embed path; an iframe is not required. Obtain the grant from the public Character API and pass it to the core client:

import { ConvaiClient } from '@convai/web-sdk/core';
import { requestPublishedChatGrant } from '@convai/web-sdk/embed';

const launchToken = await requestPublishedChatGrant(
  `${characterApiUrl}/chat-publications/${publicationId}/launch-grants`,
);
const client = new ConvaiClient();
const unsubscribe = client.on('botReady', () => {
  unsubscribe();
  client.sendUserTextMessage('Hello!');
});
await client.connectWithPublicationGrant(launchToken);

The grant is single-use and is not retained for reconnect. Request a new grant for each new session. The SDK attaches a browser-generated logical attempt ID and retries one transient grant exchange with that same ID, so a response lost after server-side consumption does not strand the token or create a duplicate session. An application performing its own retry may pass the same connectAttemptId option with the same launch token. Published chat is text-only; the server-issued room token does not permit publishing microphone or camera tracks.

The @convai/web-sdk/embed browser element creates its own per-browser requester identifier. If a server or SSR integration calls its published-chat grant helpers directly, it must pass a caller-scoped requester identifier; do not reuse a process-wide identifier across visitors.

When using ConvaiWidget, provide requestPublicationGrant so Reset, Disconnect, and later reopen operations mint a new grant instead of attempting to reuse the consumed token:

<ConvaiWidget
  convaiClient={client}
  publishedChatDescriptor={{
    display_name: publication.display_name,
    image_url: publication.image_url,
  }}
  requestPublicationGrant={async () => {
    const response = await fetch(`/api/chat/${publicationId}/grant`, {
      method: 'POST',
    });
    if (!response.ok) throw new Error('Unable to relaunch published chat');
    return (await response.json()).launch_token;
  }}
/>

For copy/paste installation without an application build step, deploy the versioned browser artifact at dist/embed/chat-embed-v1.js and use its native custom element:

<script
  type="module"
  src="https://cdn.jsdelivr.net/npm/@convai/web-sdk@<SDK_VERSION>/dist/embed/chat-embed-v1.js"
></script>
<convai-chat publication-id="PUBLICATION_UUID"></convai-chat>

The element renders directly in the host page with Shadow DOM style isolation, automatically mints a fresh grant per connection, and emits convai-ready, convai-disconnect, convai-error, convai-copy, and convai-session-reset DOM events. It accepts full-window, display-name and image-url alongside the endpoint attributes, and is themed through --convai-chat-* custom properties — see the published chat guide. The production bundle uses https://api.convai.com and https://realtime-api.convai.com only as defaults for a hand-written SaaS integration. Platform-generated snippets always emit the deployment's public Character API and Core endpoints explicitly, so an on-prem or non-production embed never falls through to another deployment and still requires no credential:

<convai-chat
  publication-id="PUBLICATION_UUID"
  publication-api-url="https://character-api.example.com"
  core-url="https://realtime.example.com"
></convai-chat>

Replace <SDK_VERSION> with the exact published package version. Do not use latest: generated embed snippets must stay bound to the SDK version validated for their target environment.

Descriptor and grant endpoints must allow credential-free cross-origin GET and POST requests. The embed never receives a publisher API key, character configuration, or reusable launch token.

For customer-managed and GovCloud deployments, the repository also builds a browser-only image from deploy/published-chat/Dockerfile. It serves the exact released bundle at /sdk/chat-embed-v1.js on port 8080, plus /healthz, and contains no API key or cloud identity. The manual build-onprem workflow requires an immutable semver tag matching package.json, refuses to overwrite an existing Docker Hub tag, and publishes both amd64 and arm64 images. Pin the resulting digest in the deployment repository; do not make an on-prem browser depend on jsDelivr, npm availability, GCP workload identity, or the Experience application image.

Documentation

Full documentation is at docs.convai.com.

Guide	Description
Quick Start	First working integration in under 5 minutes
Configuration	All ConvaiConfig options
React Integration	useConvaiClient, ConvaiWidget, and React-specific patterns
Vanilla JS	ConvaiClient, createConvaiWidget, and audio setup
Events	Full event reference — botReady, stateChange, messagesChange, interactionCreated, and more
Context Management	Dynamic context, updateContext, file upload, session management
Emotions	Per-turn emotion detection, provider options, and stateOfMind / updateEmotion()
Lipsync	ARKit / MetaHuman blendshape streams and BlendshapeQueue API
Actions	Trigger character behaviors and scene actions
Memory	Long-term memory scoped to end users
Audio & Video	Microphone, camera, screen share controls
Error Handling	error, disconnect, serverResponse, retry patterns
Auth Tokens	Server-side token exchange for production
WebSocket Transport	Alternative transport for WebRTC-constrained environments
SSE Transport	Text-only interaction transport; the one that runs under Node
Character Versioning	Connect to a draft, latest, or tagged version; list, compare, release and promote versions
Multi-Character Rooms	Rosters, joinRoom, interaction targets, live roster updates, per-member attribution
Published Text Chat	Grant-based text-only sessions, <convai-chat>, lifecycle and API reference
Link Previews	Rendering links the character sends: previews, custom chat UIs, the scheme allowlist
Events Reference (local)	Every event with its typed payload, including the v2 and multi-character events

Package entry points

Import path	Contents
@convai/web-sdk	React hooks, components, and re-exported core types
@convai/web-sdk/react	Same as default (React-explicit alias)
@convai/web-sdk/vanilla	ConvaiClient, createConvaiWidget, AudioRenderer
@convai/web-sdk/core	Framework-agnostic ConvaiClient, managers, and all types
@convai/web-sdk/embed	Server-safe publication descriptor and grant helpers
@convai/web-sdk/embed/browser	Self-contained browser bundle that registers <convai-chat>
@convai/web-sdk/lipsync-helpers	Blendshape format utilities and queue helpers
@convai/web-sdk/vanilla/websocket	Opt-in. Registers the WebSocket transport. Import alongside /vanilla when using transport: "websocket".

Module format

The package is ESM. Every entry point above also resolves through the require and default conditions, so CommonJS tooling — Jest, ts-node, CJS bundlers — can resolve and mock it:

const { ConvaiClient } = require('@convai/web-sdk/core')

There is no separate CommonJS build; require() loads the ESM files directly, which Node supports from 22.12 onward. On older Node, import it or let your bundler handle the ESM.

WebSocket transport

The default transport is WebRTC (LiveKit). A WebSocket-based transport is available for environments where WebRTC is unavailable or undesirable.

// Add this import once (e.g. in your app entry file)
import "@convai/web-sdk/vanilla/websocket";
import { ConvaiClient } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  transport: "websocket",
});

The WebSocket packages (@pipecat-ai/client-js, @pipecat-ai/websocket-transport) are excluded from your bundle unless you import the /vanilla/websocket subpath — so LiveKit-only apps pay no bundle cost.

See the WebSocket Transport guide for the full feature comparison.

SSE interaction transport (beta)

Use the interaction API for text-only integrations that need a streamed response:

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  interactionApiUrl: "https://interaction-api-stg.convai.com/v1/interactions",
});

await client.connect();
client.sendUserTextMessage("Hello!");

This sends each message as an SSE POST with Authorization: Bearer .... Set transport: "sse" explicitly if preferred. SSE carries text only — no voice, video, or lipsync — and it never calls /connect, so connect() makes no network request and the session id arrives on the first interaction via the characterSessionId event.

Because it needs neither WebRTC nor a DOM, this is the only transport that runs outside a browser. Import @convai/web-sdk/core from Node, SSR, or a serverless function — not the root or /react entry points, which pull in browser-only dependencies.

The interaction API is currently reachable at staging only; a production host is planned. Full guide: SSE Interaction Transport.

Character versioning (beta)

Every character has an editable draft and, once released, immutable tagged versions (1.0, 1.1, …). A movable latest pointer names the release the runtime uses when a client connects without a selector. Pick the version to run with characterVersion:

characterVersion	Runs
(omitted)	The effective latest — how characters that predate versioning behave
"draft"	The editable draft, for testing unreleased changes
"latest"	The promoted latest release, explicitly
"1.2" / "1.2.3"	An immutable tagged version

const client = useConvaiClient({
  apiKey: "...",
  characterId: "...",
  characterVersion: "draft",
});

On the wire the selector is joined to the id as <uuid>-draft; client.characterId keeps returning the bare UUID, and client.characterReference returns the joined form.

Manage versions from the same client — client.characterVersions is available as soon as the config has an apiKey, before connecting:

const versions = client.characterVersions!;

const { has_unpublished_changes, draft_revision_id } = await versions.list();
const diff = await versions.diff("latest", "draft", { view: "semantic" });

await versions.create("1.1", { makeLatest: true }); // release the draft as a tag
await versions.promote("1.0");                      // roll back: latest → 1.0
await versions.discardDraft(draft_revision_id!);    // throw away unreleased edits

CharacterVersionManager can also be constructed standalone with an apiKey or a Convai personal access token; every method rejects with CharacterApiError (status, detail) on a non-2xx answer. Point characterApiUrl at https://api2-stg.convai.com to author against staging.

Explicit selectors are resolved by the runtime through the Character REST platform. Staging has this today; production answers explicit selectors with 503 until it is promoted there. Full guide: Character Versioning.

Multi-character rooms (beta)

Pass characters instead of characterId to open a room with several members. characters[0] answers first; the same character id may appear more than once and each entry is an independent member with its own membershipId. Roster rooms need the LiveKit transport and a nonblank endUserId.

const client = new ConvaiClient({
  apiKey: "YOUR_API_KEY",
  endUserId: "player-42",
  characters: [
    { characterId: GUIDE },     // initial active member
    { characterId: ASSESSOR },
    { characterId: GUIDE },     // a second, independent copy of the guide
  ],
  maxNumParticipants: 2,
});

client.on("characterReady", (member) => console.log("ready", member.membershipId));
client.on("characterStatus", ({ membershipId, status, failureCode }) => {
  if (status === "failed") console.warn(membershipId, failureCode);
});

await client.connect();

// Route the next user turns to another member; null closes every gate.
await client.setInteractionTarget(client.characters[1].membershipId);

// Change the roster on a live room.
await client.updateCharacterRoster({ add: [{ characterId: GUIDE }] });
await client.updateCharacterRoster({ remove: [client.characters[0].membershipId] });

// Another participant joins the same room without sending any topology.
await other.joinRoom({ roomSessionId: client.roomSession!.roomSessionId, endUserId: "player-7" });

Key points:

• characters and characterId are mutually exclusive, and roster entries take a bare character UUID (no -draft / -1.2 selectors).
• provisioningStatus describes dispatch only; a member is usable when its own characterReady arrives.
• Transcript rows, characterAudioTrack events and botReady payloads carry membershipId, so two copies of one character never merge.
• Room-topology failures throw ConvaiRoomError with status, code, requestTraceId and isRetryable; the SDK retries ROSTER_PROVISIONING_IN_PROGRESS for you (rosterProvisioningRetry).
• Action protocol v2 and actionConfig.tools are single-character only and cannot be combined with a roster.

Full reference: docs/convai_multi_character.md.

Vision dynamic context beta

Vision dynamic context is the default WebRTC/LiveKit vision path when enableVideo: true. Camera, screen, canvas, and custom video tracks can feed unified vision context; set visionInputConfig.enabled: false only when you need to keep the video channel while opting out.

import { useConvaiClient } from "@convai/web-sdk";

const client = useConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  enableVideo: true,
  visionInputConfig: {
    framesPerTurn: 12,
    bufferFrames: 30,
    samplingWindows: [
      { count: 6, intervalMs: 300 },
      { count: 6, intervalMs: 1000 },
    ],
    stalenessSeconds: 10,
    replacePreviousVisionContext: true,
  },
  respondModes: {
    vision: "silent",
    contextUpdate: "auto",
    sceneMetadata: "silent",
    trigger: "must_respond",
  },
});

Publish visual sources with semantic labels so acknowledgments can distinguish webcam, canvas, screen, and custom feeds:

await client.videoControls.enableVideo(); // webcam source

const handle = await client.videoControls.publishCanvas(canvas, {
  source: "canvas",
  name: "canvas-pov",
  fps: 1,
});

const statusId = client.visionStatus();
const triggerId = client.visionTrigger({
  respondMode: "silent",
  frameIndices: [-1, -1],
});

await client.videoControls.unpublishVisionSource(handle);

visionStatus() and visionTrigger() return an update id. Match it against serverResponse events for outcomes such as frames_available, buffer_empty, attached, or no_active_video.

Actions

Declare affordances in actionConfig, then handle the actionResponse event. Actions run in order; a target means it's parameterized (acts on an object/character).

client.on("actionResponse", ({ actions }: ActionResponseEvent) => {
  for (const { name, target } of actions) {
    target ? runParameterized(name, target) : runSimple(name); // e.g. "Move To"/"Cube" vs "Wave"
  }
});

Agentic actions and canonical model output (v2)

The v2 protocol is an explicit opt-in. Existing connections continue to receive the v1 actionResponse contract. A v2 client declares tools at connect time, dispatches only Core-validated items, and returns one correlated terminal result for every tool_call.

import { ConvaiClient, type ActionResult } from "@convai/web-sdk/core";

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  capabilities: {
    actionProtocolVersion: 2,
    modelOutputVersion: 2,
  },
  actionConfig: {
    actions: [],
    objects: [],
    characters: [],
    tools: [
      {
        name: "search_documents",
        description: "Search the user's approved document collection",
        inputSchema: {
          type: "object",
          properties: { query: { type: "string" } },
          required: ["query"],
        },
      },
    ],
  },
});

client.on("modelOutput", async (output) => {
  for (const item of output.items) {
    if (item.type === "message") renderAssistantText(item.content);
    if (item.type !== "tool_call") continue;

    let terminalResult: ActionResult;
    try {
      const result = await executeApprovedTool(item.name, item.arguments);
      terminalResult = {
        id: item.id,
        status: "completed",
        output: result,
      };
    } catch (error) {
      terminalResult = {
        id: item.id,
        status: "error",
        error: { message: String(error) },
      };
    }

    try {
      const ack = await client.sendActionResultAndWait(terminalResult, {
        timeoutMs: 10_000,
      });
      markResultReported(item.id, ack.idempotent);
    } catch (deliveryError) {
      markResultDeliveryUncertain(item.id, deliveryError);
      // If appropriate for your session lifecycle, retry `terminalResult`
      // unchanged. Never execute the tool again merely because its ACK was lost.
    }
  }
});

Parallel raw provider output (v2)

Model output v2 may provide a separate diagnostics-only raw stream alongside the unchanged bot-llm-text chat stream. Each chunk carries Core's stable logical-turn correlation id and, on newer Core versions, an invocation id:

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  capabilities: { modelOutputVersion: 2 },
});

client.on("botLlmTextRaw", ({ text, logical_turn_id, output_id }) => {
  appendRawDiagnostics(output_id ?? logical_turn_id, text);
});

The SDK ignores this channel unless Core negotiated modelOutputVersion: 2. Raw chunks never enter chatMessages, bot output, TTS, speech, or response lifecycle state. They are retained for diagnostics and must never be executed or reparsed. The ordinary bot-llm-text text and cadence remain unchanged. On v2 connections, ChatMessage.outputId matches the raw event's optional output_id and the final modelOutput.output_id; logicalTurnId/ logical_turn_id still groups every invocation in the user turn. Older v2 Core omits the invocation id, so applications can fall back to one turn-level raw aggregate. Never correlate by arrival order. When v2 is selected, the SDK suppresses the legacy actionResponse compatibility projection so an application cannot execute the same operation twice.

Developer-defined visual output uses data-only extension items. Applications register a renderer for a specific schema and version; unknown schemas use the server-provided text fallback or remain unhandled. The SDK never evaluates code from a model output.

import { ModelOutputExtensionRegistry } from "@convai/web-sdk/core";

const extensions = new ModelOutputExtensionRegistry<HTMLElement>();
extensions.register("com.example/status-card", 1, ({ payload }) =>
  renderStatusCard(payload),
);

Narrative Design template keys

Personalize Narrative Design section objectives per session. Placeholders like {player_name} in the graph are substituted from one key map — seed it at connect, replace it at runtime.

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...", // Narrative Design enabled
  narrativeTemplateKeys: { player_name: "Alex", quest_item: "oxygen generator" },
});

// Later — replaces the whole map; set before firing the next trigger
client.updateTemplateKeys({ player_name: "Alex", quest_item: "med kit" });
client.sendTriggerMessage("QuestUpdate");

Requires Narrative Design on the character; updateTemplateKeys is a full replace, not a merge. See the Context Management guide for dashboard setup and troubleshooting.

Character state of mind

Set a temporary generation mood that shapes tone and pacing for the next response. This is separate from enableEmotion, which reports the character's detected emotion after a turn.

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  stateOfMind: "anticipation",
});

// Mid-session. Affects the next response; does not trigger one.
client.updateEmotion("joy");
client.updateEmotion(null); // clear — "neutral" and "" clear it too

Values are trimmed and lowercased. The runtime value carries into the next connect, including a reconnect; an explicit stateOfMind in the connect config takes precedence.

Requires backend support — live on production as of 2026-08-20. On a backend without it, the server replies with an Unknown message type error on serverResponse. Because /connect silently ignores unknown fields, a successful connect alone does not prove state_of_mind was applied, so watch the ack:

client.on("serverResponse", (r) => {
  if (r.event_type === "update-emotion" && r.status !== "success") {
    console.warn("state of mind unsupported by this backend:", r.message);
  }
});

License

Licensed under the Apache License 2.0. Copyright 2025 Convai.