@convai/web-sdk

@convai/web-sdk

Real-time conversational AI characters for the web.

TypeScript-first SDK for embedding Convai AI characters into React and vanilla JS applications. Voice, text, lipsync, emotions, video, and screen share — all in one package.

What's new in 1.8.0

• Protocol capabilities, opted into one at a time — capabilities selects v2 correlated client tool calls, the canonical Core-validated modelOutput envelope, and raw bot-llm-text streaming, each independent of the others. Core must confirm a v2 capability or the SDK refuses the connection, so a half-upgraded backend fails at connect rather than mid-conversation. Omit the block entirely and nothing changes: existing connections and one-argument text messages stay on the v1 wire contract. Options table.
• Correlated action results and logical turns — action results carry the call they answer, and turn correlation holds across multi-envelope and voice turns, so a response can be tied to its cause without guessing from arrival order.
• Extension renderer registry — render custom model-output extensions without the SDK knowing your UI framework.
• Published chat without a credential — a text-only connection method for published-chat launch grants, plus a native custom-element browser bundle at dist/embed/chat-embed-v1.js for iframe-free embeds. The embed never receives a publisher API key. Usage below.
• Loads in Node without a bundler — emitted ESM carries explicit relative extensions, so the published package and the Chat Publish client resolve under plain Node resolution.

Previously in 1.7.0

• Character state of mind — set a temporary generation mood with the stateOfMind config option, and change it mid-session with updateEmotion() without triggering a response. Usage below.
• Send-ahead enabled by default — NeuroSync lipsync ahead-delivery is now on by default, reversing the 1.6.0 opt-in. Fall back to the legacy paced path with blendshapeConfig.deliver_chunks_ahead: false.
• Lipsync naturalness pipeline — modular naturalness processing with a tuned MetaHuman profile.
• Adaptive glass widget styling — ConvaiWidget adapts to light and dark backgrounds.

Previously in 1.6.0

• Narrative Design template keys — personalize one narrative graph per session: seed values at connect with the narrativeTemplateKeys config option, replace them mid-session with updateTemplateKeys(). Usage below.
• Typed parameterized actions — actionResponse is now typed via the exported ConvaiAction / ActionResponseEvent, including the target of parameterized actions. Usage below.
• Vision dynamic context — camera, screen, canvas, and custom tracks feed unified vision context on WebRTC, with hardened WebSocket vision handling. Usage below.

Features

• React & vanilla JS — useConvaiClient hook, ConvaiWidget, and a framework-agnostic core
• Real-time audio/video — full-duplex WebRTC with echo cancellation, camera, and screen share
• Lipsync — ARKit and MetaHuman blendshape streams for facial animation
• Emotions & state of mind — per-turn emotion detection with intensity scale, plus a settable generation mood
• Dynamic context & vision — inject text state, scene metadata, and LiveKit video frames mid-session
• Actions & Narrative Design — typed action decisions with parameterized targets, named triggers, and per-session template keys
• Long-term memory — persistent cross-session memories scoped to each end user
• File upload — send images to the character during a live session
• WebSocket transport — opt-in alternative to WebRTC for constrained networks
• Auth tokens — server-side token exchange for production deployments

Installation

npm install @convai/web-sdk
# or
pnpm add @convai/web-sdk
# or
yarn add @convai/web-sdk

React peer dependencies: react and react-dom ^18 || ^19

Runtime requirement: secure context (https:// or http://localhost) for microphone/camera access.

Quick start

React

import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";

export function App() {
  const client = useConvaiClient({
    apiKey: import.meta.env.VITE_CONVAI_API_KEY,
    characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  });

  return <ConvaiWidget convaiClient={client} />;
}

Vanilla TypeScript

import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});

const widget = createConvaiWidget(document.body, { convaiClient: client });

window.addEventListener("beforeunload", () => {
  widget.destroy();
  void client.disconnect();
});

Published text chat

Published chat pages use a short-lived launch grant instead of the publisher's API key. Native Web SDK integration is the recommended embed path; an iframe is not required. Obtain the grant from the public Character API and pass it to the core client:

import { ConvaiClient } from '@convai/web-sdk/core';
import { requestPublishedChatGrant } from '@convai/web-sdk/embed';

const launchToken = await requestPublishedChatGrant(
  `${characterApiUrl}/chat-publications/${publicationId}/launch-grants`,
);
const client = new ConvaiClient();
const unsubscribe = client.on('botReady', () => {
  unsubscribe();
  client.sendUserTextMessage('Hello!');
});
await client.connectWithPublicationGrant(launchToken);

The grant is single-use and is not retained for reconnect. Request a new grant for each new session. The SDK attaches a browser-generated logical attempt ID and retries one transient grant exchange with that same ID, so a response lost after server-side consumption does not strand the token or create a duplicate session. An application performing its own retry may pass the same connectAttemptId option with the same launch token. Published chat is text-only; the server-issued room token does not permit publishing microphone or camera tracks.

The @convai/web-sdk/embed browser element creates its own per-browser requester identifier. If a server or SSR integration calls its published-chat grant helpers directly, it must pass a caller-scoped requester identifier; do not reuse a process-wide identifier across visitors.

When using ConvaiWidget, provide requestPublicationGrant so Reset, Disconnect, and later reopen operations mint a new grant instead of attempting to reuse the consumed token:

<ConvaiWidget
  convaiClient={client}
  publishedChatDescriptor={{
    display_name: publication.display_name,
    image_url: publication.image_url,
  }}
  requestPublicationGrant={async () => {
    const response = await fetch(`/api/chat/${publicationId}/grant`, {
      method: 'POST',
    });
    if (!response.ok) throw new Error('Unable to relaunch published chat');
    return (await response.json()).launch_token;
  }}
/>

For copy/paste installation without an application build step, deploy the versioned browser artifact at dist/embed/chat-embed-v1.js and use its native custom element:

<script
  type="module"
  src="https://cdn.jsdelivr.net/npm/@convai/web-sdk@latest/dist/embed/chat-embed-v1.js"
></script>
<convai-chat publication-id="PUBLICATION_UUID"></convai-chat>

The element renders directly in the host page with Shadow DOM style isolation, automatically mints a fresh grant per connection, and emits convai-ready, convai-disconnect, and convai-error DOM events. The production bundle uses https://api.convai.com and https://realtime-api.convai.com only as defaults for a hand-written SaaS integration. Platform-generated snippets always emit the deployment's public Character API and Core endpoints explicitly, so an on-prem or non-production embed never falls through to another deployment and still requires no credential:

<convai-chat
  publication-id="PUBLICATION_UUID"
  publication-api-url="https://character-api.example.com"
  core-url="https://realtime.example.com"
></convai-chat>

Descriptor and grant endpoints must allow credential-free cross-origin GET and POST requests. The embed never receives a publisher API key, character configuration, or reusable launch token.

For customer-managed and GovCloud deployments, the repository also builds a browser-only image from deploy/published-chat/Dockerfile. It serves the exact released bundle at /sdk/chat-embed-v1.js on port 8080, plus /healthz, and contains no API key or cloud identity. The manual build-onprem workflow requires an immutable semver tag matching package.json, refuses to overwrite an existing Docker Hub tag, and publishes both amd64 and arm64 images. Pin the resulting digest in the deployment repository; do not make an on-prem browser depend on jsDelivr, npm availability, GCP workload identity, or the Experience application image.

Documentation

Full documentation is at docs.convai.com.

Guide	Description
Quick Start	First working integration in under 5 minutes
Configuration	All ConvaiConfig options
React Integration	useConvaiClient, ConvaiWidget, and React-specific patterns
Vanilla JS	ConvaiClient, createConvaiWidget, and audio setup
Events	Full event reference — botReady, stateChange, messagesChange, interactionCreated, and more
Context Management	Dynamic context, updateContext, file upload, session management
Emotions	Per-turn emotion detection, provider options, and stateOfMind / updateEmotion()
Lipsync	ARKit / MetaHuman blendshape streams and BlendshapeQueue API
Actions	Trigger character behaviors and scene actions
Memory	Long-term memory scoped to end users
Audio & Video	Microphone, camera, screen share controls
Error Handling	error, disconnect, serverResponse, retry patterns
Auth Tokens	Server-side token exchange for production
WebSocket Transport	Alternative transport for WebRTC-constrained environments

Package entry points

Import path	Contents
@convai/web-sdk	React hooks, components, and re-exported core types
@convai/web-sdk/react	Same as default (React-explicit alias)
@convai/web-sdk/vanilla	ConvaiClient, createConvaiWidget, AudioRenderer
@convai/web-sdk/core	Framework-agnostic ConvaiClient, managers, and all types
@convai/web-sdk/embed	Server-safe publication descriptor and grant helpers
@convai/web-sdk/embed/browser	Self-contained browser bundle that registers <convai-chat>
@convai/web-sdk/lipsync-helpers	Blendshape format utilities and queue helpers
@convai/web-sdk/vanilla/websocket	Opt-in. Registers the WebSocket transport. Import alongside /vanilla when using transport: "websocket".

WebSocket transport

The default transport is WebRTC (LiveKit). A WebSocket-based transport is available for environments where WebRTC is unavailable or undesirable.

// Add this import once (e.g. in your app entry file)
import "@convai/web-sdk/vanilla/websocket";
import { ConvaiClient } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  transport: "websocket",
});

The WebSocket packages (@pipecat-ai/client-js, @pipecat-ai/websocket-transport) are excluded from your bundle unless you import the /vanilla/websocket subpath — so LiveKit-only apps pay no bundle cost.

See the WebSocket Transport guide for the full feature comparison.

Vision dynamic context beta

Vision dynamic context is the default WebRTC/LiveKit vision path when enableVideo: true. Camera, screen, canvas, and custom video tracks can feed unified vision context; set visionInputConfig.enabled: false only when you need to keep the video channel while opting out.

import { useConvaiClient } from "@convai/web-sdk";

const client = useConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  enableVideo: true,
  visionInputConfig: {
    framesPerTurn: 12,
    bufferFrames: 30,
    samplingWindows: [
      { count: 6, intervalMs: 300 },
      { count: 6, intervalMs: 1000 },
    ],
    stalenessSeconds: 10,
    replacePreviousVisionContext: true,
  },
  respondModes: {
    vision: "silent",
    contextUpdate: "auto",
    sceneMetadata: "silent",
    trigger: "must_respond",
  },
});

Publish visual sources with semantic labels so acknowledgments can distinguish webcam, canvas, screen, and custom feeds:

await client.videoControls.enableVideo(); // webcam source

const handle = await client.videoControls.publishCanvas(canvas, {
  source: "canvas",
  name: "canvas-pov",
  fps: 1,
});

const statusId = client.visionStatus();
const triggerId = client.visionTrigger({
  respondMode: "silent",
  frameIndices: [-1, -1],
});

await client.videoControls.unpublishVisionSource(handle);

visionStatus() and visionTrigger() return an update id. Match it against serverResponse events for outcomes such as frames_available, buffer_empty, attached, or no_active_video.

Actions

Declare affordances in actionConfig, then handle the actionResponse event. Actions run in order; a target means it's parameterized (acts on an object/character).

client.on("actionResponse", ({ actions }: ActionResponseEvent) => {
  for (const { name, target } of actions) {
    target ? runParameterized(name, target) : runSimple(name); // e.g. "Move To"/"Cube" vs "Wave"
  }
});

Agentic actions and canonical model output (v2)

The v2 protocol is an explicit opt-in. Existing connections continue to receive the v1 actionResponse contract. A v2 client declares tools at connect time, dispatches only Core-validated items, and returns one correlated terminal result for every tool_call.

import { ConvaiClient } from "@convai/web-sdk/core";

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  capabilities: {
    actionProtocolVersion: 2,
    modelOutputVersion: 2,
  },
  actionConfig: {
    actions: [],
    objects: [],
    characters: [],
    tools: [
      {
        name: "search_documents",
        description: "Search the user's approved document collection",
        inputSchema: {
          type: "object",
          properties: { query: { type: "string" } },
          required: ["query"],
        },
      },
    ],
  },
});

client.on("modelOutput", async (output) => {
  for (const item of output.items) {
    if (item.type === "message") renderAssistantText(item.content);
    if (item.type !== "tool_call") continue;

    try {
      const result = await executeApprovedTool(item.name, item.arguments);
      client.sendActionResult({
        id: item.id,
        status: "completed",
        output: result,
      });
    } catch (error) {
      client.sendActionResult({
        id: item.id,
        status: "error",
        error: { message: String(error) },
      });
    }
  }
});

Raw provider output in the existing text stream

To display every user-visible provider output chunk before Core parses actions or display markup, explicitly select raw bot-llm-text mode:

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  capabilities: { botLlmTextMode: "raw" },
});

client.on("message", (message) => {
  // The existing streaming message event now carries the raw bot-llm-text
  // chunks in provider order. Keep this display stream separate from speech.
  renderChat(message);
});

Omitting botLlmTextMode preserves the legacy request and filtered text behavior. This capability is independent of actionProtocolVersion and modelOutputVersion.

raw is retained for diagnostics and must never be executed or reparsed. When v2 is selected, the SDK suppresses the legacy actionResponse compatibility projection so an application cannot execute the same operation twice.

Developer-defined visual output uses data-only extension items. Applications register a renderer for a specific schema and version; unknown schemas use the server-provided text fallback or remain unhandled. The SDK never evaluates code from a model output.

import { ModelOutputExtensionRegistry } from "@convai/web-sdk/core";

const extensions = new ModelOutputExtensionRegistry<HTMLElement>();
extensions.register("com.example/status-card", 1, ({ payload }) =>
  renderStatusCard(payload),
);

Narrative Design template keys

Personalize Narrative Design section objectives per session. Placeholders like {player_name} in the graph are substituted from one key map — seed it at connect, replace it at runtime.

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...", // Narrative Design enabled
  narrativeTemplateKeys: { player_name: "Alex", quest_item: "oxygen generator" },
});

// Later — replaces the whole map; set before firing the next trigger
client.updateTemplateKeys({ player_name: "Alex", quest_item: "med kit" });
client.sendTriggerMessage("QuestUpdate");

Requires Narrative Design on the character; updateTemplateKeys is a full replace, not a merge. See the Context Management guide for dashboard setup and troubleshooting.

Character state of mind

Set a temporary generation mood that shapes tone and pacing for the next response. This is separate from enableEmotion, which reports the character's detected emotion after a turn.

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  stateOfMind: "anticipation",
});

// Mid-session. Affects the next response; does not trigger one.
client.updateEmotion("joy");
client.updateEmotion(null); // clear — "neutral" and "" clear it too

Values are trimmed and lowercased. The runtime value carries into the next connect, including a reconnect; an explicit stateOfMind in the connect config takes precedence.

Requires backend support — live on production as of 2026-08-20. On a backend without it, the server replies with an Unknown message type error on serverResponse. Because /connect silently ignores unknown fields, a successful connect alone does not prove state_of_mind was applied, so watch the ack:

client.on("serverResponse", (r) => {
  if (r.event_type === "update-emotion" && r.status !== "success") {
    console.warn("state of mind unsupported by this backend:", r.message);
  }
});

License

Licensed under the Apache License 2.0. Copyright 2025 Convai.