@convai/web-sdk

@convai/web-sdk

Real-time conversational AI characters for the web.

TypeScript-first SDK for embedding Convai AI characters into React and vanilla JS applications. Voice, text, lipsync, emotions, video, and screen share — all in one package.

What's new in 1.7.0

• Character state of mind — set a temporary generation mood with the stateOfMind config option, and change it mid-session with updateEmotion() without triggering a response. Usage below.
• Send-ahead enabled by default — NeuroSync lipsync ahead-delivery is now on by default, reversing the 1.6.0 opt-in. Fall back to the legacy paced path with blendshapeConfig.deliver_chunks_ahead: false.
• Lipsync naturalness pipeline — modular naturalness processing with a tuned MetaHuman profile.
• Adaptive glass widget styling — ConvaiWidget adapts to light and dark backgrounds.

Previously in 1.6.0

• Narrative Design template keys — personalize one narrative graph per session: seed values at connect with the narrativeTemplateKeys config option, replace them mid-session with updateTemplateKeys(). Usage below.
• Typed parameterized actions — actionResponse is now typed via the exported ConvaiAction / ActionResponseEvent, including the target of parameterized actions. Usage below.
• Vision dynamic context — camera, screen, canvas, and custom tracks feed unified vision context on WebRTC, with hardened WebSocket vision handling. Usage below.

Features

• React & vanilla JS — useConvaiClient hook, ConvaiWidget, and a framework-agnostic core
• Real-time audio/video — full-duplex WebRTC with echo cancellation, camera, and screen share
• Lipsync — ARKit and MetaHuman blendshape streams for facial animation
• Emotions & state of mind — per-turn emotion detection with intensity scale, plus a settable generation mood
• Dynamic context & vision — inject text state, scene metadata, and LiveKit video frames mid-session
• Actions & Narrative Design — typed action decisions with parameterized targets, named triggers, and per-session template keys
• Long-term memory — persistent cross-session memories scoped to each end user
• File upload — send images to the character during a live session
• WebSocket transport — opt-in alternative to WebRTC for constrained networks
• Auth tokens — server-side token exchange for production deployments

Installation

npm install @convai/web-sdk
# or
pnpm add @convai/web-sdk
# or
yarn add @convai/web-sdk

React peer dependencies: react and react-dom ^18 || ^19

Runtime requirement: secure context (https:// or http://localhost) for microphone/camera access.

Quick start

React

import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";

export function App() {
  const client = useConvaiClient({
    apiKey: import.meta.env.VITE_CONVAI_API_KEY,
    characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  });

  return <ConvaiWidget convaiClient={client} />;
}

Vanilla TypeScript

import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});

const widget = createConvaiWidget(document.body, { convaiClient: client });

window.addEventListener("beforeunload", () => {
  widget.destroy();
  void client.disconnect();
});

Documentation

Full documentation is at docs.convai.com.

Guide	Description
Quick Start	First working integration in under 5 minutes
Configuration	All ConvaiConfig options
React Integration	useConvaiClient, ConvaiWidget, and React-specific patterns
Vanilla JS	ConvaiClient, createConvaiWidget, and audio setup
Events	Full event reference — botReady, stateChange, messagesChange, interactionCreated, and more
Context Management	Dynamic context, updateContext, file upload, session management
Emotions	Per-turn emotion detection, provider options, and stateOfMind / updateEmotion()
Lipsync	ARKit / MetaHuman blendshape streams and BlendshapeQueue API
Actions	Trigger character behaviors and scene actions
Memory	Long-term memory scoped to end users
Audio & Video	Microphone, camera, screen share controls
Error Handling	error, disconnect, serverResponse, retry patterns
Auth Tokens	Server-side token exchange for production
WebSocket Transport	Alternative transport for WebRTC-constrained environments
SSE Transport	Text-only interaction transport; the one that runs under Node

Package entry points

Import path	Contents
@convai/web-sdk	React hooks, components, and re-exported core types
@convai/web-sdk/react	Same as default (React-explicit alias)
@convai/web-sdk/vanilla	ConvaiClient, createConvaiWidget, AudioRenderer
@convai/web-sdk/core	Framework-agnostic ConvaiClient, managers, and all types
@convai/web-sdk/lipsync-helpers	Blendshape format utilities and queue helpers
@convai/web-sdk/vanilla/websocket	Opt-in. Registers the WebSocket transport. Import alongside /vanilla when using transport: "websocket".

WebSocket transport

The default transport is WebRTC (LiveKit). A WebSocket-based transport is available for environments where WebRTC is unavailable or undesirable.

// Add this import once (e.g. in your app entry file)
import "@convai/web-sdk/vanilla/websocket";
import { ConvaiClient } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  transport: "websocket",
});

The WebSocket packages (@pipecat-ai/client-js, @pipecat-ai/websocket-transport) are excluded from your bundle unless you import the /vanilla/websocket subpath — so LiveKit-only apps pay no bundle cost.

See the WebSocket Transport guide for the full feature comparison.

SSE interaction transport (beta)

Use the interaction API for text-only integrations that need a streamed response:

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  interactionApiUrl: "https://interaction-api-stg.convai.com/v1/interactions",
});

await client.connect();
client.sendUserTextMessage("Hello!");

This sends each message as an SSE POST with Authorization: Bearer .... Set transport: "sse" explicitly if preferred. SSE carries text only — no voice, video, or lipsync — and it never calls /connect, so connect() makes no network request and the session id arrives on the first interaction via the characterSessionId event.

Because it needs neither WebRTC nor a DOM, this is the only transport that runs outside a browser. Import @convai/web-sdk/core from Node, SSR, or a serverless function — not the root or /react entry points, which pull in browser-only dependencies.

The interaction API is currently reachable at staging only; a production host is planned. Full guide: SSE Interaction Transport.

Vision dynamic context beta

Vision dynamic context is the default WebRTC/LiveKit vision path when enableVideo: true. Camera, screen, canvas, and custom video tracks can feed unified vision context; set visionInputConfig.enabled: false only when you need to keep the video channel while opting out.

import { useConvaiClient } from "@convai/web-sdk";

const client = useConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  enableVideo: true,
  visionInputConfig: {
    framesPerTurn: 12,
    bufferFrames: 30,
    samplingWindows: [
      { count: 6, intervalMs: 300 },
      { count: 6, intervalMs: 1000 },
    ],
    stalenessSeconds: 10,
    replacePreviousVisionContext: true,
  },
  respondModes: {
    vision: "silent",
    contextUpdate: "auto",
    sceneMetadata: "silent",
    trigger: "must_respond",
  },
});

Publish visual sources with semantic labels so acknowledgments can distinguish webcam, canvas, screen, and custom feeds:

await client.videoControls.enableVideo(); // webcam source

const handle = await client.videoControls.publishCanvas(canvas, {
  source: "canvas",
  name: "canvas-pov",
  fps: 1,
});

const statusId = client.visionStatus();
const triggerId = client.visionTrigger({
  respondMode: "silent",
  frameIndices: [-1, -1],
});

await client.videoControls.unpublishVisionSource(handle);

visionStatus() and visionTrigger() return an update id. Match it against serverResponse events for outcomes such as frames_available, buffer_empty, attached, or no_active_video.

Actions

Declare affordances in actionConfig, then handle the actionResponse event. Actions run in order; a target means it's parameterized (acts on an object/character).

client.on("actionResponse", ({ actions }: ActionResponseEvent) => {
  for (const { name, target } of actions) {
    target ? runParameterized(name, target) : runSimple(name); // e.g. "Move To"/"Cube" vs "Wave"
  }
});

Narrative Design template keys

Personalize Narrative Design section objectives per session. Placeholders like {player_name} in the graph are substituted from one key map — seed it at connect, replace it at runtime.

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...", // Narrative Design enabled
  narrativeTemplateKeys: { player_name: "Alex", quest_item: "oxygen generator" },
});

// Later — replaces the whole map; set before firing the next trigger
client.updateTemplateKeys({ player_name: "Alex", quest_item: "med kit" });
client.sendTriggerMessage("QuestUpdate");

Requires Narrative Design on the character; updateTemplateKeys is a full replace, not a merge. See the Context Management guide for dashboard setup and troubleshooting.

Character state of mind

Set a temporary generation mood that shapes tone and pacing for the next response. This is separate from enableEmotion, which reports the character's detected emotion after a turn.

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  stateOfMind: "anticipation",
});

// Mid-session. Affects the next response; does not trigger one.
client.updateEmotion("joy");
client.updateEmotion(null); // clear — "neutral" and "" clear it too

Values are trimmed and lowercased. The runtime value carries into the next connect, including a reconnect; an explicit stateOfMind in the connect config takes precedence.

Requires backend support — live on production as of 2026-08-20. On a backend without it, the server replies with an Unknown message type error on serverResponse. Because /connect silently ignores unknown fields, a successful connect alone does not prove state_of_mind was applied, so watch the ack:

client.on("serverResponse", (r) => {
  if (r.event_type === "update-emotion" && r.status !== "success") {
    console.warn("state of mind unsupported by this backend:", r.message);
  }
});

License

Licensed under the Apache License 2.0. Copyright 2025 Convai.