@convai/web-sdk

@convai/web-sdk

Real-time conversational AI characters for the web.

TypeScript-first SDK for embedding Convai AI characters into React and vanilla JS applications. Voice, text, lipsync, emotions, video, and screen share — all in one package.

Features

• React & vanilla JS — useConvaiClient hook, ConvaiWidget, and a framework-agnostic core
• Real-time audio/video — full-duplex WebRTC with echo cancellation, camera, and screen share
• Lipsync — ARKit and MetaHuman blendshape streams for facial animation
• Emotions — per-turn emotion detection with intensity scale
• Dynamic context & vision — inject text state, scene metadata, and LiveKit video frames mid-session
• Long-term memory — persistent cross-session memories scoped to each end user
• File upload — send images to the character during a live session
• WebSocket transport — opt-in alternative to WebRTC for constrained networks
• Auth tokens — server-side token exchange for production deployments

Installation

npm install @convai/web-sdk
# or
pnpm add @convai/web-sdk
# or
yarn add @convai/web-sdk

React peer dependencies: react and react-dom ^18 || ^19

Runtime requirement: secure context (https:// or http://localhost) for microphone/camera access.

Quick start

React

import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";

export function App() {
  const client = useConvaiClient({
    apiKey: import.meta.env.VITE_CONVAI_API_KEY,
    characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  });

  return <ConvaiWidget convaiClient={client} />;
}

Vanilla TypeScript

import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
});

const widget = createConvaiWidget(document.body, { convaiClient: client });

window.addEventListener("beforeunload", () => {
  widget.destroy();
  void client.disconnect();
});

Documentation

Full documentation is at docs.convai.com.

Guide	Description
Quick Start	First working integration in under 5 minutes
Configuration	All ConvaiConfig options
React Integration	useConvaiClient, ConvaiWidget, and React-specific patterns
Vanilla JS	ConvaiClient, createConvaiWidget, and audio setup
Events	Full event reference — botReady, stateChange, messagesChange, interactionCreated, and more
Context Management	Dynamic context, updateContext, file upload, session management
Emotions	Per-turn emotion detection with provider options
Lipsync	ARKit / MetaHuman blendshape streams and BlendshapeQueue API
Actions	Trigger character behaviors and scene actions
Memory	Long-term memory scoped to end users
Audio & Video	Microphone, camera, screen share controls
Error Handling	error, disconnect, serverResponse, retry patterns
Auth Tokens	Server-side token exchange for production
WebSocket Transport	Alternative transport for WebRTC-constrained environments

Package entry points

Import path	Contents
@convai/web-sdk	React hooks, components, and re-exported core types
@convai/web-sdk/react	Same as default (React-explicit alias)
@convai/web-sdk/vanilla	ConvaiClient, createConvaiWidget, AudioRenderer
@convai/web-sdk/core	Framework-agnostic ConvaiClient, managers, and all types
@convai/web-sdk/lipsync-helpers	Blendshape format utilities and queue helpers
@convai/web-sdk/vanilla/websocket	Opt-in. Registers the WebSocket transport. Import alongside /vanilla when using transport: "websocket".

WebSocket transport

The default transport is WebRTC (LiveKit). A WebSocket-based transport is available for environments where WebRTC is unavailable or undesirable.

// Add this import once (e.g. in your app entry file)
import "@convai/web-sdk/vanilla/websocket";
import { ConvaiClient } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
  apiKey: "...",
  characterId: "...",
  transport: "websocket",
});

The WebSocket packages (@pipecat-ai/client-js, @pipecat-ai/websocket-transport) are excluded from your bundle unless you import the /vanilla/websocket subpath — so LiveKit-only apps pay no bundle cost.

See the WebSocket Transport guide for the full feature comparison.

Vision dynamic context beta

Vision dynamic context is the default WebRTC/LiveKit vision path when enableVideo: true. Camera, screen, canvas, and custom video tracks can feed unified vision context; set visionInputConfig.enabled: false only when you need to keep the video channel while opting out.

import { useConvaiClient } from "@convai/web-sdk";

const client = useConvaiClient({
  apiKey: import.meta.env.VITE_CONVAI_API_KEY,
  characterId: import.meta.env.VITE_CONVAI_CHARACTER_ID,
  enableVideo: true,
  visionInputConfig: {
    framesPerTurn: 12,
    bufferFrames: 30,
    samplingWindows: [
      { count: 6, intervalMs: 300 },
      { count: 6, intervalMs: 1000 },
    ],
    stalenessSeconds: 10,
    replacePreviousVisionContext: true,
  },
  respondModes: {
    vision: "silent",
    contextUpdate: "auto",
    sceneMetadata: "silent",
    trigger: "must_respond",
  },
});

Publish visual sources with semantic labels so acknowledgments can distinguish webcam, canvas, screen, and custom feeds:

await client.videoControls.enableVideo(); // webcam source

const handle = await client.videoControls.publishCanvas(canvas, {
  source: "canvas",
  name: "canvas-pov",
  fps: 1,
});

const statusId = client.visionStatus();
const triggerId = client.visionTrigger({
  respondMode: "silent",
  frameIndices: [-1, -1],
});

await client.videoControls.unpublishVisionSource(handle);

visionStatus() and visionTrigger() return an update id. Match it against serverResponse events for outcomes such as frames_available, buffer_empty, attached, or no_active_video.

After the beta package is published, install it with:

npm install @convai/web-sdk@beta

Actions

Declare affordances in actionConfig, then handle the actionResponse event. Actions run in order; a target means it's parameterized (acts on an object/character).

client.on("actionResponse", ({ actions }: ActionResponseEvent) => {
  for (const { name, target } of actions) {
    target ? runParameterized(name, target) : runSimple(name); // e.g. "Move To"/"Cube" vs "Wave"
  }
});

License

Licensed under the Apache License 2.0. Copyright 2025 Convai.