@unith-ai/react

Unith React SDK

A React hooks library for building complex digital human experiences that run on Unith AI.

Prerequisite

Before proceeding with using this library, you're expected to have an account on Unith AI, create a digital human and take note of your API key. You can create an account here in minutes!

Installation

Install the package in your project through package manager.

npm install @unith-ai/react
# or
yarn add @unith-ai/react
# or
pnpm install @unith-ai/react

Azure Speech SDK (Optional)

If using microphoneProvider: "azure", install the Azure Speech SDK:

npm install microsoft-cognitiveservices-speech-sdk

The SDK is dynamically imported only when the Azure provider is used, so it won't affect bundle size if you use a different provider like eleven_labs.

Usage

This library provides React hooks for integrating Unith AI digital humans into your React applications.

useConversation Hook

The useConversation hook manages the digital human conversation state and provides methods to control the session.

import { useConversation } from "@unith-ai/react";

function MyComponent() {
  const conversation = useConversation({
    orgId: "your-org-id",
    headId: "your-head-id",
    apiKey: "your-api-key",
  });

  // Use conversation methods and state
}

Configuration

The hook accepts a configuration object with the following properties:

Required Parameters

• orgId - Your organization ID
• headId - The digital human head ID to use
• apiKey - API key for authentication

Optional Parameters

• mode - Conversation mode (default: "default")
• language - Language code for the conversation (default: browser language)
• allowWakeLock - Prevent screen from sleeping during conversation (default: true)
• microphoneProvider - Provider for the microphone - "azure" | "eleven_labs"
• voiceInterruptions - Flag to enable response interruption when voice signal is recognized. - boolean (default: false)
• dataTag - string Custom data to be used for analytics.
• username - string Custom identifier for user.
• microphoneEvents - Callbacks for microphone events
  • onMicrophoneSpeechRecognitionResult ({ transcript: string }) - Called when microphone recognizes your user's speech. This returns a transcript. Ideal behavior is to call the .sendMessage method with your transcript as microphone doesn't automatically commit / send users text to our AI.
  • onMicrophonePartialSpeechRecognitionResult () - Called when microphone recognizes that the user is trying to speak. This doesn't returns a transcript. Ideal behavior is to check if digital human is currently responding (speaking) and trigger the stop response method (.stopResponse).
  • onMicrophoneStatusChange ({status}) Called when microphone status changes
    • status "ON" | "OFF" | "PROCESSING" Shows current status of microphone.
  • onMicrophoneError ({ message: string }) - Called when microphone has an error with the error message.
• elevenLabsOptions ElevenLabsOptions - Custom options to customize behavior of ElevenLabs STT provider
  • noiseSuppression Boolean
  • vadSilenceThresholdSecs Number
  • vadThreshold Number
  • minSpeechDurationMs Number
  • minSilenceDurationMs Number
  • disableDynamicSpeechRecognition Boolean - This disables ElevenLabs dynamic speech recognition language detection and uses the language specified during digital human creation.

Returned Values

The hook returns an object containing methods and state:

Methods

• startDigitalHuman(element, options?) - Initialize and start the digital human
  • element HTMLElement - DOM element where the video will be rendered
  • options Partial<ConversationEvents> - Optional event callbacks
  • Returns: Promise<string | undefined> - The user ID
• getBackgroundVideo() - Retrieve the idle background video URL
  • Returns: Promise<string> - Video URL
• startSession() - Start the conversation session and begin audio & video playback
  • Returns: Promise<void>
• sendMessage(text) - Send a text message to the digital human
  • text string - Message text to send
  • Returns: Promise<void>
• toggleMute() - Toggle the mute status of the audio output
  • Returns: number | undefined - New volume (0 for muted, 1 for unmuted)
• keepSession() - Send keep-alive event to prevent session timeout
  • Returns: Promise<void>
• toggleMicrophone() - Toggles microphone status from OFF to ON & vice versa.
  • Returns: Promise<void>
• getUserId() - Get the current user's ID
  • Returns: string | undefined
• endSession() - End the conversation session and clean up resources
  • Returns: Promise<void>
• stopResponse() - Stops the ongoing response and notifies you via two callbacks:
  • onStoppingStart() - Called immediately when stop is initiated.
  • onStoppingEnd() - Called once the current response is stopped.

State

• status "connecting" | "connected" | "disconnecting" | "disconnected" - Current WebSocket connection status
• isConnected boolean - True if status is "connected"
• isDisconnected boolean - True if status is "disconnected"
• isNotConnected boolean - True if status is not "connected"
• sessionStarted boolean - True if session has been started
• mode "listening" | "speaking" | "thinking" | "stopping" - Current conversation mode
• isSpeaking boolean - True if mode is "speaking"
• messages MessageEventData[] - Array of conversation messages
• suggestions string[] - Array of suggestion strings.
• messageCounter number - Count of messages sent
• userId string | null - Current user's unique identifier
• headInfo ConnectHeadType | null - Information about the digital human
  • name string - Digital human head name
  • phrases string[] - Array with phrases set during digital human creation
  • language string - Language code setup during digital human creation
  • avatar string - Static image URL for digital human
• microphoneAccess boolean - True if microphone access was granted
• isMuted boolean - True if audio is muted
• timeOutWarning boolean - True when session timeout warning is active
• timeOutBanner boolean - True when session has timed out
• capacityError boolean - True if a capacity error occurred

Basic Example

import { useConversation } from "@unith-ai/react";
import { useRef, useEffect } from "react";

function DigitalHumanChat() {
  const videoRef = useRef(null);
  const conversation = useConversation({
    orgId: "your-org-id",
    headId: "your-head-id",
    apiKey: "your-api-key",
  });

  useEffect(() => {
    if (videoRef.current) {
      conversation.startDigitalHuman(videoRef.current, {
        onConnect: ({ userId, headInfo, microphoneAccess }) => {
          console.log("Connected:", userId);
        },
        onMessage: ({ timestamp, sender, text, visible }) => {
          console.log("Message:", text);
        },
        onError: ({ message, endConversation, type }) => {
          console.error("Error:", message);
        },
      });
    }
  }, []);

  const handleSendMessage = () => {
    conversation.sendMessage("Hello!");
  };

  const handleStartSession = () => {
    conversation.startSession();
  };

  return (
    <div>
      <div ref={videoRef} style={{ width: "100%", height: "500px" }} />

      {conversation.isConnected && !conversation.sessionStarted && (
        <button onClick={handleStartSession}>Start Conversation</button>
      )}

      {conversation.sessionStarted && (
        <button onClick={handleSendMessage}>Send Message</button>
      )}

      <div>
        {conversation.messages.map((msg, index) => (
          <div key={index}>
            <strong>{msg.sender}:</strong> {msg.text}
          </div>
        ))}
      </div>
    </div>
  );
}

Advanced Example with Event Callbacks

import { useConversation } from "@unith-ai/react";
import { useRef, useEffect, useState } from "react";

function AdvancedChat() {
  const videoRef = useRef(null);
  const conversationRef = useRef<ReturnType<typeof useConversation> | null>(null);
  const [inputText, setInputText] = useState("");
  const [micStatus, setMicStatus] = useState("OFF");

  const conversation = useConversation({
    orgId: "your-org-id",
    headId: "your-head-id",
    apiKey: "your-api-key",
    mode: "default",
    language: "en-US",
    username: "test-username"
  });

  useEffect(() => {
    conversationRef.current = conversation;
  }, [conversation]);

  useEffect(() => {
    if (videoRef.current && conversationRef.current) {
      conversationRef.current.startDigitalHuman(videoRef.current, {
        microphoneProvider: 'eleven_labs',
        voiceInterruptions: true,
        microphoneEvents: {
            onMicrophoneError(prop) {
                console.log(`Microphone error: ${prop.message}`);
            },
            onMicrophoneSpeechRecognitionResult(prop) {
                conversationRef.current.sendMessage(prop.transcript);
            },
            onMicrophonePartialSpeechRecognitionResult(){
                conversationRef.current.stopResponse();
            },
            onMicrophoneStatusChange(prop) {
                console.log('Microphone status changed:', prop.status);
                setMicStatus(prop.status);
            },
        },
        onConnect: ({ userId, headInfo, microphoneAccess }) => {
          console.log("Connected with user ID:", userId);
          console.log("Digital human:", headInfo.name);
        },
        onMessage: ({ timestamp, sender, text, visible }) => {
          console.log(`[${sender}] ${text}`);
        },
        onSpeakingStart: () => {
          console.log("Digital human started speaking");
        },
        onSpeakingEnd: () => {
          console.log("Digital human finished speaking");
        },
        onSuggestions: ({ suggestions }) => {
          console.log({ suggestions });
        },
        onTimeoutWarning: () => {
          console.log("Session will timeout soon");
        },
        onTimeout: () => {
          console.log("Session timed out");
        },
        onError: ({ message, type }) => {
          if (type === "toast") {
            alert(message);
          }
        },
        onStoppingStart: () => {
          console.log("stop started");
        },
        onStoppingEnd: () => {
          console.log("response stopped.");
        }
      });
    }
  }, []);

  const handleSendMessage = async e => {
    e.preventDefault();
    if (inputText.trim()) {
      await conversation.sendMessage(inputText);
      setInputText("");
    }
  };

  const handleKeepSession = () => {
    conversation.keepSession();
  };

  return (
    <div>
      <div ref={videoRef} style={{ width: "100%", height: "500px" }} />

      <div>
        <p>Status: {conversation.status}</p>
        <p>Mode: {conversation.mode}</p>
        {conversation.isSpeaking && <p>Digital human is speaking...</p>}
      </div>

      {conversation.isConnected && !conversation.sessionStarted && (
        <button onClick={() => conversation.startSession()}>
          Start Conversation
        </button>
      )}

      {conversation.timeOutWarning && (
        <div>
          <p>Your session will timeout soon</p>
          <button onClick={handleKeepSession}>Keep Session Active</button>
        </div>
      )}

      {conversation.sessionStarted && (
        <form onSubmit={handleSendMessage}>
          <input
            type="text"
            value={inputText}
            onChange={e => setInputText(e.target.value)}
            disabled={conversation.mode !== "listening"}
            placeholder="Type your message..."
          />
          <button type="submit" disabled={conversation.mode !== "listening"}>
            Send
          </button>
          <button type="button" onClick={() => conversation.toggleMute()}>
            {conversation.isMuted ? "Unmute" : "Mute"}
          </button>
        </form>
      )}

      <div>
        <h3>Messages ({conversation.messageCounter})</h3>
        {conversation.messages.map(
          (msg, index) =>
            msg.visible && (
              <div key={index}>
                <strong>{msg.sender}:</strong> {msg.text}
                <small> ({msg.timestamp.toLocaleTimeString()})</small>
              </div>
            )
        )}
      </div>
    </div>
  );
}

Message Structure

Messages in the conversation follow this structure:

interface MessageEventData {
  timestamp: Date;
  sender: "user" | "ai";
  text: string;
  visible: boolean;
}

Event Callbacks

When calling startDigitalHuman, you can pass event callbacks:

• onConnect ({userId, headInfo, microphoneAccess}) - Called when the WebSocket connection is established
  • userId Boolean Unique Identifier for the users session.
  • headInfo ConnectHeadType Object with data about the digital human.
    • name String Digital human head name
    • phrases String[] Array with phrases set during digital human creation.
    • language String Language code setup during digital human creation.
    • avatar String Static image url for digital human.
  • microphoneAccess Boolean True if microphone access was granted, False otherwise.
• onDisconnect () - Called when the connection is closed
• onStatusChange ({status}) - Called when connection status changes
  • status "connecting" | "connected" | "disconnecting" | "disconnected" Shows current websocket connection status.
• onMessage ({ timestamp, speaker, text, visible }) - Called when websocket receives a message or sends a response.
  • timestamp Date Timestamp when message was received/sent
  • sender "user" | "ai" Shows who the message came from.
  • text String Message text
  • visible Boolean Flag that you can use to control visibility of message. Sometimes, message comes before the video response starts playing. In such cases, this is usually false. Listen the onSpeakingStart event to change visibility when the video response starts playing.
• onMuteStatusChange - Called when mute status changes
• onSpeakingStart - Called when the digital human starts speaking
• onSpeakingEnd - Called when the digital human finishes speaking
• onSuggestions ({suggestions}) - Invoked when the system generates or updates query suggestions.
  • suggestions String[] A list of suggested query strings.
• onTimeout - Called when the session times out due to inactivity
• onTimeoutWarning - Called before the session times out. This event warns you that the customers session is going to end in a bit. You can call the keepSession method to extend the customers session.
• onKeepSession - Called when a keep-alive request is processed
• onError - Called when an error occurs
• onStoppingStart - Called immediately when a stop request is initiated
• onStoppingEnd - Called once the current response stops playing

Error Handling

Handle errors using the onError callback:

conversation.startDigitalHuman(videoRef.current, {
  onError: ({ message, endConversation, type }) => {
    if (type === "toast") {
      // Show toast notification
      showToast(message);
      if (endConversation) {
        // Restart the session
        conversation.endSession();
      }
    } else if (type === "modal") {
      // Show modal dialog
      showModal(message);
    }
  },
});

Getting Background Video

Retrieve the background video URL for welcome screens:

const { getBackgroundVideo } = useConversation({
  orgId: "your-org-id",
  headId: "your-head-id",
  apiKey: "your-api-key",
});

useEffect(() => {
  async function loadBackgroundVideo() {
    const videoUrl = await getBackgroundVideo();
    // Use videoUrl for your background/welcome screen
  }
  loadBackgroundVideo();
}, []);

Stopping ongoing response

To stop a response:

conversationRef.current.stopResponse();

TypeScript Support

Full TypeScript types are included with the library. Import types as needed:

import { useConversation } from "@unith-ai/react";
import type {
  HeadConfigOptions,
  MessageEventData,
  Status,
  Mode,
} from "@unith-ai/react";

Best Practices

• Call startSession() after user interaction - This ensures audio context is properly initialized, especially on mobile browsers
• Handle the listening mode - Only send messages when mode === "listening" to avoid interrupting the digital human
• Clean up on unmount - The hook automatically calls endSession() on unmount, but you can call it manually if needed
• Use keepSession() - Respond to onTimeoutWarning by calling keepSession() to extend the session
• Handle errors gracefully - Always implement the onError callback to handle connection and capacity errors

Development

Please refer to the README.md file in the root of this repository.