@telnyx/ai-agent-lib

Telnyx AI Agent Library

A TypeScript/React library for building AI-powered voice conversation applications using Telnyx's WebRTC infrastructure. This library provides a comprehensive set of tools for managing AI agent connections, real-time transcriptions, conversation states, and audio streaming.

Features

• 🎯 Easy AI Agent Integration - Connect to Telnyx AI agents with minimal setup
• 🎙️ Real-time Transcription - Automatic speech-to-text with live updates
• 🔊 Audio Stream Management - Built-in audio monitoring and playback controls
• ⚛️ React Hooks & Components - Ready-to-use React components and hooks
• 🔄 State Management - Automatic state synchronization using Jotai
• 📱 Connection Management - Robust connection handling with error recovery
• 🎚️ Agent State Tracking - Monitor agent states (listening, speaking, thinking)

Installation

npm install @telnyx/ai-agent-lib

Peer Dependencies

This library requires React 19.1.1+ as a peer dependency:

npm install react@^19.1.1 react-dom@^19.1.0

Quick Start

1. Wrap your app with the provider

import React from 'react';
import { createRoot } from 'react-dom/client';
import { TelnyxAIAgentProvider } from '@telnyx/ai-agent-lib';
import App from './App';

createRoot(document.getElementById('root')!).render(
  <TelnyxAIAgentProvider agentId="your-agent-id">
    <App />
  </TelnyxAIAgentProvider>
);

2. Use hooks in your components

import React, { useEffect, useRef, useState } from 'react';
import {
  useClient,
  useTranscript,
  useConnectionState,
  useConversation,
  useAgentState
} from '@telnyx/ai-agent-lib';

function VoiceChat() {
  const client = useClient();
  const transcript = useTranscript();
  const connectionState = useConnectionState();
  const conversation = useConversation();
  const agentState = useAgentState();
  const audioRef = useRef<HTMLAudioElement>(null);
  const [messageInput, setMessageInput] = useState('');

  // Setup audio playback
  useEffect(() => {
    if (conversation?.call?.remoteStream && audioRef.current) {
      audioRef.current.srcObject = conversation.call.remoteStream;
    }
  }, [conversation]);

  const handleSendMessage = () => {
    if (messageInput.trim()) {
      client.sendConversationMessage(messageInput);
      setMessageInput('');
    }
  };

  const isCallActive = conversation?.call?.state === 'active';

  return (
    <div>
      <h2>Connection: {connectionState}</h2>
      <h3>Agent State: {agentState}</h3>
      
      <button 
        onClick={() =>
          client.startConversation({
            callerName: 'Jane Doe',
            customHeaders: [{ name: 'X-Account-Number', value: '123456' }]
          })
        }
        disabled={connectionState !== 'connected'}
      >
        Start Conversation
      </button>
      
      <button onClick={() => client.endConversation()}>
        End Conversation
      </button>

      {/* Text message input - only available during active call */}
      {isCallActive && (
        <div style={{ margin: '20px 0' }}>
          <h4>Send Text Message</h4>
          <input
            type="text"
            value={messageInput}
            onChange={(e) => setMessageInput(e.target.value)}
            placeholder="Type a message..."
            onKeyPress={(e) => e.key === 'Enter' && handleSendMessage()}
          />
          <button 
            onClick={handleSendMessage}
            disabled={!messageInput.trim()}
          >
            Send Message
          </button>
        </div>
      )}

      <audio ref={audioRef} autoPlay playsInline controls />

      <div>
        <h3>Transcript ({transcript.length} items)</h3>
        {transcript.map((item) => (
          <div key={item.id}>
            <strong>{item.role}:</strong> {item.content}
            <small> - {item.timestamp.toLocaleTimeString()}</small>
          </div>
        ))}
      </div>
    </div>
  );
}
}

API Reference

TelnyxAIAgentProvider Props

Prop	Type	Required	Default	Description
agentId	string	✅	-	Your Telnyx AI agent ID
versionId	string	❌	"main"	Agent version to use
environment	"production" | "development"	❌	"production"	Telnyx environment
debug	boolean	❌	false	Enable debug logging

Hooks

useClient()

Returns the TelnyxAIAgent instance for direct API access.

Methods:

• connect() - Connect to Telnyx platform
• disconnect() - Disconnect and cleanup
• startConversation(options?) - Start a new conversation with optional caller metadata and headers
• endConversation() - End the current conversation
• sendConversationMessage(message: string) - Send a text message during an active conversation
• transcript - Get current transcript array

startConversation Options:

Option	Type	Description
destinationNumber	string	docs
callerNumber	string	docs
callerName	string	docs
customHeaders	{ name: string; value: string }[]	docs

Events:

• agent.connected - Agent successfully connected
• agent.disconnected - Agent disconnected
• agent.error - Connection or operational error
• transcript.item - New transcript item received
• conversation.update - Conversation state updated
• conversation.agent.state - Agent state changed

useConnectionState()

Returns the current connection state: "connecting" | "connected" | "disconnected" | "error"

useTranscript()

Returns an array of TranscriptItem objects representing the conversation history.

TranscriptItem:

{
  id: string;
  role: "user" | "assistant";
  content: string;
  timestamp: Date;
}

useConversation()

Returns the current conversation notification object containing call information and state.

useAgentState()

Returns the current agent state: "listening" | "speaking" | "thinking"

Direct Usage (Without React)

You can also use the library without React:

import { TelnyxAIAgent } from '@telnyx/ai-agent-lib';

const agent = new TelnyxAIAgent({
  agentId: 'your-agent-id',
  versionId: 'main',
  environment: 'production'
});

// Connect to the platform
await agent.connect();

// Listen for events
agent.on('agent.connected', () => {
  console.log('Agent connected!');
});

agent.on('transcript.item', (item) => {
  console.log(`${item.role}: ${item.content}`);
});

agent.on('conversation.agent.state', (state) => {
  console.log(`Agent is now: ${state}`);
});

agent.on('conversation.update', (notification) => {
  if (notification.call?.state === 'active') {
    console.log('Call is now active - you can send messages');
    
    // Send a text message during the conversation
    agent.sendConversationMessage('Hello, I have a question about your services.');
  }
});

// Start a conversation with optional caller metadata
await agent.startConversation({
  callerNumber: '[phone]',
  callerName: 'John Doe',
  customHeaders: [
    { name: 'X-User-Plan', value: 'gold' },
    { name: 'X-Session-Id', value: 'session-abc' }
  ],
  audio: { autoGainControl: true, noiseSuppression: true }
});

// Send messages during an active conversation
// Note: This will only work when there's an active call
setTimeout(() => {
  agent.sendConversationMessage('Can you help me with my account?');
}, 5000);

// Access transcript
console.log(agent.transcript);

// Cleanup
await agent.disconnect();

Advanced Usage

Sending Text Messages During Conversations

You can send text messages to the AI agent during an active conversation using the sendConversationMessage method. This is useful for providing context, asking questions, or sending information that might be easier to type than speak.

// React example
function ChatInterface() {
  const client = useClient();
  const conversation = useConversation();
  const [message, setMessage] = useState('');

  const handleSendMessage = () => {
    if (conversation?.call?.state === 'active' && message.trim()) {
      client.sendConversationMessage(message);
      setMessage('');
    }
  };

  return (
    <div>
      {conversation?.call?.state === 'active' && (
        <div>
          <input
            value={message}
            onChange={(e) => setMessage(e.target.value)}
            placeholder="Type a message to the agent..."
            onKeyPress={(e) => e.key === 'Enter' && handleSendMessage()}
          />
          <button onClick={handleSendMessage}>Send</button>
        </div>
      )}
    </div>
  );
}

// Direct usage example
agent.on('conversation.update', (notification) => {
  if (notification.call?.state === 'active') {
    // Now you can send text messages
    agent.sendConversationMessage('I need help with order #12345');
  }
});

Important Notes:

• Messages can only be sent during an active conversation (when call.state === 'active')
• The agent will receive and process text messages just like spoken input
• Text messages may appear in the transcript depending on the agent configuration

Custom Audio Handling

The library automatically handles audio stream monitoring and agent state detection based on audio levels. The audio stream is available through the conversation object:

const conversation = useConversation();
const audioStream = conversation?.call?.remoteStream;

Error Handling

const client = useClient();

useEffect(() => {
  const handleError = (error: Error) => {
    console.error('Agent error:', error);
    // Handle error (show notification, retry connection, etc.)
  };

  client.on('agent.error', handleError);
  
  return () => {
    client.off('agent.error', handleError);
  };
}, [client]);

State Persistence

The library uses Jotai for state management, which automatically handles state updates across components. All state is ephemeral and resets when the provider unmounts.

TypeScript Support

This library is built with TypeScript and provides full type definitions. All hooks and components are fully typed for the best development experience.

Examples

Check out the example repository for a complete example application demonstrating all features of the library.

Dependencies

• @telnyx/webrtc - Telnyx WebRTC SDK
• eventemitter3 - Event handling
• jotai - State management

License

This library is part of the Telnyx ecosystem. Please refer to Telnyx's terms of service and licensing agreements.

Support

For support, please contact Telnyx support or check the Telnyx documentation.

Contributing

This library is maintained by Telnyx. For bug reports or feature requests, please contact Telnyx support.elnyx AI Agent Library