deus-openai-rt-widget

Real-time Audio Processing Web Application

This repository contains a web application built with React, TypeScript, and Vite, designed for real-time audio processing and interaction with an API. The application leverages OpenAI's API for advanced audio and speech functionalities.

Features

• Real-time Audio Interaction: Capture and play audio in real-time using WavRecorder and WavStreamPlayer.
• API Integration: Connects to a real-time API for processing audio data, with support for OpenAI's API.
• User Interface: Interactive UI with buttons and audio visualizations using React components.
• Local Relay Server: Option to run a local server to hide API keys and execute custom server-side logic.
• Memory and Event Handling: Manage session data and handle various audio and conversation events.

Getting Started

Prerequisites

• Node.js and npm installed on your machine.

Installation

1. Clone the repository:

   git clone <repository-url>
   cd <repository-directory>
   

2. Install dependencies:

   npm install
   

3. Set up environment variables:

   • Create a .env file in the root directory.
   • Add your OpenAI API key: OPENAI_API_KEY=your_api_key_here
   • Optionally, set up a local relay server URL: REACT_APP_LOCAL_RELAY_SERVER_URL=http://localhost:8081

Running the Application

1. Start the local relay server (if configured):

   npm run relay
   

2. Start the application:

   npm start
   

3. Open your browser and navigate to http://localhost:3000 to access the application.

Usage

• Connect/Disconnect: Use the button on the interface to connect or disconnect from the audio session.
• Audio Visualization: View real-time audio visualizations on the canvas elements.
• Memory Management: The application can store and manage session-specific data.

Contributing

Contributions are welcome! Please fork the repository and submit a pull request for any improvements or bug fixes.

Acknowledgments

• Built with React, TypeScript, and Vite.
• Utilizes OpenAI's API for advanced audio processing.