use-lipsync

use-lipsync

React hook for text-driven lip sync with ARKit morph targets. Supports English + Indian languages (Hindi, Tamil, Telugu, Kannada, Malayalam).

Install

npm install use-lipsync

Usage

import { useRef, useState } from "react";
import { useLipSync } from "use-lipsync";

function Avatar() {
  const audioRef = useRef(null);
  const [isPlaying, setIsPlaying] = useState(false);
  const [morphValues, setMorphValues] = useState({ jawOpen: 0, mouthFunnel: 0, mouthSmileLeft: 0, mouthSmileRight: 0 });

  const handleMorphUpdate = ({ jawOpen, mouthFunnel, mouthSmileLeft }) => {
    // Apply these morph targets to your 3D model
  };

  useLipSync(audioRef, isPlaying, handleMorphUpdate, "வணக்கம் hello");

  return (
    <audio
      ref={audioRef}
      src="speech.mp3"
      onPlay={() => setIsPlaying(true)}
      onEnded={() => setIsPlaying(false)}
    />
  );
}

API

useLipSync(audioRef, isPlaying, onMorphUpdate, spokenText)

Param	Type	Description
audioRef	RefObject<HTMLAudioElement>	Ref to the audio element
isPlaying	boolean	Whether audio is currently playing
onMorphUpdate	(morphs) => void	Callback with { jawOpen, mouthFunnel, mouthSmileLeft }
spokenText	string	The text being spoken (supports Indic scripts)

Supported Languages

• English (passthrough)
• Hindi (Devanagari)
• Tamil
• Telugu
• Kannada
• Malayalam