twilio-stt-gspeech

twilio-stt-gspeech

Speech to text in Twilio over websocket using Google Speech API

Introduction

This package is an extension of this article. In the above article, live transcribing phone calls using Twilio Media Streams and Google Speech-to-text is explained. This package helps not only in converting speech to text but also returns a promise so that it can be binded to a twilio webhook.

Requirements

• A Free Twilio Account
• A Google Cloud Account
• Installed ngrok
• Installed the Twilio CLI

API

init (websocketServer: Server, googleSpeechClient: SpeechClient, debugOption?: boolean): void

• This is the very fist method that has to be invoked, when a web socket is created.
• It takes the parameters
  • websocketServer, the created websocket server has to passed
  • googleSpeechClient, instance of google SpeechCLient has to be passed
  • debugOption, is an optional Parameter to view the debug messages over the console.

listen (callSid: string, timeout: number): Promise

• This is the listener API to which for a particular caller's Speech To Text value will be received as an output
• This function returns a promise.
  • When the promise resolved Speech To Text value is obtained
  • When the promise rejected might be due to timeout
• This function takes the parameters such as
  • callSid, the caller's SID, which is an unique ID generated by Twilio
  • timeout, number in milliseconds. For how long one can wait for the promise to resolve.

Example

Setting up a local server

Before getting into this please configure your - twilio voice configuration's request url with https://{host}/voice - Download your Google Cloud Service account credentials as JSON and set the path accordingly - For Windows, in command prompt set GOOGLE_APPLICATION_CREDENTIALS=D:\path\to\credentials\file.json - For linux export GOOGLE_APPLICATION_CREDENTIALS=D:\path\to\credentials\file.json

Create a project folder, index.js file and install the mentioned modules

$ mkdir twilio-transcribe-test
$ cd twilio-transcribe-test
$ touch index.js
$ npm i express @google-cloud/speech twilio ws twilio-stt-gspeech

Now we need to setup the server and configure the twilio speech to text using google speech API. Open the index.js file and add the following code.

const express = require('express');
const http = require("http");
const speech = require('@google-cloud/speech');
const WebSocket = require('ws');

const { init } = require('twilio-stt-gspeech');

const app = express();
const server = http.createServer(app);

wss = new WebSocket.Server({ server });
init(wss, new speech.SpeechClient(), true);

app.use(express.json());
app.use(express.urlencoded({
    extended: true
}))

app.use('/voice', require('./router'));

server.listen(8080, () => {
    console.log("server running at 8080");
});

In the above code, init method has to be imported from twilio-stt-gspeech package. Next the parameters for the init method includes created websocket, google speech client, and an optional boolean parameter to display debug messages. Next, a listener is added for the route /voice. Create a folder with name as router and a file inside the folder and name it as index.js.

$ mkdir router
$ cd router
$ touch index.js

Open the index.js file and add the following code

const { VoiceResponse } = require('twilio').twiml;

exports.init = (host, { CallSid }) => {

    const twiml = new VoiceResponse();
    twiml.start()
    .stream({
        url: `wss://${host}/${CallSid}`,
        name: CallSid
    });
    twiml.redirect('/voice/stream');
    return twiml.toString();
}

exports.stream = ({ CallSid }) => {

    const twiml = new VoiceResponse();
    twiml.stop()
    .stream({
        name: CallSid
    });
    return new Promise( (resolve, reject) => {
    
        require('twilio-stt-gspeech').listen(CallSid)
        .then( data => {
            twiml.say("You said "+data);
            resolve(twiml.toString());
        })
        .catch((err) => {
            console.log("Error occurred", err);
            reject();
        })
    });
}

When a call is made to a twilio number, and as per the twilio webhook endpoint which is configured to https://{host}/voice/ the init method is invoked. The init method opens up a websocket stream for the caller and expects the user to speak. The websocket is opened using the twilio API start and inside it stream API is invoked using the following parameters that has to be mentioned explicitly. url is the the websocket url with paths containing the caller's SID and language with country code (Eg. en-IN for Indian English). By default the language is en-US. The next parameter is the name the value should be the caller's SID. And finally it completes the request by redirecting to another route i.e., https://{host}/voice/stream/. The redirected route will listen for the speech to text value from the package's(twilio-stt-gspeech) listen API. This API can be configured with a timeout so that for how long the route can wait to receive the speech to text value. Hence, when the user speaks, it is received via websocket, and when the user pauses after speaking, then the Speech to Text value is resolved via the listen method's promise.