@aws-sdk/client-transcribe-streaming

@aws-sdk/client-transcribe-streaming

Introduction

Amazon Transcribe streaming enables you to send an audio stream and receive back a stream of text in real time. The API makes it easy for developers to add real-time speech-to-text capability to their applications. It can be used for a variety of purposes. For example:

• Streaming transcriptions can generate real-time subtitles for live broadcast media.
• Lawyers can make real-time annotations on top of streaming transcriptions during courtroom depositions.
• Video game chat can be transcribed in real time so that hosts can moderate content or run real-time analysis.
• Streaming transcriptions can provide assistance to the hearing impaired.

The JavaScript SDK Transcribe Streaming client encapsulates the API into a JavaScript library that can be run on browsers, Node.js and potentially React Native. By default, the client uses HTTP/2 connection on Node.js, and uses WebSocket connection on browsers and React Native.

Installing

To install the this package, simply type add or install @aws-sdk/client-transcribe-streaming using your favorite package manager:

• npm install @aws-sdk/client-transcribe-streaming
• yarn add @aws-sdk/client-transcribe-streaming
• pnpm add @aws-sdk/client-transcribe-streaming

Getting Started

In the sections bellow, we will explain the library by an example of using startStreamTranscription method to transcribe English speech to text.

If you haven't, please read the root README for guidance for creating a sample application and installation. After installation, in the index.js, you can import the Transcribe Streaming client like:

// ES5 example
const { TranscribeStreamingClient, StartStreamTranscriptionCommand } = require("@aws-sdk/client-transcribe-streaming");

If require is not available on the platform you are working on(browsers). You can import the client like:

// ES6+ example
import {
  TranscribeStreamingClient,
  StartMedicalStreamTranscriptionCommand,
} from "@aws-sdk/client-transcribe-streaming";

Constructing the Service Client

You can create a service client like bellow:

const client = new TranscribeStreamingClient({
  region,
  credentials,
});
// region and credentials are optional in Node.js

Acquire Speech Stream

The Transcribe Streaming client accepts streaming speech input as an async iterable. You can construct them from either an async generator or using Symbol.asyncIterable to emit binary chunks.

Here's an example of using async generator:

const audioStream = async function* () {
  await device.start();
  while (device.ends !== true) {
    const chunk = await device.read();
    yield chunk; /* yield binary chunk */
  }
};

Then you need to construct the binary chunk into an audio chunk shape that can be recognized by the SDK:

const audioStream = async function* () {
  for await (const chunk of audioSource()) {
    yield { AudioEvent: { AudioChunk: chunk } };
  }
};

Acquire from Node.js Stream API

In Node.js you will mostly acquire the speech in Stream API, from HTTP request or devices. Stream API in Node.js (>= 10.0.0) itself is an async iterable. You can supply the streaming into the SDK input without explicit convert. You only need to construct the audio chunk shape that can be recognized by the SDK:

const audioSource = req; //Incoming message
const audioStream = async function* () {
  for await (const payloadChunk of audioSource) {
    yield { AudioEvent: { AudioChunk: payloadChunk } };
  }
};

If you see don't limit the chunk size on the client side, for example, streams from fs, you might see The chunk is too big error from the Transcribe Streaming. You can solve it by setting the HighWaterMark:

const { PassThrough } = require("stream");
const { createReadStream } = require("fs");
const audioSource = createReadStream("path/to/speech.wav");
const audioPayloadStream = new PassThrough({ highWaterMark: 1 * 1024 }); // Stream chunk less than 1 KB
audioSource.pipe(audioPayloadStream);
const audioStream = async function* () {
  for await (const payloadChunk of audioPayloadStream) {
    yield { AudioEvent: { AudioChunk: payloadChunk } };
  }
};

Depending on the audio source, you may need to PCM encode you audio chunk.

Acquire from Browsers

The Transcribe Streaming SDK client also supports streaming from browsers. You can acquire the microphone data through getUserMedia API. Note that this API is supported by a subset of browsers. Here's a code snippet of acquiring microphone audio stream using microphone-stream

const mic = require("microphone-stream");
// this part should be put into an async function
micStream.setStream(
  await window.navigator.mediaDevices.getUserMedia({
    video: false,
    audio: true,
  })
);
const audioStream = async function* () {
  for await (const chunk of micStream) {
    yield { AudioEvent: { AudioChunk: pcmEncodeChunk(chunk) /* pcm Encoding is optional depending on the source */ } };
  }
};

You can find the a full front-end example here

PCM encoding

Currently Transcribe Streaming service only accepts PCM encoding. If your audio source is not already encoded, you need to PCM encoding the chunks. Here's an example:

const pcmEncodeChunk = (chunk) => {
  const input = mic.toRaw(chunk);
  var offset = 0;
  var buffer = new ArrayBuffer(input.length * 2);
  var view = new DataView(buffer);
  for (var i = 0; i < input.length; i++, offset += 2) {
    var s = Math.max(-1, Math.min(1, input[i]));
    view.setInt16(offset, s < 0 ? s * 0x8000 : s * 0x7fff, true);
  }
  return Buffer.from(buffer);
};

Send the Speech Stream

const command = new StartStreamTranscriptionCommand({
  // The language code for the input audio. Valid values are en-GB, en-US, es-US, fr-CA, and fr-FR
  LanguageCode: "en-US",
  // The encoding used for the input audio. The only valid value is pcm.
  MediaEncoding: "pcm",
  // The sample rate of the input audio in Hertz. We suggest that you use 8000 Hz for low-quality audio and 16000 Hz for
  // high-quality audio. The sample rate must match the sample rate in the audio file.
  MediaSampleRateHertz: 44100,
  AudioStream: audioStream(),
});
const response = await client.send(command);

Handling Text Stream

If the request succeeds, you will get a response containing the streaming transcript like this. Just like the input speech stream, the transcript stream is an async iterable emitting the partial transcripts. Here is a code snippet of accessing the transcripts

// This snippet should be put into an async function
for await (const event of response.TranscriptResultStream) {
  if (event.TranscriptEvent) {
    const message = event.TranscriptEvent;
    // Get multiple possible results
    const results = event.TranscriptEvent.Transcript.Results;
    // Print all the possible transcripts
    results.map((result) => {
      (result.Alternatives || []).map((alternative) => {
        const transcript = alternative.Items.map((item) => item.Content).join(" ");
        console.log(transcript);
      });
    });
  }
}

Pipe Transcripts Stream

In Node.js, you can pipe this TranscriptResultStream to other destinations easily with the from API:

const { Readable } = require("stream");
const transcriptsStream = Readable.from(response.TranscriptResultStream);
transcriptsStream.pipe(/* some destinations */);

Error Handling

If you are using async...await style code, you are able to catch the errors with try...catch block. There are 2 categories of exceptions can be thrown:

• Immediate exceptions thrown before transcription is started, like signature exceptions, invalid parameters exceptions, and network errors;
• Streaming exceptions that happens after transcription is started, like InternalFailureException or ConflictException.

For immediate exceptions, the SDK client will retry the request if the error is retryable, like network errors. You can config the client to behave as you intend to.

For streaming exceptions, because the streaming transcription is already started, client cannot retry the request automatically. The client will throw these exceptions and users can handle the stream behavior accordingly.

Here's an example of error handling flow:

try {
  const response = await client.send(command);
  await handleResponse(response);
} catch (e) {
  if (e instanceof InternalFailureException) {
    /* handle InternalFailureException */
  } else if (e instanceof ConflictException) {
    /* handle ConflictException */
  }
} finally {
  /* clean resources like input stream */
}

Notes for React Native

This package is compatible with React Native (>= 0.60). However, it is not tested with any React Native libraries that converts microphone record into streaming data. Community input for integrating streaming microphone record data is welcome.

Thank you for reading this guide. If you want to know more about how streams are encoded, how connection is established, please refer to the Service API guide.

Contributing

This client code is generated automatically. Any modifications will be overwritten the next time the @aws-sdk/client-transcribe-streaming package is updated. To contribute to client you can check our generate clients scripts.

License

This SDK is distributed under the Apache License, Version 2.0, see LICENSE for more informatio

Changelog

3.699.0 (2024-11-22)

Bug Fixes

• credential-provider-ini: pass clientConfig to sso and sso-oidc inner clients (#6688) (0ca3da3)

Features

• client-auto-scaling: Now, Amazon EC2 Auto Scaling customers can enable target tracking policies to take quicker scaling decisions, enhancing their application performance and EC2 utilization. To get started, specify target tracking to monitor a metric that is available on Amazon CloudWatch at seconds-level interval. (26d5f85)
• client-bcm-pricing-calculator: Initial release of the AWS Billing and Cost Management Pricing Calculator API. (13fd8a4)
• client-bedrock-agent-runtime: InvokeInlineAgent API release to help invoke runtime agents without any dependency on preconfigured agents. (372482a)
• client-chatbot: Adds support for programmatic management of custom actions and aliases which can be associated with channel configurations. (acdb624)
• client-codepipeline: AWS CodePipeline V2 type pipelines now support ECRBuildAndPublish and InspectorScan actions. (cce4188)
• client-cognito-identity-provider: Add support for users to sign up and sign in without passwords, using email and SMS OTPs and Passkeys. Add support for Passkeys based on WebAuthn. Add support for enhanced branding customization for hosted authentication pages with Amazon Cognito Managed Login. Add feature tiers with new pricing. (3ef966f)
• client-connect: Amazon Connect Service Feature: Add APIs for Amazon Connect Email Channel (a83926c)
• client-cost-explorer: This release adds the Impact field(contains Contribution field) to the GetAnomalies API response under RootCause (af7c9b2)
• client-elastic-load-balancing-v2: This release adds support for advertising trusted CA certificate names in associated trust stores. (5e46efc)
• client-emr: Advanced Scaling in Amazon EMR Managed Scaling (4092279)
• client-inspector2: Extend inspector2 service model to include ServiceQuotaExceededException. (dc13e07)
• client-lambda: Add ProvisionedPollerConfig to Lambda event-source-mapping API. (511ab41)
• client-mailmanager: Added new "DeliverToQBusiness" rule action to MailManager RulesSet for ingesting email data into Amazon Q Business customer applications (048d165)
• client-neptune-graph: Add 4 new APIs to support new Export features, allowing Parquet and CSV formats. Add new arguments in Import APIs to support Parquet import. Add a new query "neptune.read" to run algorithms without loading data into database (48054dc)
• client-omics: This release adds support for resource policy based cross account S3 access to sequence store read sets. (dbe3f07)
• client-quicksight: This release includes: Update APIs to support Image, Layer Map, font customization, and Plugin Visual. Add Identity center related information in ListNamsespace API. Update API for restrictedFolder support in topics and add API for SearchTopics, Describe/Update DashboardsQA Configration. (df75ca5)
• client-sagemaker: This release adds APIs for new features for SageMaker endpoint to scale down to zero instances, native support for multi-adapter inference, and endpoint scaling improvements. (f082da5)
• client-ses: This release adds support for starting email contacts in your Amazon Connect instance as an email receiving action. (14fc94c)
• client-sfn: Add support for variables and JSONata in TestState, GetExecutionHistory, DescribeStateMachine, and DescribeStateMachineForExecution (c9c278a)
• clients: update client endpoints as of 2024-11-22 (bcd27a2)