augnito-rn-sdk

Augnito React Native SDK

Overview

Use the Augnito React Native SDK to enable Text To Speech and voice commands into a React Native application.

Supported Platforms

• Android
• iOS

Requirements

Microphone access/recording permission is required in order to work.

Android

Add the following line to AndroidManifest.xml

<uses-permission android:name="android.permission.RECORD_AUDIO"/>

iOS

Add the following to Info.plist:

<key>NSMicrophoneUsageDescription</key>
<string>Microphone access required</string>

Installation

The Augnito React Native SDK has to be installed by running:

npm install augnito-rn-sdk
cd ios
pod install

Note on M1 family processor: refer to environment setup to solve compatibility issues for pods

Getting Started

The majority of the functionality goes through the DictationManager class via the use of callbacks.

To instantiate a DictationManager object an AugnitoConfig must be provided with valid authorization and keys provided.

Using artifacts from the SDK

import {
  DictationManager,
  AugnitoApiServer,
  AugnitoConfig,
  TextUtils,
  ActionRecipe,
} from 'augnito-rn-sdk';

const augnitoConfig: AugnitoConfig = new AugnitoConfig(
  '<your server>',
  '<your accountcode>',
  '<your accesskey>',
  '<your lmid>',
  '<your usertag>',
  '<your sourceapp>'
);

And on an initialization method

  const dictationManager: DictationManager = useMemo(
    () => DictationManager.fromConfig(augnitoConfig),
    []
  );

Alternatively it can be initialized with a provided connection URL:

  const dictationManager: DictationManager = useMemo(
    () => DictationManager.fromCustomUrl('custom url'),
    []
  );

---

Dictation Manager

The DictationManager is used to initialize, receive, and stop communication with Augnito's server to provide Speech To Text and Commands support into a React Native app.

The way it works is via methods and callbacks.

Logging

Logging can be enabled via the DictationManager:

const enableLogging = true;
...
  const dictationManager: DictationManager = useMemo(
    () => DictationManager.fromConfig(augnitoConfig, enableLogging),
    []
  );

Dictation Manager Methods

method	notes
toggleDictation	Turns on or off the audio processing and server communication.

Dictation Manager Properties

property	notes
isBusy	Indicates the Dictation Manager is currently performing an operation.

Dictation Manager Callbacks

Callbacks are used to receive the output of the SDK as well as errors that may occur. The callbacks are provided via the DictationManager constructor as named optional parameters.

callback	notes
onConnected	Invoked when the connection to the server has been established. This does not guarantees the audio stream has started.
onDisconnected	Invoked when the server closes connection. It can be triggered on purpose by the DictationManager under certain conditions such as being unable to start audio streaming.
onError	Invoked when an error occurs. Not all errors invoke a connection termination.
onPartialResult	Invoked when an hypothesis or non processed final text has been obtained.
onFinalResult	Invoked when a final text output (not command) has been processed.
onCommandResult	Invoked when a command has been processed.

Example usage

  dictationManager.onConnected = useCallback(() => {
    setIsRecording(true);
    setIsLoading(false);
    setTitle(titleListening);
  }, []);

  dictationManager.onDisconnected = useCallback(() => {
    setIsRecording(false);
    setIsLoading(false);
    setTitle(titleStart);
  }, []);

Dictation Manager Error Types

Dictation Manager will report back on errors via the onError callback. Besides the method itself an object with further details is provided.

The errorType property is the best way to determine what could be happening underneath. Possible types:

property	notes
noNetworkConnection
lowBandwidth
serviceDown
noDictationStopMic
invalidAuthCredentials
socketDisconnect
socketConnectionError
audioRecorder
audioRecorderCouldNotInitializePermission
audioRecorderCouldNotInitializeGeneric
unknown

Even further information will be provided on the errorMessage property when possible.

A note on noDictationStopMic: this will not terminate the connection but it's and indicator the microphone is open but idle. The consumer app may want to warn the user or terminate the session if this keeps happening.

---

Commands

Commands are represented by the ActionRecipe object. It includes the information required to be processed by the consumer app.

Commands are returned by the SDK via the onCommandResult callback (DictationManager).

ActionRecipe

As stated the result of a command (either Static or Dynamic) will be an instance of an ActionRecipe.

When a command is returned by the SDK the consumer needs to analyze it and determine what to do based on the properties of the ActionRecipe instance.

For example, a Static Command resulting ActionRecipe will always have it's isStaticCommand property set as true.

Other fields such as searchText, chooseNumber or selectFor are used to determine what to do with the command and in what context. For instance a Replace X with Y dynamic command will produce an ActionRecipe like:

// replace oranges with apples
{
  name: 'replace', // AugnitoCommands.replace entry
  isCommand: true,
  searchText: 'oranges',
  selectFor: 'apples'
  ...
}

ActionRecipe fields

property	notes
name	Command name. Can be matched against one of the entries of AugnitoCommands class.
action
chooseNumber	Unit: on commands that affect items it indicates how many.
searchText	On most select/action commands this represents the item to be searched. For example paragraph, line, word, etc. On some commands this indicates what is being searched for.
selectFor	Usually used to determine an action on most commands. For example delete, underline, etc.
isCommand	Indicates the ActionRecipe is a command.
isStaticCommand	Indicates this ActionRecipe is a static command.
nextPrevious	Direction on how this command should affect the items.
receivedText	Original received text.
receivedTextWithoutSpace	Original text without spaces.

Using Commands

Commands are meant to be used on their on and not in conjunction with regular dictation.

For example:

Dictating Select next 3 words will produce a command

Dictating Patient presents with fever and chills select last 2 words will only produce a transcription

Matching commands

Commands can be matched with entries on the AugnitoCommands dictionary. In the previous example the name equals to AugnitoCommands.replace.

Static Commands

Commands that have not effect on specific units or items are denominated as static commands.

Microphone Control Static Commands
stopMic

Selection Static Commands
selectAll
selectChar
selectWord
selectLine
selectNextLine
selectPreviousLine
selectParagraph

Lists Static Commands
startNumberList
startBulletList
stopBulletList
stopNumberList
stopList

Action Static Commands
undoIt
redoIt

Text Modification Commands
startBoldText
stopBoldText
startBulletText
stopBulletText
startItalicText
stoptItalicText

Modification Commands
deleteIt
pasteIt
copyIt
cutIt
headerIt
underlineIt
boldIt
italicizeIt
capitalizeIt
bulletIt

Navigation Static Commands
moveUp
moveDown
moveLeft
moveRight
goToLineStart
goToLineEnd
giveSpace
backspace
goToDocumentEnd
goToDocumentStart
goToNextPage
goToPreviousPage
goToNextParagraph
goTo

Field Navigation Static Commands
nextField
previousField

Dynamic Commands

Dynamic commands are more complex commands built based on the speech input and may contain extra information (always within the ActionRecipe itself).

Line and Paragraph number

Commands that affect a line or paragraph number via an action command such as select, bold, delete, etc.

^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(line|para|paragraph|\n\n|\n)(number[ed]?)(.*?)(\.)?$

Try online

A correctly parsed Line and Paragraph number will have a structure similar to:

{
  name: AugnitoCommands.goToLineNumber // or AugnitoCommands.goToParaNumber
  isCommand: true,
  chooseNumber: 10,
  searchText: "line",
  selectFor: "goToStart"
  ...
}

Example voice command

• choose line number 10
• delete paragraph number 5
• bold the para number 1

---

Go To New Line or New Paragraph

Simple Go To new line or new paragraph command.

{
  name: AugnitoCommands.selectLine // or AugnitoCommands.selectParagraph
  isCommand: true,
  searchText: "line",
  selectFor: "gotoend"
  ...
}

Example voice command

• go to new line
• go to new paragraph

---

Active X Dynamic Command

Action against the active line, paragraph, space, etc.

^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(line|para|paragraph|\n\n|\n)(number[ed]?)(.*?)(\.)?$

Try online

Example voice command

• select active paragraph
• delete the current line

{
  name: AugnitoCommands.selectActiveChar
  isCommand: true,
  ...
}

{
  name: AugnitoCommands.selectActiveWord
  isCommand: true,
  ...
}

{
  name: AugnitoCommands.selectActiveLine
  isCommand: true,
  ...
}

{
  name: AugnitoCommands.selectActiveParagraph
  isCommand: true,
  ...
}

---

Navigation Without Go To Dynamic Command

Processes a direction plus a combination of item and distance.

^(last|previous|next|down)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n)$

Try online

Example voice commands

• next 3 lines
• last 5 paras

---

Go To X Dynamic Command

Processor to identify movement with direction and unit.

^(go|goto|gotothe|move|moveto|movethe)(last|previous|next|down)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n)$

Try online

Example voice commands

• go last 4 word
• go to last space
• move the previous 4 words

// go last 4 word
{
  name: AugnitoCommands.selectWord
  isCommand: true,
  chooseNumber: 4,
  nextPrevious: 'last',
  selectFor: 'gotostart'
  ...
}

// go to last space
{
  name: AugnitoCommands.selectChar
  isCommand: true,
  chooseNumber: 0,
  nextPrevious: 'last',
  selectFor: 'gotostart'
  ...
}

// move the previous 4 words
{
  name: AugnitoCommands.selectWord
  isCommand: true,
  chooseNumber: 4,
  nextPrevious: 'previous',
  selectFor: 'gotostart'
  ...
}

Select / Action on Item with Unit and Direction Dynamic Command

Identifies an action on an item that includes direction and unit amount.

^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(the)?(last|previous|next)(.*?)(word[s]?|line[s]?|sentence[s]?|paragraph[s]?|para[s]?|char[s]?|character[s]?|space|\n\n|\n\ns|\n|\ns)$

Try online

Example voice commands

• choose the next 10 characters
• underline previous new line
• capitalize the next 5 words
• delete previous 5 paragraphs

// choose the next 10 characters
{
  name: AugnitoCommands.selectChar
  isCommand: true,
  chooseNumber: 10,
  nextPrevious: 'next',
  searchText: 'characters',
  selectFor: ''
  ...
}

// underline previous new line
{
  name: AugnitoCommands.selectLine
  isCommand: true,
  chooseNumber: 1,
  nextPrevious: 'previous',
  searchText: '\n',
  selectFor: 'underline'
  ...
}

// capitalize the next 5 words
{
  name: AugnitoCommands.selectWord
  isCommand: true,
  chooseNumber: 5,
  nextPrevious: 'next',
  searchText: 'words',
  selectFor: 'capitalize'
  ...
}

// delete previous 5 paragraphs
{
  name: AugnitoCommands.selectParagraph
  isCommand: true,
  chooseNumber: 5,
  nextPrevious: 'previous',
  searchText: '\n\n',
  selectFor: 'delete'
  ...
}

Replace X with Y Dynamic Command

Identifies the replace operation between two elements.

replace[d]?([A-Z a-z 0-9]+)with([A-Z a-z 0-9]+)

Try online

Example voice commands

• replace oranges with apples
• replace blue with red
• replace cars with buses

// replace oranges with apples
{
  name: AugnitoCommands.replace
  isCommand: true,
  searchText: 'oranges',
  selectFor: 'apples'
  ...
}

Select/Action Group Dynamic Command

Identifies a Select/Action command.

^(select|choose|copy text|copytext|cut text|cuttext|correct|bold|underline|delete|header|capitalize|unbold|debold|dbold|uncapitalize|remove|capitalise|dcapitalise|decapitalicize|decapitalize|uncapitalize|Uncap|d capitalise|d capitalize|d underline|dunderline|deunderline|ununderline|goto|moveto|move|italicize|italicise|unitalicise|unitalicize)(\sthe)?\s?(.*?)$

Try online

Example voice commands

• delete mistaken
• bold firstname
• capitalize last name

// delete mistaken
{
  name: AugnitoCommands.select
  isCommand: true,
  searchText: 'mistaken',
  selectFor: 'delete'
  ...
}

---

Utils and Helpers

TextUtils support class

The SDK includes a support class used to handle basic operations on the TextInput control.

The methods included for the processor are:

method	notes
processFinalResult	based on the current selection status, existing text and received text returns the final state of the text and where the cursor should be placed.
selectLastLines	based on the current text, cursor position and amount of desired amount of lines the method returns a range indicating where in the string the selection starts and ends.
selectLastWords	based on the current text, cursor position and amount of desired amount of words the method returns a range indicating where in the string the selection starts and ends.

The usage of these methods is shown on the example app:

  const processSelectWords = (
    text: string,
    caretPosition: number,
    words: number,
    selectFor: string,
    updateStateFunction: (value: React.SetStateAction<string>) => void
  ) => {
    const range = TextUtils.selectLastWords(text, words, caretPosition);

    if (selectFor === Commands.Delete) {
      const finalText =
        text.substring(0, range.start) + text.substring(range.end);

      setRequiresUpdateNativePropSelection(true);
      updateStateFunction(finalText);
      setCurrentSelection({
        start: range.start,
        end: range.start,
      });
    } else {
      setRequiresUpdateNativePropSelection(true);
      setCurrentSelection({
        start: range.start,
        end: range.end,
      });
    }
  };

iOS and background UI modes

In order to keep the app working on the background the info.plist requires the following entries:

<key>UIBackgroundModes</key>
<array>
  <string>audio</string>
  <string>fetch</string>
</array>

---

Sample App

The SDK includes an app that shows how to use the SDK on basic text controls. The app is located on the /example directory.

Requirements and usage

The app requires the same environment setup as for any React Native app. The configuration for each supported OS and Platform is displayed on the official page.

Besides the regular setup, packages and and a local .env file must be present. On the example directory:

npm install

For the .env the .env.example can be cloned and renamed. Later, each parameter needs to be set to the one provided.

Running the Sample App

Once cloned or downloaded go to the Example directory and if the environment is properly configured follow the regular steps:

npx react-native start
npx react-native run-ios