augnito-plate-button

Augnito Plate Button

Overview

Augnito Plate Button enables Augnito SDK support into within Plate enabling speech to text on it. It requires a React app with Plate and it's peer dependencies installed.

Some commands make use of specific plugins such as bold, start lists, etc.

Required peer dependencies

Before installing the Augnito Plate Button a React app with a functional Plate editor bust be already set up. The instructions on how to bootstrap a React app are out of the scope of this library.

Plate

npm install @udecode/plate

Plate's peers dependencies are also required

npm install slate slate-react slate-history slate-hyperscript react react-dom styled-components

Up to this point this is a basic Plate installation. Please check the Version Support section in order to avoid peer dependencies errors.

Plugins

In order to use some commands such as marks (bold, italics, etc), lists and others the related plugin must be installed and added to the plate instance. These must be installed and enabled as described on Plate's documentation.

Installation

Once all the dependencies are installed the Augnito Plate Button can be installed via npm using:

npm i augnito-plate-button

Basic Usage

1- Import the library and styles into the component where the editor will be hosted

import { ToolbarAugnito, AugnitoAPIServer } from 'augnito-plate-button';
import 'augnito-plate-button/dist/style.css';

2- Prepare your SDK provided values

const server = AugnitoAPIServer.UK;
const accountCode = '';
const accessKey = '';
const userTag = '';
const deviceId = '';
const sourceApp = '';
const lmId = '';

3- Use the component

This requires plate and the toolbar to be nested under a PlateProvide

<PlateProvider>
    <HeadingToolbar>
        <ToolbarAugnito
          server={server}
          accountCode={accountCode}
          accessKey={accessKey}
          userTag={userTag}
          deviceId={deviceId}
          sourceApp={sourceApp}
          lmId={lmId}></ToolbarAugnito>
    </HeadingToolbar>
    <Plate />
</PlateProvider>

4 - Restricting nodes Augnito Plate Button will affect

To prevent certain nodes being considered a NodeMatch can be provided in the options when creating the toolbar component. If any ancestor nodes in the branch are matched the leaf node won't be evaluated.

NOTE: Restricting nodes Augnito Plate Button can interact with will prevent the select all command from functioning. It will also change go to document start and go to document end to go to the first or last text leaf that does not have a matching ancestor.

function App() 
{
    ...
    const titleNodeType:NodeMatch = {type:"title"};
    ...
    <PlateProvider>
        <HeadingToolbar>
            <ToolbarAugnito
                server={server}
                accountCode={accountCode}
                accessKey={accessKey}
                userTag={userTag}
                deviceId={deviceId}
                sourceApp={sourceApp}
                lmId={lmId}
                ancestorExclusion={titleNodeType}
            ></ToolbarAugnito>
        </HeadingToolbar>
        <Plate />
    </PlateProvider>
}

5 Mobile Mic support

Augnito Voice Services API allows user to dictate using the Augnito Mic app on a phone and the output will go to the client application.

To enable mobile microphone support set the includeMicrophoneButton prop of the toolbar to true. This will add a button to the toolbar that when clicked will pop up a QR code to be scanned in the mobile app. Scanning the QR code, or clicking, will dismiss the pop up.

There are two optional handlers. HandleMobileScan will fire when the QR code is scanned and handleMobileConnectionRequest will fire when the mobile microphone app connects.

---

Full example component

Assuming a working React App with Plate and it's dependencies set up:

import './App.css';
import { Plate, HeadingToolbar } from '@udecode/plate';
import { ToolbarAugnito, AugnitoAPIServer } from 'augnito-plate-button';
import 'augnito-plate-button/dist/style.css';

const server = AugnitoAPIServer.UK;
const accountCode = '';
const accessKey = '';
const userTag = '';
const deviceId = '';
const sourceApp = '';
const lmId = '';

function App() {
  return (
    <div className="App">
      <div className="editor">
          <PlateProvider>
              <HeadingToolbar>
                  <ToolbarAugnito
                      server={server}
                      accountCode={accountCode}
                      accessKey={accessKey}
                      userTag={userTag}
                      deviceId={deviceId}
                      sourceApp={sourceApp}
                      lmId={lmId}
                  ></ToolbarAugnito>
              </HeadingToolbar>
              <Plate />
          </PlateProvider>
      </div>
    </div>
  );
}

export default App;

ToolbarAugnito Props

Required

prop	type
server	AugnitoAPIServer (int where India = 0, UK = 1, US = 2)
accountCode	string
accessKey	string
userTag	string
sourceApp	string
lmId	string

Optional

prop	type	notes
deviceId	string
handlePartialText	(partial:string) => void	callback fired when partial text is received . Check the Advance Example.
handleCommand	(command: Partial<ActionRecipe>) => boolean	handles a command. Return true to interrupt normal flow. Check the Advance Example.
handleFinalText	(final: Partial<ActionRecipe>) => boolean	handles the resulting final text. Return true to interrup normal flow. Check the Advance Example.
handleIdleMic	() => void	callback fired when there has not been activity but the microphone is open. Check the Advance Example.
handleSessionEvent	(data: AugnitoSocketResponse) => void	callback to intercept Session Events. Check the Advance Example.
handleMobileScan	() => void	callback when a mobile device succesfully scans the QR code to link with the application.
handleMobileConnectionRequest	() => void	callback when a mobile device connects to the application.
augnitoStore	AugnitoStore	store to work with features such as select commands.
includeActivityIndicator	boolean	toggle the button activity indicator. The indicator shows the mic status and received partial text.
includeMicrophoneButton	boolean	toggle the mobile microphone qr generating button. Clicking the button will display a qr code for synching with a mobile device

Commands

Besides regular speech to text the button can process the following commands:

command	notes
stop mic	stops dictation. Sames as clicking the dictation button once started.
new line	same as hitting carriage return. If used within a list it creates a new list entry (see lists commands)
go to document start
go to document end
go to next sentence
go to previous sentence
go to sentence start | goto beginning of sentence
go to sentence end | goto end of sentence
undo it
redo it
select next word
select next N words
deselect	removes the just the selection (contents on the selection remain unchanged).
select last word
select last N words
select next sentence	if cursor not at the end of a sentence selects the current sentence.
select last sentence	same principe with select next sentence.
delete that	deletes the word under the cursor or the contents of a selection.
delete it	same as delete that command.
bold it	bolds the word under the cursor or the selected text. Requires createBoldPlugin().
bold that	same as bold it command.
italicize it	makes the current word or the selected text italic Requires createItalicPlugin().
italicize that	same as italicize it command.
underline it	underlines the current word or the selected text Requires createUnderlinePlugin().
underline that	same as underline it command.
capitalize it	capitalizes the current word or the selected text.
capitalize that	same as capitalize it command.
next field	moves to and selects the next dynamic field (next [] pair or next [content] occurrence).
previous field	moves to and selects the previous dynamic field (previous [] pair or previous [content] occurrence).
start number list	starts a numbered (ordered) list. Requires createListPlugin().
start bullet list	starts a bullet list. Requires createListPlugin().
stop bullet list | stop number list	ends the current list context. Requires createListPlugin().

Dynamic selection commands

Dyanmic selection doesn't update the nodes until they are selected in the current version of plate which prevents it working properly.

Support for selection command is included with the library. It requires the AugnitoSelectPlugin in order to work.

Augnito Select Plugin setup

1 - Import required parts

import {
  ToolbarAugnito,
  buildAugnitoStore,
  createAugnitoSelectPlugin,
  AugnitoAPIServer
} from 'augnito-plate-button';

import 'augnito-plate-button/dist/style.css';

2 - Setup plate to react to dynamic commands

Just like Plate's highlight the plugins now must be wrapped within a useMemo hook

const augnitoStore = useMemo(() => buildAugnitoStore(), []);
const searchTerm = augnitoStore.use.searchTerm();

const plugins = useMemo(
  () =>
    createPlugins(
      [
        //... list of desired plugins here
        createAugnitoSelectPlugin({
          options: {
            augnitoStore
          }
        })
      ],
      {
        components: createPlateUI()
      }
    ),
  [searchTerm]
);

3 - Include the AugnitoStore as a prop for ToolbarAugnito

<PlateProvider plugins={plugins}>
    <HeadingToolbar>
        <ToolbarAugnito
            server={server}
            accountCode={accountCode}
            accessKey={accessKey}
            userTag={userTag}
            deviceId={deviceId}
            sourceApp={sourceApp}
            lmId={lmId}
            augnitoStore={augnitoStore}
    ></ToolbarAugnito>
  </HeadingToolbar>
    <Plate />
</PlateProvider>

Full example

import {
  Plate,
  HeadingToolbar,
  createBoldPlugin,
  createItalicPlugin,
  createUnderlinePlugin,
  createListPlugin,
  createPlateUI,
  createPlugins
} from '@udecode/plate';
import {
  ToolbarAugnito,
  buildAugnitoStore,
  createAugnitoSelectPlugin,
  AugnitoAPIServer
} from 'augnito-plate-button';

import './App.css';

import { useMemo } from 'react';
import 'augnito-plate-button/dist/style.css';

const server = AugnitoAPIServer.UK;
const accountCode = '';
const accessKey = '';
const userTag = '';
const deviceId = '';
const sourceApp = '';
const lmId = '';

function App() {
  const augnitoStore = useMemo(() => buildAugnitoStore(), []);
  const searchTerm = augnitoStore.use.searchTerm();

  const plugins = useMemo(
    () =>
      createPlugins(
        [
          createListPlugin(), // list support
          createBoldPlugin(), // bold mark
          createItalicPlugin(), // italic mark
          createUnderlinePlugin(), // underline mark
          createAugnitoSelectPlugin({
            options: {
              augnitoStore
            }
          })
        ],
        {
          // Plate components
          components: createPlateUI()
        }
      ),
    [searchTerm]
  );

  return (
    <div className="App">
      <div className="editor">
        <PlateProvider plugins={plugins}>
          <HeadingToolbar>
            <ToolbarAugnito
              server={server}
              accountCode={accountCode}
              accessKey={accessKey}
              userTag={userTag}
              deviceId={deviceId}
              sourceApp={sourceApp}
              lmId={lmId}
              augnitoStore={augnitoStore}
            ></ToolbarAugnito>
          </HeadingToolbar>
          <Plate/>
        </PlateProvider>
      </div>
    </div>
  );
}

export default App;

List of Dynamic Commands

command	notes
select <word | phrase>	searches and selects the word or phrase. If more than one appears a numbered indicator allows to specify which one to select.
bold <word | phrase>	searches and bolds the word or phrase. If more than one appears a numbered indicator allows to specify which one to change.
underline <word | phrase>	searches and underlines the word or phrase. If more than one appears a numbered indicator allows to specify which one to change.
capitalize <word | phrase>	searches and capitalizes the word or phrase. If more than one appears a numbered indicator allows to specify which one to change.
italicize <word | phrase>	searches and converts to cursive the word or phrase. If more than one appears a numbered indicator allows to specify which one to change.

---

Advanced Example

The library can be further extended via ToolbarAugnito Props. The way to achieve this is to "short circuit" the handleCommand and handleFinalText methods.

Full example of a React (Typescript) app overriding these methods.

import './App.css';

import { useMemo, useCallback } from 'react';
import {
  Plate,
  HeadingToolbar,
  createBoldPlugin,
  createItalicPlugin,
  createUnderlinePlugin,
  createListPlugin,
  createPlateUI,
  createPlugins
} from '@udecode/plate';
import {
  ToolbarAugnito, // button (enabled TTS)
  buildAugnitoStore, // creates sync store
  createAugnitoSelectPlugin, // dynamic command support plugin
  ActionRecipe, // handler type for final text and command
  AugnitoAPIServer
} from 'augnito-plate-button';
import 'augnito-plate-button/dist/style.css'; // styles

// provided auth values
const server = AugnitoAPIServer.UK;
const accountCode = '';
const accessKey = '';
const userTag = '';
const deviceId = '';
const sourceApp = '';
const lmId = '';

function App() {
  // support for dynamic commands
  const augnitoStore = useMemo(() => buildAugnitoStore(), []);
  const searchTerm = augnitoStore.use.searchTerm();

  // use memo required for dynamic commands
  const plugins = useMemo(
    () =>
      createPlugins(
        [
          createListPlugin(), // list support
          createBoldPlugin(), // bold mark
          createItalicPlugin(), // italic mark
          createUnderlinePlugin(), // underline mark
          createAugnitoSelectPlugin({
            options: {
              augnitoStore
            }
          })
        ],
        {
          // Plate components
          components: createPlateUI()
        }
      ),
    [searchTerm]
  );

  // #region custom handlers
  const handlePartialText = useCallback((partial: string) => {
    console.log(partial);
  }, []);

  const handleIdleMic = useCallback(() => {
    console.log('idle mic');
  }, []);

  const handleSessionEvent = useCallback((data:Partial<AugnitoSocketResponse>)=>{
    console.log(data);
  },[]);

  const handleFinalText = useCallback((finalText: Partial<ActionRecipe>) => {
    const handled = true; // or false if the SDK should continue normal flow
    console.log({
      handler: 'final text',
      handled,
      text: finalText.receivedText
    });
    return handled;
  }, []);

  const handleCommand = useCallback((command: Partial<ActionRecipe>) => {
    const handled = true; // or false if the SDK should continue normal flow
    console.log({ handler: 'command', handled, action: command.action });
    switch (command.action) {
      case '':
        break;

      default:
        break;
    }
    return handled;
  }, []);

    const handleMobileScan = useCallback(()=>{
        console.log("Mobile scanned");
    },[]);

    const handleMobileConnectionRequest = useCallback(()=>{
        console.log("Mobile connected");
    },[]);
  
  // #endregion

  return (
    <div className="App">
      <div className="editor">
        <PlateProvider plugins={plugins}>
          <HeadingToolbar>
            <ToolbarAugnito
              server={server}
              accountCode={accountCode}
              accessKey={accessKey}
              userTag={userTag}
              deviceId={deviceId}
              sourceApp={sourceApp}
              lmId={lmId}
              augnitoStore={augnitoStore}
              includeMicrophoneButton={true}
              handleCommand={handleCommand}
              handleFinalText={handleFinalText}
              handleIdleMic={handleIdleMic}
              handlePartialText={handlePartialText}
              handleSessionEvent={handleSessionEvent}
              handleMobileScan={handleMobileScan}
              handleMobileConnectionRequest={handleMobileConnectionRequest}
            ></ToolbarAugnito>
          </HeadingToolbar>
          <Plate />
        </PlateProvider>
      </div>
    </div>
  );
}

export default App;

---

Version support

The Augnito Plate Button and Augnito Select Plugin use many functions, API, structures and code from Plate, Slate, Rect and such. To ensure performance and bundle size those dependencies are expected to be installed on the consumer solution source (as stated on the required dependencies section) which means these are present only once on the project. The versions that should not break compatibility are:

library	semver
@udecode/plate	>=18.14.0
react	>=17
react-dom	>=17
slate	>=0.82.0
slate-history	>=0.86.0
slate-hyperscript	>=0.77.0
slate-react	>=0.79.0
styled-components	^5