symbl-real-time-nebula

symbl-real-time-nebula

A real time copilot assistant for sales agent to help make quick suggestions whiles they are on a call with a customer.

Requirements for initialising the SDK

To initialise the sdk(CopilotAssistantAgent) you need to pass an object with the following parameters.

• symblAccessToken - Symbl Token. You can get it from Platform (required)
• nebulaApiKey - Nebula API Key. You can get it from Nebula (required)
• conversationId - Conversation ID from symbl session. (required)
• connectionId - Connection ID from symbl session. (required)
• trackers - An array of trackers to be used for the conversation. (optional)
• enableAllTrackers - A boolean to enable all trackers. (optional) Which is true by default.
• enableInterimResults - A boolean to enable interim results. (optional) Which is true by default.

NOTE: You can't pass both trackers and enableAllTrackers to be true at the same time. If you pass both, trackers will be used.

Trackers Object

The trackers array is a collection of tracker objects. Each tracker is used to identify specific phrases or words within a given context. Trackers are primarily characterized by a name and a vocabulary list. An optional id can also be included for each tracker, typically sourced from a management API.

Structure

Each element in the trackers array is an object with the following fields:

• name (string): The name of the tracker, representing the category or type of phrases it tracks.
• vocabulary (array of strings): A list of phrases or words that this tracker is responsible for identifying.
• id (string, optional): A unique identifier for the tracker, usually obtained from a management API.

When the sdk is initialised, it will automatically connect to the nebula websocket and start listening for suggestions based on the first stage and the meta data provided.

const sdk = CopilotAssistantAgent({
  symblAccessToken, // required
  nebulaApiKey, // required
  conversationId, // required
  connectionId, // required
  trackers, // optional
  enableAllTrackers, // optional
  enableInterimResults,
});

Callback functions

onSuggestions Callback

The onSuggestions method accepts a callback function that is invoked when suggestions are available.

Syntax

sdk.onSuggestions(callback);

The callback function takes an object as a parameter. This object contains a suggestion property which is of type string

{
  suggestion: "Hi ...";
}

• suggestion - Is of type string and contains the suggestion from nebula.

sdk.onSuggestions(({ suggestion }) => {
  console.log(suggestion);
});

onCompletedTranscriptions

This method is called when the sdk receives a transcription from symbl with the isFinal as true. It returns the following parameters.

Syntax:

sdk.onCompletedTranscriptions(callback);

The callback function takes an array of objects as a parameter. This array contains the transcriptions from symbl of format below and is called messages

{
  messages: [
    {
      from: {
        id: string,
        name: string,
        userId: string,
      },
      payload: {
        content: string,
        contentType: "text/plain",
      },
      id: string,
      channel: {
        id: "realtime-api",
      },
      metadata: {
        disablePunctuation: true,
        originalContent: string,
        words: string,
        originalMessageId: string,
      },
      dismissed: false,
      duration: {
        startTime: string,
        endTime: string,
        timeOffset: number,
        duration: number,
      },
      entities: [],
    },
  ];
}

• messages - Is of type array and contains the transcriptions from symbl of format; messages

  • from - Is of type object and contains the user or speaker information.
    • id - Is of type string and contains the id of the user or speaker.
    • name - Is of type string and contains the name of the user or speaker.
    • userId - Is of type string and contains the userId of the user or speaker.
  • payload - Is of type object and contains the payload of the transcription.
    • content - Is of type string and contains the content of the transcription.
    • contentType - Is of type string and contains the contentType of the transcription.
  • id - Is of type string and contains the id of the transcription.
  • channel - Is of type object and contains the channel of the transcription.
    • id - Is of type string and contains the id of the channel.
  • metadata - Is of type object and contains the metadata of the transcription.
    • disablePunctuation - Is of type boolean and contains the disablePunctuation of the transcription.
    • originalContent - Is of type string and contains the originalContent of the transcription.
    • words - Is of type string and contains the words of the transcription.
    • originalMessageId - Is of type string and contains the originalMessageId of the transcription.
  • dismissed - Is of type boolean and contains the dismissed of the transcription.
  • duration - Is of type object and contains the duration of the transcription.
    • startTime - Is of type string and contains the startTime of the transcription.
    • endTime - Is of type string and contains the endTime of the transcription.
    • timeOffset - Is of type number and contains the timeOffset of the transcription.
    • duration - Is of type number and contains the duration of the transcription.
  • entities - Is of type array and contains the entities of the transcription.

  sdk.onCompletedTranscriptions(({ messages }) => {
    console.log(messages);
  });
  

onRealtimeTranscript

This method is called when the sdk receives a transcription from symbl in realtime as the users speak. It returns the following parameters.

Syntax:

sdk.onRealtimeTranscript(callback);

The callback function takes an object as a parameter. This object contains a message property which is of type object

{
  message: {
    type: "recognition_result",
    isFinal: false,
    payload: {
      raw: {
        alternatives: [
          {
            words: [],
            transcript: "Hello",
            confidence: 0,
          },
        ],
      },
    },
    punctuated: {
      transcript: "Hello",
    },
    user: {
      name: "agent",
      userId: "[e-mail]",
    "id": "1a726dac-1c95-484d-a99e-69e36950bf32"
  }
}}

• message - Is of type object and contains the transcription from symbl.
  • type - Is of type string and contains the type of the transcription.
  • isFinal - Is of type boolean and contains the isFinal of the transcription.
  • payload - Is of type object and contains the payload of the transcription.
    • raw - Is of type object and contains the raw of the transcription.
      • alternatives - Is of type array and contains the alternatives of the transcription.
        • words - Is of type array and contains the words of the transcription.
        • transcript - Is of type string and contains the transcript of the transcription.
        • confidence - Is of type number and contains the confidence of the transcription.
  • punctuated - Is of type object and contains the punctuated of the transcription.
    • transcript - Is of type string and contains the transcript of the transcription.
  • user - Is of type object and contains the user or speaker information.
    • name - Is of type string and contains the name of the user or speaker.
    • userId - Is of type string and contains the userId of the user or speaker.
  • id - Is of type string and contains the id of the transcription.

To access the actual transcript, you can use the following code snippet.

sdk.onRealtimeTranscript(({ message }) => {
  const { transcript } = message.puntuated;
  console.log(transcript);
});

You can also access the user or speaker name using the following code snippet.

sdk.onRealtimeTranscript(({ message }) => {
  const { name } = message.user;
  console.log(name);
});

onSymblResponse

This method is called when the sdk receives any response from symbl. It returns the following parameters.

Syntax:

sdk.onSymblResponse(callback);
// or
sdk.onSymblResponse(({ response }) => {
  console.log(response);
});

The callback function takes an object as a parameter. This object contains a response property. To check the possible details of the response visit this url Syml Streaming API Docs Also note to look of for the following in the docs after visiting the url;

• Response reference
  • message
  • message_response
  • entity_response`
  • tracker_response
  • topic_response`
  • bookmark_response
  • insight_response
  • request_modified

onTrackerResponse

This method is called when the sdk receives a tracker response from symbl. It returns the following parameters.

Syntax:

sdk.onTrackerResponse(callback);
// or
sdk.onTrackerResponse(({ trackers }) => {
  console.log(trackers);
});

The callback function takes an array of objects. This object contains a trackers property which is of type array

{
  trackers: [
    {
      name: "test",
      matches: [
        {
          type: "vocabulary",
          value: "test",
          messageRefs: [
            {
              id: "7c4b04c6-cc86-4ef8-8a97-a26c1eb9a21f",
              text: "It was a test.",
              offset: 9,
            },
          ],
          insightRefs: [],
        },
      ],
    },
  ];
}

• trackers - Is of type array and contains the trackers from symbl.
  • name - Is of type string and contains the name of the tracker.
  • matches - Is of type array and contains the matches of the tracker.
    • type - Is of type string and contains the type of the match.
    • value - Is of type string and contains the value of the match.
    • messageRefs - Is of type array and contains the messageRefs of the match.
      • id - Is of type string and contains the id of the messageRefs.
      • text - Is of type string and contains the text of the messageRefs.
      • offset - Is of type number and contains the offset of the messageRefs.
    • insightRefs - Is of type array and contains the insightRefs of the match.

Functions

getTranscriptions

This method returns the transcriptions received from symbl. It accepted an option parameter called fromMemory which is false by default. When set to true, it returns the transcriptions from memory. When set to false, it returns the transcriptions making request to symbl api. It returns as an array of objects, with the format;

[
  {
    "from": {
      "id": string,
      "name": string,
      "userId": string
    },
    "payload": {
      "content": string,
      "contentType": "text/plain"
    },
    "id": string,
    "channel": {
      "id": "realtime-api"
    },
    "metadata": {
      "disablePunctuation": true,
      "originalContent": string,
      "words": string,
      "originalMessageId": string
    },
    "dismissed": false,
    "duration": {
      "startTime": string,
      "endTime": string,
      "timeOffset": number,
      "duration": number
    },
    "entities": []
  }
]

Syntax:

const transcriptions = await sdk.getTranscriptions(fromMemory);
// or
const transcriptions = await sdk.getTranscriptions(true);

stop

This method stops the sdk from listening to the symbl websocket and closes the connection.

Syntax:

sdk.stop();

callNebulaForSuggestions

This function calls the nebula api for suggestions. The suggestions are received from the onSuggestions callback function. It returns the following parameters.

Parameters:

• options (Object): The options for calling Nebula.
  • systemPrompt (string): The system prompt for Nebula which is optional.
  • messages (Array): The messages to send to Nebula.
    • messages[].role (string): The role of the message (either "human" or "assistant").
    • messages[].text (string): The text content of the message.

    You can get more information about Nebula Chat model Returns:

    • A promise that resolves when the response from Nebula is processed. The return type is Promise<void>.

    Example:

    await sdk.callNebulaAPI({
      systemPrompt: "Help answer question asked politely",
      messages: [
        {
          role: "human",
          text: "Agent: Hello how are you \n User: This is Samson",
        },
      ],
    });
    

    Latency

    One important requirement for the solution is to keep latency to the minimum while keeping some reasonable response from Nebula LLM.

    How we measure latency?

    We broke it down in different blocks:

    1. Capture and time to send audio to Symbl.ai (depending on your location/network that can range between 100-500 ms)
    2. Receive tracker events and transcription (500 - 1000 ms)
    3. Prompt generation and Nebula response time (~80 ms, worst case). This is the time it takes until we get first character of the Nebula response.
    4. Display Latency (~0ms(For the JS, sdk) and ~900ms when using the python server sidem using socket.io)
    5. Also before the initialisation process, that is, the connection to the websocket symbl conversation, there is a delay of about 1-2 seconds.

    Results

    Average latency (excluding the initial audio communication): ~1865ms

    Note: External factors such as network variability, server load, and the performance of the client's device significantly impact end-to-end latency.