@akkadu/rtc-streamer-consumer

• RTC Consumer
  • Purpose of this module
  • Core concepts
    • Publisher Ids
    • How Agora plays back audio
    • Language State
    • Autoplay issues
    • Documentation of the class interface
    • Special edge cases
      • iPhone Subscription and video playback
      • AgoraRTS limitations
    • Communication with the module
  • Structure
  • Commands
  • Development
  • Debugging
    • Logging
    • Environment setup

RTC Consumer

To understand the larger context where this module operates, if you are not already familiar, take a look at our Agora RTC documentation here

In addition this module extends the rtc-streamer-base the documentation of which can be found here

Purpose of this module

This module extends the module rtc-streamer-base to allows receiving remote streams, more specifically it allows us to

• Subscribe and unsubscribe specific publishers
• Subscribe and unsubscribe a language
• Mute and unmute a specific publishers
• Mute and unmute a language
• Play or stop a specific publisher
• Play or stop a language

As you can see above it has a two levels of abstraction, where a publisher has their on id, but an id will also be tied to a language.

Core concepts

Publisher Ids

Each Agora client needs to have a UID. To use the Languages model data from our database served through our streams api, for hosts we are using their actual userId from the db as their UID. As for non-logged-in audience members we are just using a unique high resolution timestamp.

With this a user might have a UID of 12, but we append a stream- prefix to make it into a string. Hence all host users will be identified by an id like "stream-12" where the number is their id from the db. The reasoning for this is in upcoming sections of "How Agora plays back audio" and "Language State"

How Agora plays back audio

In order for us to initialize Agora we need to have a container in the dom in which we can allow Agora to insert stream elements. Each incoming remote stream will have it's own container which Agora manipulates to play video and audio. So what we basically have is one master container that can have multiple stream inside of itself. This master containers id needs to be passed in the configuration of this module with config container. In general we have added a div into the dom where this module is used with id='stream-container'.

When we want to subscribe to for instance stream-12 as mentioned in the previous part, we will have to give that to the stream.play(method) used in our method playStreamsById, then Agora takes this id and appends elements inside of it.

As you can see below Agora adds a div with an id of player_12 inside our container and adds and audio and a video tag to do playback

  <!-- Created by us -->
  <div id="stream-container">
    <!-- Also created by us -->
    <div id="stream-12"> 
      <!-- Created by Agora -->
      <div id="player_12">
        <video id="video641" style="width: 100%; height: 100%; position: absolute; object-fit: contain;" autoplay="" muted="" playsinline=""></video>
        <audio id="audio641" playsinline=""></audio>
      </div>
    </div>
  </div>

If you are using AgoraRTS (see section debugging to see how to manually enable it) the elements will look a bit different. In fact it seems that Agora is using a canvas element to draw out the incoming video stream. Without knowing the implementation better, my educated guess is that they are sending either the combined audio and video in one stream, or separately, over a websocket connection. After this they decode that to something that can be drawn to the canvas and played back using web audio api and asm/wasm decoders.

<!-- Created by us -->
<div id="stream-container">
  <!-- Also create by us -->
  <div id="stream-641">
    <!-- Created by Agora -->
    <div data-mode="contain" style="width: 100%; height: 100%; position: relative; overflow: hidden;">
      <canvas width="1920" height="1080" id="player-641" style="background-color: rgb(0, 0, 0); width: 100%; height: initial; top: 94.8438px; left: initial; position: absolute;"></canvas>
    </div>
  </div>
</div>

So now we know that Agora needs a container to append it's stream elements inside and we need a consistent way to identify separate streams. Since an integer or just 12 does not work as a css selector we've added the stream- prefix. For consistency that prefix syntax is used as stream selector in all parts of the RTC module.

Language State

Language state seeks to answer these questions per language:

• Should we autoplay any subscriber who starts publishing in this language?
• Should we subscribe publishers of this language?
• Should we play audio from publishers of this language?
• Should we play video (including screen and webcam sharing) from publishers of this language?
• Should we subscribe to audio from publishers of this language?
• Should we subscribe to video from publishers of this language?

The default languageState per language is

const defaultLanguageState = {
  autoPlay:false,
  subscribe:false,
  playAudio:false,
  playVideo:false,
  subscribeToAudio:false,
  subscribeToVideo:false
}

which is stored per language

this.consumerConfig.languageState[`${language}`] = defaultLanguageState

This is then modified during runtime of the module via it's methods.

In addition to this we track which publishers are connected and which language they are publishing inside this.consumerConfig.connectedPublishers. More specifically when a new stream is published Agora client will fire a 'stream-added' event:

this.client.on('stream-added', (evt) => {
  this.handleStreamAdded(evt)
})

then in handleStreamAdded we call rtc-streamer-base method of getPublisherLanguage to define which language the added publisher is supposed to be speaking. For instance if we are subscribed to en-US, and the our languageState says to autoPlay all English streams, and we get a new en-US publisher, the handleStreamAdded function will get that information from rtc-base getPublisherLanguage (which itself is based on data from streams api) and we know to autoplay this publisher. Then if another publisher connects and we see that they are supposed to speak zh-CN, we would not autoplay that stream.

connecterPublishers object looks like this:

{
 'stream-57':{
    subscribed:true, // whether or not we have subscribed this stream
    subscribeError:false, // see below
    language:'en-US', // what is this publishers language
    stream: AgoraStream, // a separate Agora stream type given to us
    domId: 'stream-57', // for easier access
    hasAudio: true, // whether or not the publisher is sharing audio
    hasVideo: false // whether or not the publisher is sharing video
 }  
}

subscribeError This error is tracked for some edge case handling. Namely if we are on an autoplay blocked device such as Iphones, when subscribing to a language let's say 1 out of 4 publishers subscribe fails, we'd still want to play the three other publishers. You can see this in the play methods of playStreamsById. If the fourth publisher comes online again this will be stopped by the browser and a stream-stuck event will be fired. More on this in section Autoplay issues

Autoplay issues

On certain devices and browsers, especially on Safari and never Chrome browsers as outlined by Agora the playback of a stream must be initialized by user action. In a normal streaming situation We might have a situation for instance that our event has two en-US publishers, which are defined in client.baseConfig.publishers as

 {
  'en-US':['stream-57', 'stream-58'] 
 }

But on the other hand maybe only host of stream-57 is connected and we have subscribed to them already, hence connectedPublishers looks like this:

{
 'stream-57':{
    subscribed:true,
    subscribeError:false,
    language:'en-US',
    stream: AgoraStream // a separate Agora stream type given to us
    publisher.domId = domId
    publisher.hasAudio = hasAudio
    publisher.hasVideo = hasVideo

 }  
}

and our language state would look like below. For the sake of the example we are just subscribing audio note that this is not possible for ios devices, more on this in section Iphone Subscription

{
 'en-US':
  {
    autoPlay:true,
    subscribe:true,
    playAudio:true,
    playVideo:false,
    subscribeToAudio:true,
    subscribeToVideo:false
  }  
}

now let's assume that stream-58 comes online and the handleStreamAdded function fires. Since we are subscribing and autoplaying this streams language, we try to autoplay it. However because this is not initialized by a user interaction the stream will be stuck. This will emit a stream-stuck event in the playStreamsById method:

 if (err.status && err.status !== 'aborted') {
  this.emitter.emit('consumer:stream-stuck', { domId, language:publisher.language })
 }

To resume this stream, we need to call playLanguage again by user action. At the moment we have a resume button show up on the UI which is a good way to handle the situation. Calling play on a already playing language or a specific stream does not cause issues.

Documentation of the class interface

We use a index.d.ts file to document the interface of this class. The reason to do so was that by doing so we don't have to bundle our function definitions with the code and intellisense can find the definitions by the typings reference. Of course we could use a bundler to remove the comments as well at build time and distribute the bundle to reduce the footprint of the code, but if we did this then anyone using this module would not have typings available for them and we'd have to provide another way to give them that, which is exactly what the index.d.ts file is doing. An added bonus with this is that you can use this file with Typescript, although it hasn't been tested to work in a ts environment and would most likely need some fixes to it.

Special edge cases

iPhone Subscription and video playback

Whereas normally we can pass selection on whether or not we want to subscribe to audio or video with safari we cannot do this. As mentioned in the Agora Docs

Safari does not support independent subscription. Set options as null for Safari, otherwise theSAFARI_NOT_SUPPORTED_FOR_TRACK_SUBSCRIPTION error occurs.

Hence we are setting the options to null on subscription in subscribeStreamsById

 const subscribeOptions = this.browserInfo.isIos ? null : {
    audio: subscribeToAudio, 
    video: subscribeToVideo
  }

Another trouble here is that is recent tests we've noticed that if

1. Host is streaming video
2. Audience using an iphone only plays audio, not video The audio will not play. We have a relevant issue open on this topic here the quick fix to this was done in this PR where effectively we are always playing video if available but just hiding it if we'd want to have only the audio play.

AgoraRTS limitations

Note! We are running a old version of AgoraRTS and I only recently realized Agora is providing updates to it. It might be advised to update to the latest version. More on the topic here

These notes apply to Agora RTS version 2.8.0.600

Resume Agora RTC has a stream.resume method, but AgoraRTS does not have it. Newer versions might have this changed. At the moment we are checking if the method is defined and catching errors locally

stream.resume?.()
  .then(() => this.emitter.emit('consumer:playing-id', { domId }))
  .catch(error => this.emitter.emit('error',error))

Communication with the module

To simplify usage in the frontend and to make it easier to have consistent behavior with Agora's inconsistent streaming module all communication is done through emits. The purpose here is to:

1. Allow for a consistent module API even if we change away from Agora
2. Workaround some of the limitations of Agora discussed below
3. Decouple frontend state from actions. With emits we can have one function initializing an action and just reacting to the module emit no matter the source of the event.

This means that when you call a method you should expect a corresponding emit to be fired from the streamer. To see what emits functions are sending see index.d.ts file or use intellisense's documentation user @fires point

Most actions against Agora are asyncronous and furthermore we cannot await them always. For instance the method client.subscribe documented here only has an onFailure callback, so the only way to know that we have subscribed to a stream is to listen to events from the client and update the information locally. In this case we are doing it in the subscribe handler:

this.client.on('stream-subscribed', (evt) => {
  this.handleStreamSubscribed(evt)
})

Structure

Documentation

• index.d.ts defines the modules interface, this is your main source of method definitions

Implementation

• index.js

Tests

• ./tests/test.js

Commands

Build yarn build

Run Tests yarn test

Development

There are two ways of development, you can either develop inside the repository using the rtc-test-server module or follow the instructions here to export this module outside the repository

Debugging

Logging

This module accepts a logger to be passed into it and it logs on levels

• error
• warn
• debug

In addition warn and error event are being emitted. A compatible logger is for instance our own @akkadu/logger. The internal log function is safe for missing logger methods, and if for instance the given logger does not implement level debug this does not cause errors, the logs just do not show.

Environment setup

Sometimes it is good to be able to either force the browser to user AgoraRTC or AgoraRTS for testing purposes. This can be done via local storage in the browser console:

AgoraRTS

localStorage.forceRTCFallback = true

AgoraRTC

localStorage.forceRTC = true

if both keys are defined in localStorage, AgoraRTC takes precedence.