@webex/web-client-media-engine

Web Client Media Engine (WCME)

Web Client Media Engine is common web code for interacting with the multistream media server.

UML diagram generated with tplant.

Setup

1. Run yarn to install dependencies.
2. Run yarn watch to build and watch for updates.
3. Run yarn test to build, run tests, lint, and run test coverage.

Multistream Connection

The MultistreamConnection object is the primary interface for clients to interact with the media server. It maintains a list of transceivers, as well as handles media requests to and from the media server. Each MultistreamConnection has a single RTCPeerConnection that is shared across all MediaTypes. It also creates a RTCDataChannel to send and receive JMP messages.

Example

Establish a multistream connection and start sending audio/video:

const multistreamConnection = new MultistreamConnection();

const offer = await multistreamConnection.createOffer();
// after sending offer to server and receiving answer
await multistreamConnection.setAnswer(answer);

const audioSendSlot = multistreamConnection.createSendSlot(MediaType.AudioMain);
const localAudioStream = createMicrophoneStream(LocalMicrophoneStream);
audioSendSlot.publishStream(localAudioStream);

const videoSendSlot = multistreamConnection.createSendSlot(MediaType.VideoMain);
const localVideoStream = createCameraStream(LocalCameraStream);
videoSendSlot.publishStream(localVideoStream);

SDP Management

Clients want to be able to send audio and video (limited to main audio, main video, content audio, and content video) as well as receive any number of audio and video streams. Unified plan in WebRTC means there's a limitation that only a single stream can be signaled per mline. The media server, however, only supports four mlines (main audio, main video, content audio, and content video), so in order to receive more we need to manipulate the SDP in a way that works for both the browser and the server.

To accomplish this, there are mlines that only the browser knows about -- these mlines are "filtered out" before sending an offer to the server, and corresponding mlines are "injected" into the answer from the server. This works for a few reasons:

• The browser is configured to use one ICE bundle group for each media type: this means that the additional mlines don't require extra ICE connections.
• The browser only needs packets tagged in a certain way to receive them correctly. The media server supports this tagging, so packets are routed to the correct RTCRtpReceiver.
• JMP signaling allows signaling to the media server how to tag the packets, so the client can tell the server how to tag media such that it's received correctly.

An example (truncated) client SDP:

  v=0
  ...
  // An audio mline for sending main audio
  m=audio 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:0
  a=sendrecv
  a=jmp
  a=jmp-source:0 csi=677569024
  // A video mline for sending main video
  m=video 63722 UDP/TLS/RTP/SAVPF 127 125 108 124 123 35 114
  a=mid:1
  a=sendrecv
  a=jmp
  a=jmp-source:1 csi=677569025
  // An audio mline for sending content audio
  m=audio 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:2
  a=sendrecv
  a=content:slides
  a=jmp
  a=jmp-source:2 csi=777569024
  // A video mline for sending content video
  m=video 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:3
  a=sendrecv
  a=content:slides
  a=jmp
  a=jmp-source:3 csi=777569025
  // 3 recvonly audio mlines for receiving audio
  m=audio 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:4
  a=recvonly
  m=audio 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:5
  a=recvonly
  m=audio 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:6
  a=recvonly
  // 3 recvonly video mlines for receiving video
  m=video 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:7
  a=recvonly
  m=video 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:8
  a=recvonly
  m=video 56200 UDP/TLS/RTP/SAVPF 111
  a=mid:9
  a=recvonly
  // mline for datachannel
  m=application 55527 UDP/DTLS/SCTP webrtc-datachannel
  a=mid:10

In the above SDP, only mlines 0, 1, 2, 3 and 10 will be sent to Homer. The rest are only seen by the client and are used to generate RTCRtpTransceivers to act as handles for media received on those mlines. When the answer from Homer is received, it will only contain mlines 0, 1, 2, 3 and 10. The client will then generate synthetic answer mlines to match the "hidden" ones (4, 5, 6, 7, 8, and 9) before setting the remote description.

Mlines 4, 5, 6, 7, 8, and 9 serve only as ways for generating tracks associated with a MID on the client. To request media on those tracks, the client sends a MediaRequest (described below) with the appropriate MID.

MultistreamConnection handles all the SDP manipulation required to accomplish the above for both the offer and the answer. It also performs additional preprocessing on the offer such as injecting content types and JMP attributes. All of this is handled by the appropriate SDP "mungers" (IngressSdpMunger and EgressSdpMunger), which are called before setting the local offer, sending the offer to the remote server, and setting the remote answer.

Transceivers and Slots

In WebRTC, RTCRtpTransceivers represent a pairing of send and receive SRTP streams between the client and server. WCME defines two classes of transceivers: SendOnlyTransceiver and RecvOnlyTransceiver. Each MediaType (VideoMain, AudioMain, VideoSlides, or AudioSlides) can only have one SendOnlyTransceiver but may have multiple RecvOnlyTransceivers. MultistreamConnection maintains a list of all transceivers in a connection per MediaType.

Although SendOnlyTransceivers are only used for sending, its underlying RTCRtpTransceiver direction is set to "sendrecv" in order to make it compatible with Homer. Each SendOnlyTransceiver maintains the state of the sending stream -- whether or not it is published, whether or not it has been requested by a remote peer -- as well as handles replacing the existing stream with a new one (e.g. when switching camera devices).

Likewise, RecvOnlyTransceivers maintain the state of receiving streams.

Clients should never have to interact with transceivers directly. Instead, all interactions with transceivers are handled via "slots". Each SendOnlyTransceiver is associated with a single SendSlot, while each RecvOnlyTransceiver is associated with a single ReceiveSlot.

Receive Slots

WebRTC clients need to be able to receive remote media, but creating a stream for every remote participant's media would result in a huge SDP, and would require that the SDP was updated every time a client joined or left the session. So instead of allocating an RTCRtpReceiver for every remote participant, the receivers are treated as "slots" on which a remote participant (CSI) can be requested. ReceiveSlots have an ID (which is the MID of the corresponding mline), and media is requested via JMP using these IDs. For example, if a client wants to receive the 3 most active speakers' audio, then it needs to create only 3 main audio receive slots, even if there are 25 participants in the meeting.

The media received on a ReceiveSlot can change over time, depending on the policy requested on that slot and/or future media requests on that slot. sourceAdvertisement messages are used by the server to notify the client which CSI is being received on a ReceiveSlot at a given time.

Send Slots

Similarly, send slots are used to handle sending local media.

By default, all SendOnlyTransceivers are inactive (that is, the direction in the SDP offer is set to "inactive") until a SendSlot is created, during which the transceiver can be activated (the direction set to "sendrecv"). Only one SendSlot per media type should be created at a time, and SendSlots can be activated or deactivated on the fly.

Requesting Media

The requestMedia API is used to request media from the media server. It takes in a MediaType (VideoMain, AudioMain, VideoSlides, or AudioSlides) and an array of MediaRequest objects. A MediaRequest object consists of a policy (ActiveSpeaker or ReceiverSelected), some policy-specific information, and an array of ReceiveSlots on which the requested media will be received.

requestMedia(mediaType: MediaType, mediaRequests: MediaRequest[]): void

The MediaRequest object is an abstraction on the JMP media request object, and allows crafting different combinations of policies and ReceiveSlots to achieve different behaviors. A MediaRequest consists of:

• Policy
• PolicySpecificInfo
• ReceiveSlot[]

Details for these fields are the same as they are for the JMP media request. Information can be found here.

Examples

Requesting the audio of the 3 active speakers:

// Create the receive slots
const audioSlot1 = await createReceiveSlot(MediaType.AudioMain);
const audioSlot2 = await createReceiveSlot(MediaType.AudioMain);
const audioSlot3 = await createReceiveSlot(MediaType.AudioMain);
requestMedia(MediaType.AudioMain, [
  new MediaRequest(Policy.ActiveSpeaker, new ActiveSpeakerInfo(100, false, false, true), [
    audioSlot1,
    audioSlot2,
    audioSlot3,
  ]),
]);

Requesting a the video of a specific CSI:

// Create the receive slots
const videoSlot1 = await createReceiveSlot(MediaType.VideoMain);
requestMedia(MediaType.VideoMain, [
  new MediaRequest(Policy.ReceiverSelected, new ReceiverSelectedInfo(csiToSelect), [videoSlot1]),
]);

Exports From Other Repositories

In addition to WCME exports, all necessary imports from webrtc-core and json-multistream are also re-exported for your convenience, so they can be used in other code without the need to import from those repositories separately.

Logging

Logging is done through the js-logger library and the Logger class is exported to be used with other repositories. This can be done by importing and setting a handler for Logger.

import { Logger } from '@webex/web-client-media-engine';

Logger.setHandler((msgs, context) => {
  // do something with logs
  console.log(context.name, msgs);
});

Loggers from webrtc-core and json-multistream are also re-exported if you want to handle logs from those repositories separately.