@webex/web-media-effects

web-media-effects

Web Media Effects is a suite of media effects developed for web SDKs and WebRTC media applications.

Introduction

There are three effects included in this library:

• Virtual background (e.g., blur, image replacement, video replacement)
• Noise reduction (e.g., background noise removal)
• Gain (for testing)

Common Methods

The effects are built on top of a plugin interface that makes building and extending effects more straight-forward.

The effects plugins have four primary methods to control the plugin:

• load(stream) takes in a MediaStream object and returns a new MediaStream with the effect applied
• enable() enables the plugin after it's loaded
• disable() disables the plugin after it's loaded
• dispose() tears down the effect

Additionally, there are a few convenience methods:

• getUnderlyingStream() returns the new outgoing (i.e., "effected") underlying stream
• setEnabled(boolean) sets the effect state by passing in a boolean (convenient for state managers)

Virtual Background

The virtual background effect is a wrapper around ladon-ts that provides a virtual background for video calling. The virtual background may be an image, an mp4 video, or the user's background with blur applied. The blur option allows for varied levels of strength and quality where higher levels require more compute resources.

The virtual-background-effect takes an optional VirtualBackgroundEffectOptions config object in its constructor. The effect's options can be changed at runtime via an updateOptions() method. When disabled, the effect simply passes through the original video images so that the outgoing stream does not need to be changed.

The effect uses a background thread worker by default to prevent slowdowns on the main UI thread. The main UI thread can be used instead by adding the property generator: 'local' in the VirtualBackgroundEffectOptions object. However, this is not recommended as the worker thread performs much better.

NOTE: For backwards compatibility, the default mode is set to BLUR.

Options

There are a few different options that can be supplied to the constructor or updateOptions() method that affect the behavior

Name	Description	Values	Required
authToken	Used to authenticate the request for the backend models	An encoded string token	Yes
generator	Determines where the model runs (on main thread or background thread)	local worker	Defaults to worker
frameRate	Determines how many frames per second are sent to the model	0-60	Defaults to 30
quality	Determines the accuracy of the model (higher requires more CPU)	LOW MEDIUM HIGH ULTRA	Defaults to LOW
mirror	Whether the output image should be flipped horizontally	true false	Defaults to false
mode	Determines what kind of background to render behind the user	BLUR IMAGE VIDEO	Defaults to BLUR
blurStrength	How strongly the background should be blurred	WEAK MODERATE STRONG STRONGER STRONGEST	Required in BLUR mode
bgImageUrl	Path to the background image to replace the original background	Fully qualified URL	Required in IMAGE mode
bgVideoUrl	Path to the background video to replace the original background	Fully qualified URL (mp4 only)	Required in VIDEO mode
env	Which environment the effect is running in.	EffectEnv.Production EffectEnv.Integration	Defaults to EffectEnv.Production

Mode

The virtual background plugin applies a background effect to the original media stream by performing image segmentation on the incoming video frames. The plugin is capable of applying three different kinds of effects called modes: background blur, background image replacement, and background video replacement.

The mode configuration option is determines what background effect to apply. There are three accepted values for the mode: BLUR, IMAGE, and VIDEO. Each mode has at least one required option that needs to be set in the options object, which is outlined below in the Options section.

NOTE: For Typescript users, the mode can be selected by using the exported VirtualBackgroundMode enum, for convenience.

Usage

Supply a video stream to the effect and when loaded, it will return a new stream with the effect applied.

// Create a new video stream by a getting user's video media.
const originalVideoStream = await navigator.mediaDevices.getUserMedia({ video: { width, height } });

// Create the effect.
const effect = new VirtualBackgroundEffect({
  authToken: '<encoded-string>',
  mode: `BLUR`,
  blurStrength: `STRONG`,
  quality: `LOW`,
});

// Load the effect with the original stream.
const newStream = await effect.load(originalVideoStream);

// Attach the new stream to a video element to see the effect in action.
myVideoElement.srcObject = newStream;

Noise Reduction

The noise reduction effect removes background noise from an audio stream to provide clear audio for calling.

The noise-reduction-effect takes a NoiseReductionEffectOptions config object in its constructor. A developer can optionally pass a workletProcessorUrl parameter (or legacyProcessorUrl) in the config to use a different of test version of the audio processor. An audioContext parameter can be passed into the config as well in order to supply an existing AudioContext; otherwise, a new one will be created.

The effect loads the background thread AudioWorkletProcessor into the main thread AudioWorklet in order to keep the audio computations from impacting UI performance.

Options

There are a few different options that can be supplied to the constructor or updateOptions() method that affect the behavior

Name	Description	Values	Required
authToken	Used to authenticate the request for the backend processors	An encoded string token	Yes
audioContext	An optional AudioContext for custom behavior	AudioContext	No
mode	Determines whether to run in WORKLET mode or LEGACY mode for older browsers	WORKLET LEGACY	Defaults to WORKLET
legacyProcessorUrl	A url to fetch the legacy processor that attaches to the deprecated ScriptProcessorNode	A fully qualified URL	No
workletProcessorUrl	A url to fetch the AudioWorkletProcessor to attach to the AudioWorkletNode	A fully qualified URL	No
env	Which environment the effect is running in.	EffectEnv.Production EffectEnv.Integration	No

Usage

Supply an audio stream to the effect and when loaded, it will return a new stream with the effect applied.

// Create a new audio stream by getting a user's audio media.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

// Create the effect.
const effect = new NoiseReductionEffect({
  authToken: '<encoded-string>',
  workletProcessorUrl: 'https://my-worklet-processor-url', // For 'WORKLET' mode
  legacyProcessorUrl: 'https://my-legacy-processor-url', // For 'LEGACY' mode
  mode: 'WORKLET', // or 'LEGACY'
});

// Load the effect with the original stream.
const newStream = await effect.load(stream);

Example

The example app included in this repo is designed to help test functionality and troubleshoot issues. You can run the example app by following the instructions in the README in the example folder. You can also view a live example here.

Development

1. Run yarn to install dependencies.
2. Run yarn prepare to prepare dependencies.
3. Run yarn watch to build and watch for updates.
4. Run yarn test to build, run tests, lint, and run test coverage.

Visual Studio Code

Install the recommended extensions when first opening the workspace (there should be a prompt). These plugins will help maintain high code quality and consistency across the project.

NOTE: VS Code is setup to apply formatting and linting rules on save (Prettier runs first, then ESLint). The rules applied are defined in settings.json.