![publish](https://github.com/webex/web-media-effects/actions/workflows/publish.yml/badge.svg)
![example-deploy](https://github.com/webex/web-media-effects/actions/workflows/example-deploy.yml/badge.svg)
web-media-effects
Web Media Effects is a suite of media effects developed for web SDKs and WebRTC media applications.
Introduction
There are three effects included in this library:
- Virtual background (e.g., blur, image replacement, video replacement)
- Noise reduction (e.g., background noise removal)
- Gain (for testing)
Common Methods
The effects are built on top of a plugin interface that makes building and extending effects more straight-forward.
The effects plugins have four primary methods to control the plugin:
load(input)
accepts a track or stream and returns a new track or stream with the effect appliedenable()
enables the plugin after it's loadeddisable()
disables the plugin after it's loadeddispose()
tears down the effect
Upon enabling or disabling the effect, an event is fired.
effect.on('track-updated', (track: MediaStreamTrack) => {
});
Additionally, there are a few convenience methods:
getOutputStream()
returns the new outgoing (i.e., "effected") streamgetOutputTrack()
returns the active output tracksetEnabled(boolean)
sets the effect state by passing in a boolean (convenient for state managers)
Virtual Background
The virtual background effect is a wrapper around ladon-ts that provides a virtual background for video calling. The virtual background may be an image, an mp4 video, or the user's background with blur applied. The blur option allows for varied levels of strength and quality where higher levels require more compute resources.
The virtual-background-effect
takes an optional VirtualBackgroundEffectOptions
config object in its constructor. The effect's options can be changed at runtime via an updateOptions()
method. When disabled, the effect simply passes through the original video images so that the outgoing stream does not need to be changed.
The effect uses a background thread worker by default to prevent slowdowns on the main UI thread. The main UI thread can be used instead by adding the property generator: 'local'
in the VirtualBackgroundEffectOptions
object. However, this is not recommended as the worker thread performs much better.
NOTE: For backwards compatibility, the default mode
is set to BLUR
.
Options
There are a few different options that can be supplied to the constructor or updateOptions()
method that affect the behavior
Name | Description | Values | Required |
---|
authToken | Used to authenticate the request for the backend models | An encoded string token | Yes |
generator | Determines where the model runs (on main thread or background thread) | local worker | Defaults to worker |
frameRate | Determines how many frames per second are sent to the model | 0-60 | Defaults to 30 |
quality | Determines the accuracy of the model (higher requires more CPU) | LOW MEDIUM HIGH ULTRA | Defaults to LOW |
mirror | Whether the output image should be flipped horizontally | true false | Defaults to false |
mode | Determines what kind of background to render behind the user | BLUR IMAGE VIDEO | Defaults to BLUR |
blurStrength | How strongly the background should be blurred | WEAK MODERATE STRONG STRONGER STRONGEST | Required in BLUR mode |
bgImageUrl | Path to the background image to replace the original background | Fully qualified URL | Required in IMAGE mode |
bgVideoUrl | Path to the background video to replace the original background | Fully qualified URL (mp4 only) | Required in VIDEO mode |
env | Which environment the effect is running in. | EffectEnv.Production EffectEnv.Integration | Defaults to EffectEnv.Production |
avoidSimd | Avoid using the SIMD processor, if SIMD is supported (for testing) | true , false | Defaults to false |
Mode
The virtual background plugin applies a background effect to the original media stream by performing image segmentation on the incoming video frames. The plugin is capable of applying three different kinds of effects called modes: background blur, background image replacement, and background video replacement.
The mode
configuration option is determines what background effect to apply. There are three accepted values for the mode
: BLUR
, IMAGE
, and VIDEO
. Each mode has at least one required option that needs to be set in the options object, which is outlined below in the Options section.
NOTE: For Typescript users, the mode can be selected by using the exported VirtualBackgroundMode
enum, for convenience.
Usage
Supply a video stream to the effect and when loaded, it will return a new stream with the effect applied.
const originalVideoStream = await navigator.mediaDevices.getUserMedia({ video: { width, height } });
const effect = new VirtualBackgroundEffect({
authToken: '<encoded-string>',
mode: `BLUR`,
blurStrength: `STRONG`,
quality: `LOW`,
});
const newStream = await effect.load(originalVideoStream);
myVideoElement.srcObject = myStream;
Noise Reduction
The noise reduction effect removes background noise from an audio stream to provide clear audio for calling.
The noise-reduction-effect
takes a NoiseReductionEffectOptions
config object in its constructor. A developer can optionally pass a workletProcessorUrl
parameter (or legacyProcessorUrl
) in the config to use a different of test version of the audio processor. An audioContext
parameter can be passed into the config as well in order to supply an existing AudioContext
; otherwise, a new one will be created.
The effect loads the background thread AudioWorkletProcessor into the main thread AudioWorklet in order to keep the audio computations from impacting UI performance.
Options
There are a few different options that can be supplied to the constructor or updateOptions()
method that affect the behavior
Name | Description | Values | Required |
---|
authToken | Used to authenticate the request for the backend processors | An encoded string token | Yes |
audioContext | An optional AudioContext for custom behavior | AudioContext | No |
mode | Determines whether to run in WORKLET mode or LEGACY mode for older browsers | WORKLET LEGACY | Defaults to WORKLET |
legacyProcessorUrl | A url to fetch the legacy processor that attaches to the deprecated ScriptProcessorNode | A fully qualified URL | No |
workletProcessorUrl | A url to fetch the AudioWorkletProcessor to attach to the AudioWorkletNode | A fully qualified URL | No |
env | Which environment the effect is running in. | EffectEnv.Production EffectEnv.Integration | No |
Usage
Supply an audio track or stream to the effect, the effect will handle updating the stream on enable/disable. In the case of a track being passed, listen to the 'track-updated'
event to receive the updated track on enable/disable.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const effect = new NoiseReductionEffect({
authToken: '<encoded-string>',
workletProcessorUrl: 'https://my-worklet-processor-url',
legacyProcessorUrl: 'https://my-legacy-processor-url',
mode: 'WORKLET',
});
await effect.load(stream);
Example
The example app included in this repo is designed to help test functionality and troubleshoot issues. You can run the example app by following the instructions in the README in the example folder. You can also view a live example at https://effects.webex.com.
Development
- Run
yarn
to install dependencies. - Run
yarn prepare
to prepare dependencies. - Run
yarn watch
to build and watch for updates. - Run
yarn test
to build, run tests, lint, and run test coverage.
Visual Studio Code
Install the recommended extensions when first opening the workspace (there should be a prompt). These plugins will help maintain high code quality and consistency across the project.
NOTE: VS Code is setup to apply formatting and linting rules on save (Prettier runs first, then ESLint). The rules applied are defined in settings.json.