psyseg-js-sdk

PsySeg-JS-SDK

The Personify Segmentation JavaScript Software Development Kit (PsySeg JS SDK) is a library that utilizes our state-of-the-art machine learning models and algorithms to perform user segmentation and apply special effects on the background.

About this repo

This repository contains the logic and supported API for extracting user and applying virtual background.
Please install this npm package first via NodeJS as "npm i psyseg-js-sdk"

APIs:

• export function ColorSpaceType()
• export function PsySegExtraParams (erode = 1)
• export function PsySegSetupInfo (width, height)
• export function PsySegBuf (width, height, channels, data)
• export async function psy_seg_create (pSetupInfo, reload = true)
• export async function psy_seg_get_alpha (pPsySeg, pInColor, colorSpace, pOutAlpha)
• export async function psy_seg_overlay_background (pPsySeg, pInColor, pInBackground, colorSpace, pOutColor, pPsySegExtraParams = null)
• export async function psy_seg_overlay_background_new (pPsySeg, pInColor, pInBackground, colorSpace, pOutColor, pPsySegExtraParams = null)
• export async function psy_seg_blur_background (pPsySeg, pInColor, colorSpace, pOutColor, blurSize = 9, pPsySegExtraParams = null)
• export async function psy_seg_remove_background (pPsySeg, pInColor, colorSpace, pOutColor, pPsySegExtraParams = null)
• export async function psy_seg_destroy (pPsySeg)

Backends/Platforms:

• CPU Backend: pure-JS backend for browsers
• TensorflowJS: WebGL backend for browsers supported by TFJS
• TensorflowLite: WebAssembly backend for the browsers (best performance & recommendation)

Detailed APIs description

1. ColorSpaceType()

a. usage: define input's colorspace for converting to right format of SDK colorspace (works with RGBA only)
b. notice: currently, this feature is not supported. Input's colorspace is forced to RGBA format

2. PsySegExtraParams (...)

a. usage: advanced configuration for customer about processing parameters
b. notice: currently, it has only "erode" parameter
c. return: PsySegExtraParams object

3. PsySegSetupInfo (...)

a. usage: define input / output size for user segmentation process. Example: user provide input frame (640 x 480), SDK's output will have same size (640 x 480). This size is fixed at initial setup and don't change until stopping process.
b. return: PsySegSetupInfo object
c. parameter

• witdh: width of processed frame in pixels (type: Number)
• height: height of processed frame in pixels (type: Number)

4. PsySegBuf (...)

a. usage: buffer data for processing. This SDK only work with buffer data type.
b. return: PsySeg buffer data object
c. parameter

• witdh: width of processed frame in pixels (type: Number)
• height: height of processed frame in pixels (type: Number)
• channels: number of channels of input (ex: colorspace of input is RGBA => it has 4 channels) (type: Number)
• data: image data (type: ImageData)

5. psy_seg_create (...)

a. usage: create PsySeg instance for managing buffer data and executing segmentation process.
b. notice: DNN model should be only loaded at first SDK initialization (init page). When process is restarted, it is not neccessary to reloaded.
c. return: PsySeg internal instance
d. parameter

• pSetupInfo: setup information (type: PsySegSetupInfo)
• reload: requesting for loading DNN model (true = load / false = not load) (type: Boolean)

6. psy_seg_get_alpha (...)

a. usage: extracting alpha mask that seperating persona and background
b. notice: alpha mask data is pushed directly into pOutAlpha buffer
c. return: true on success, false otherwise
d. parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutAlpha: output buffer data (type: PsySegBuf)

7. psy_seg_overlay_background (...)

a. usage: extracting alpha mask that seperating persona and background
b. notice: virtual background output data is pushed directly into pOutColor buffer
c. return: true on success, false otherwise
d. parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• pInBackground: background buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutColor: output buffer data (type: PsySegBuf)
• pPsySegExtraParams: advanced configuration (type: PsySegExtraParams)

8. psy_seg_overlay_background_new (...)

a. usage: extracting alpha mask that seperating persona and background
b. notice: virtual background output data is pushed directly into pOutColor buffer. This function applies different approach with psy_seg_overlay_background(...)
c. return: true on success, false otherwise
d. parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• pInBackground: background buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutColor: output buffer data (type: PsySegBuf)
• pPsySegExtraParams: advanced configuration (type: PsySegExtraParams)

9. psy_seg_blur_background (...)

a. usage: extracting alpha mask that seperating persona and background
b. notice: blurred background output data is pushed directly into pOutColor buffer.
c. return: true on success, false otherwise
d. parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutColor: output buffer data (type: PsySegBuf)
• blurSize: blurred level for background (type: Number - must be odd value)
• pPsySegExtraParams: advanced configuration (type: PsySegExtraParams)

10. psy_seg_remove_background (...)

a. usage: extracting alpha mask that seperating persona and background
b. notice: persona data is pushed directly into pOutColor buffer.
c. return: true on success, false otherwise
d. parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutColor: output buffer data (type: PsySegBuf)
• pPsySegExtraParams: advanced configuration (type: PsySegExtraParams)

11. psy_seg_destroy (...)

a. usage: destroy all related tensors and data buffers and close the PsySeg instance
b. notice: psy_seg_create(...) and psy_seg_destroy(...) must be used as a couple at initializing and stopping for avoiding memory leak

Examples

Step 1: Adding header scripts (for wasm module) and import psyseg-js-sdk npm package

<!-- scripts are stored in our server -->
<script src='SDK/model/tflite.js' async></script>
<script src='SDK/wasm/psyseg.js' async></script>

import * as PsySeg from "psyseg-js-sdk"

Step 2: Create setup information and buffer data for input / output / background

<!-- Specification Data -->
segWidth = 320;
segHeight = 240;

<!-- Configure variables -->
<!-- Create setup information with size of input frame: width / height -->
pSetupInfo = PsySeg.PsySegSetupInfo(segWidth, segHeight);

<!-- Get background buffer if needed (only for overlay background function -->
pBackground = PsySeg.PsySegBuf(pSetupInfo.colorWidth, pSetupInfo.colorHeight, 4, null);
	
<!-- Input buffer including: width / height / channels / data -->
pInColor = PsySeg.PsySegBuf(pSetupInfo.colorWidth, pSetupInfo.colorHeight, 4, null);

<!-- Output buffer including: width / height / channels / data -->
pOutColor = PsySeg.PsySegBuf(pSetupInfo.colorWidth, pSetupInfo.colorHeight, 4, null);

<!-- Colorspace of input frame -->
colorSpace = PsySeg.ColorSpaceType().COLOR_SPACE_RGBA;

Step 3: Create PsySeg instance

<!-- Instance for managing package -->
<!-- First initialization, setting reload = true. Otherwise, setting reload = false -->
pPsySeg = await PsySeg.psy_seg_create(pSetupInfo, true);

Step 4: Loading background data and running overlay effects (example for overlay effects)

<!-- Add background data via createBackground(...) function -->
<!-- createBackground(...) is user-defined function that return PsySegBuf for background data -->
pBackground = await createBackground(document.getElementById("background").getAttribute("src"), segWidth, segHeight);

<!-- Get input frame from camera (webcame is user-defined camera object) -->
pInColor.data = webcam.inputFrame;

<!-- Overlay background -->
await PsySeg.psy_seg_overlay_background(pPsySeg, pInColor, pBackground, colorSpace, pOutColor)
.then(async (status) => {

	<!-- status = true means that processing is successful -->
	if (status) {

		<!-- drawCanvas(...) is user-defined function for drawing canvas -->
		await drawCanvas(pOutColor, "result");

	} else {
		console.log("Cannot get overlay background");
	}
})
.catch(e => console.log("Cannot get overlay background due to " + e));

Contact

For getting more information, please contact PersonifyInc