psy-js-sdk

PsySeg-JS-SDK

The Personify Segmentation JavaScript Software Development Kit (PsySeg JS SDK) is a library that utilizes our state-of-the-art machine learning models and algorithms to perform user segmentation and apply special effects on the background.

About this repo

This repository contains the logic and supported API for extracting user and applying virtual background.
Please install this npm package first via NodeJS as "npm i psyseg-js-sdk"

###APIs:

• export function ColorSpaceType()
• export function PsySegExtraParams (erode = 1)
• export function PsySegSetupInfo (width, height)
• export function PsySegBuf (width, height, channels, data)
• export async function psy_seg_create (pSetupInfo, reload = true)
• export async function psy_seg_get_alpha (pPsySeg, pInColor, colorSpace, pOutAlpha)
• export async function psy_seg_overlay_background (pPsySeg, pInColor, pInBackground, colorSpace, pOutColor, pPsySegExtraParams = null)
• export async function psy_seg_overlay_background_new (pPsySeg, pInColor, pInBackground, colorSpace, pOutColor, pPsySegExtraParams = null)
• export async function psy_seg_blur_background (pPsySeg, pInColor, colorSpace, pOutColor, blurSize = 9, pPsySegExtraParams = null)
• export async function psy_seg_remove_background (pPsySeg, pInColor, colorSpace, pOutColor, pPsySegExtraParams = null)
• export async function psy_seg_destroy (pPsySeg)

###Backends/Platforms:

• CPU Backend: pure-JS backend for browsers
• TensorflowJS: WebGL backend for browsers supported by TFJS
• TensorflowLite: WebAssembly backend for the browsers (best performance & recommendation)

Detailed APIs description

ColorSpaceType()

usage: define input's colorspace for converting to right format of SDK colorspace (works with RGBA only) notice: currently, this feature is not supported. Input's colorspace is forced to RGBA format

PsySegExtraParams (erode = 1)

usage: advanced configuration for customer about processing parameters notice: currently, it has only "erode" parameter return: PsySegExtraParams object

PsySegSetupInfo (width, height)

usage: define input / output size for user segmentation process. Example: user provide input frame (640 x 480), SDK's output will have same size (640 x 480). This size is fixed at initial setup and don't change until stopping process. parameter

• witdh: width of processed frame in pixels (type: Number)
• height: height of processed frame in pixels (type: Number) return: PsySegSetupInfo object

PsySegBuf (width, height, channels, data)

usage: buffer data for processing. This SDK only work with buffer data type. parameter

• witdh: width of processed frame in pixels (type: Number)
• height: height of processed frame in pixels (type: Number)
• channels: number of channels of input (ex: colorspace of input is RGBA => it has 4 channels) (type: Number)
• data: image data (type: ImageData) return: PsySeg buffer data object

psy_seg_create (pSetupInfo, reload = true)

usage: create PsySeg instance for managing buffer data and executing segmentation process. parameter

• pSetupInfo: setup information (type: PsySegSetupInfo)
• reload: requesting for loading DNN model (true = load / false = not load) (type: Boolean) return: PsySeg internal instance notice: DNN model should be only loaded at first SDK initialization (init page). When process is restarted, it is not neccessary to reloaded

psy_seg_get_alpha (pPsySeg, pInColor, colorSpace, pOutAlpha)

usage: extracting alpha mask that seperating persona and background parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutAlpha: output buffer data (type: PsySegBuf) return: true on success, false otherwise notice: alpha mask data is pushed directly into pOutAlpha buffer

psy_seg_overlay_background (pPsySeg, pInColor, pInBackground, colorSpace, pOutColor, pPsySegExtraParams = null)

usage: extracting alpha mask that seperating persona and background parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• pInBackground: background buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutColor: output buffer data (type: PsySegBuf)
• pPsySegExtraParams: advanced configuration (type: PsySegExtraParams) return: true on success, false otherwise notice: virtual background output data is pushed directly into pOutColor buffer

psy_seg_overlay_background_new (pPsySeg, pInColor, pInBackground, colorSpace, pOutColor, pPsySegExtraParams = null)

usage: extracting alpha mask that seperating persona and background parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• pInBackground: background buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutColor: output buffer data (type: PsySegBuf)
• pPsySegExtraParams: advanced configuration (type: PsySegExtraParams) return: true on success, false otherwise notice: virtual background output data is pushed directly into pOutColor buffer. This function applies different approach with psy_seg_overlay_background(...)

psy_seg_blur_background (pPsySeg, pInColor, colorSpace, pOutColor, blurSize = 9, pPsySegExtraParams = null)

usage: extracting alpha mask that seperating persona and background parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutColor: output buffer data (type: PsySegBuf)
• blurSize: blurred level for background (type: Number - must be odd value)
• pPsySegExtraParams: advanced configuration (type: PsySegExtraParams) return: true on success, false otherwise notice: blurred background output data is pushed directly into pOutColor buffer.

psy_seg_remove_background (pPsySeg, pInColor, colorSpace, pOutColor, pPsySegExtraParams = null)

usage: extracting alpha mask that seperating persona and background parameter

• pPsySeg: PsySeg instance that manages attributes and execution methods (type: PsySeg)
• pInColor: input buffer data (type: PsySegBuf)
• colorSpace: input colorspace (type: ColorSpaceType)
• pOutColor: output buffer data (type: PsySegBuf)
• pPsySegExtraParams: advanced configuration (type: PsySegExtraParams) return: true on success, false otherwise notice: persona data is pushed directly into pOutColor buffer.

psy_seg_destroy (pPsySeg)

usage: destroy all related tensors and data buffers and close the PsySeg instance notice: psy_seg_create(...) and psy_seg_destroy(...) must be used as a couple at initializing and stopping for avoiding memory leak

Examples

Step 1: Adding header scripts (for wasm module) and import psyseg-js-sdk npm package

<!-- scripts are stored in our server -->
<script src='SDK/model/tflite.js' async></script>
<script src='SDK/wasm/psyseg.js' async></script>

import * as PsySeg from "psyseg-js-sdk"

Step 2: Create setup information and buffer data for input / output / background

<!-- Specification Data -->
segWidth = 320;
segHeight = 240;

<!-- Configure variables -->
<!-- Create setup information with size of input frame: width / height -->
pSetupInfo = PsySeg.PsySegSetupInfo(segWidth, segHeight);

<!-- Get background buffer if needed (only for overlay background function -->
pBackground = PsySeg.PsySegBuf(pSetupInfo.colorWidth, pSetupInfo.colorHeight, 4, null);
	
<!-- Input buffer including: width / height / channels / data -->
pInColor = PsySeg.PsySegBuf(pSetupInfo.colorWidth, pSetupInfo.colorHeight, 4, null);

<!-- Output buffer including: width / height / channels / data -->
pOutColor = PsySeg.PsySegBuf(pSetupInfo.colorWidth, pSetupInfo.colorHeight, 4, null);

<!-- Colorspace of input frame -->
this.colorSpace = PsySeg.ColorSpaceType().COLOR_SPACE_RGBA;

Step 3: Create PsySeg instance

<!-- Instance for managing package -->
<!-- First initialization, setting reload = true. Otherwise, setting reload = false -->
pPsySeg = await PsySeg.psy_seg_create(pSetupInfo, true);

Step 4: Loading background data and running overlay effects (example for overlay effects)

<!-- Add background data via createBackground(...) function -->
<!-- createBackground(...) is user-defined function that return PsySegBuf for background data -->
pBackground = await createBackground(document.getElementById("background").getAttribute("src"), segWidth, segHeight);

<!-- Get input frame from camera (webcame is user-defined camera object) -->
pInColor.data = webcam.inputFrame;

<!-- Overlay background -->
await PsySeg.psy_seg_overlay_background(pPsySeg, pInColor, pBackground, colorSpace, pOutColor)
.then(async (status) => {
	<!-- status = true means that processing is successful -->
	if (status) {
		<!-- drawCanvas(...) is user-defined function for drawing canvas -->
		await drawCanvas(pOutColor, "result");
	} else {
		console.log("Cannot get overlay background");
	}
})
.catch(e => console.log("Cannot get overlay background due to " + e));

Contact

For getting more information, please contact PersonifyInc