@personifyinc/psyseg-sdk

PsySeg SDK for JavaScript

The Personify Segmentation JavaScript Software Development Kit (PsySeg JS SDK) is a library that utilizes our state-of-the-art machine learning models and algorithms to perform user segmentation and apply special effects on the background.

Prerequisites

This project requires NodeJS and NPM. Node and NPM are really easy to install. To make sure you have them available on your machine, try running the following command.

$ npm -v && node -v
6.4.1
v8.16.0

Getting Started

Installation

To install and set up the latest sdk, run:

$ npm install @personifyinc/psyseg-sdk

To install and set up specific sdk version, run:

$ npm install @personifyinc/psyseg-sdk@x.x.x

Usage

To use the SDK interfaces, simply import @personifyinc/psyseg-sdk as you normally would.

import PsySeg, { IPsySegResponses, IPsySegEffectParams, IPsySegProps } from '@personifyinc/psyseg-sdk';

To use the SDK class, simply import @personifyinc/psyseg-sdk as you normally would.

import PsySeg from '@personifyinc/psyseg-sdk';

const psySegInstance = new PsySeg()

Interfaces

IPsySegResponses

export interface IPsySegResponses {
    status: boolean
    licenseControl?: string
    data?: ImageData
    modelSize?: {
        width: number
        height: number
    }
}

IPsySegEffectParams

export interface IPsySegEffectParams {
    mode?: string
    erode?: number
    enhanceFrame?: boolean
    gamma?: number
    isSkipAlpha?: boolean
    blur?: number
    dstSize?: {
        width: number,
        height: number
    }
    overlaySize?: number
    overlayPosition?: number
    shape?: string
    isRemoveBlob?: boolean
}

IPsySegProps

export interface IPsySegProps {
    psySegWasmURL?: string
    tfliteWasmURL?: string
    tfliteSimdWasmURL?: string
    segmentationModelURL?: string
    license?: string
    accessToken?: string
    accessKey?: string
}

APIs

• version
• init
• replaceBackground
• blurBackground
• removeBackground
• runAlpha
• toMask
• shapeForeground
• maskShapeBackground

version

Get the current version of the installed SDK.

const version = psySegInstance.version()
//=> v3.0.0

init

It is required because it initializes the wasm file and validates the license.

const psySegProps: IPsySegProps = {
    psySegWasmURL: '',
    tfliteWasmURL: '',
    tfliteSimdWasmURL: '',
    segmentationModelURL: '',
    license: '',
    accessToken: '',
    accessKey: ''
}
await psySegInstance.init(psySegProps)

Attribute	Type	Required	Default	Description
psySegWasmURL	string	no	empty	Where to host the wasm file, if this attribute empty it should be hosted from our CDN https://psyjs-cdn.personifyinc.com/core/wasm/psyseg-x.x.x.wasm
tfliteWasmURL	string	no	empty	Where to host the wasm file, if this attribute empty it should be hosted from our CDN https://psyjs-cdn.personifyinc.com/core/tflite/wasm/tflite-20221011.wasm
tfliteSimdWasmURL	string	no	empty	Where to host the wasm file, if this attribute empty it should be hosted from our CDN https://psyjs-cdn.personifyinc.com/core/tflite/wasm/tflite-simd-20221011.wasm
segmentationModelURL	string	no	empty	Where to host the model tflite file, if this attribute empty it should be hosted from our CDN https://psyjs-cdn.personifyinc.com/core/tflite/model/psysegmentation-112x192-20230419_e36.asset
license	string	no	empty	For getting more information, please contact PersonifyInc
accessToken	string	no	empty	For getting more information, please contact PersonifyInc
accessKey	string	no	empty	For getting more information, please contact PersonifyInc

replaceBackground

Extract user from background and replace it with desired ones.

const inputColor: ImageData = inputImageData
const inputBackground: ImageData = backgroundImageData
const effectParams: IPsySegEffectParams = {
    mode: 'full',
    erode: 1,
    enhanceFrame: false,
    gamma: 0,
    isSkipAlpha: false,
    overlaySize: 1,
    overlayPosition: 0.5,
    isRemoveBlob: true,
}

const result: IPsySegResponses = await psySegInstance.replaceBackground(
    inputColor,
    inputBackground,
    effectParams
)

Attribute	Type	Required	Default	Description
inputColor	ImageData	yes		The ImageData of Video Frame or Image Element
inputBackground	ImageData	yes		The ImageData of Image Element
effectParams	IPsySegEffectParams	no	{}	The advanced configuration
effectParams.mode	string	no	full	Setting mode for PsySeg to reach the best performance or quality based on different purposes of customer.By default, full mode is selected. This option provides the best quality, but it has a tradeoff for performance
effectParams.erode	number	no	1	Erode level for edge smoothing
effectParams.enhanceFrame	boolean	no	false	Enable lighting feature
effectParams.gamma	number	no	0	Gamma value for lighting feature
effectParams.isSkipAlpha	boolean	no	false	Enable skip alpha feature
effectParams.overlaySize	number	no	1	Overlay size of user extraction, 0 < overlaySize <= 1, step is 0.1
effectParams.overlayPosition	number	no	0.5	Overlay position from left to right for user extraction, 0 < overlayPosition <= 1, step is 0.1
effectParams.isRemoveBlob	boolean	no	true	Remove the blobs surrounding the user

blurBackground

Extracting user from background and making a background blurred to hidden unnecessary object.

const inputColor: ImageData = inputImageData
const effectParams: IPsySegEffectParams = {
    mode: 'full',
    erode: 1,
    enhanceFrame: false,
    gamma: 0,
    isSkipAlpha: false,
    blur: 10,
}

const result: IPsySegResponses = await psySegInstance.blurBackground(
    inputColor,
    effectParams
)

Attribute	Type	Required	Default	Description
inputColor	ImageData	yes		The ImageData of Video Frame or Image Element
effectParams	IPsySegEffectParams	no	{}	The advanced configuration
effectParams.mode	string	no	full	Setting mode for PsySeg to reach the best performance or quality based on different purposes of customer.By default, full mode is selected. This option provides the best quality, but it has a tradeoff for performance
effectParams.erode	number	no	1	Erode level for edge smoothing
effectParams.enhanceFrame	boolean	no	false	Enable lighting feature
effectParams.gamma	number	no	0	Gamma value for lighting feature
effectParams.isSkipAlpha	boolean	no	false	Enable skip alpha feature
effectParams.blur	number	no	10	Blur level

removeBackground

Extract only user from background.

const inputColor: ImageData = inputImageData
const effectParams: IPsySegEffectParams = {
    mode: 'full',
    erode: 1,
    enhanceFrame: false,
    gamma: 0,
    isSkipAlpha: false,
}

const result: IPsySegResponses = await psySegInstance.removeBackground(
    inputColor,
    effectParams
)

Attribute	Type	Required	Default	Description
inputColor	ImageData	yes		The ImageData of Video Frame or Image Element
effectParams	IPsySegEffectParams	no	{}	The advanced configuration
effectParams.mode	string	no	full	Setting mode for PsySeg to reach the best performance or quality based on different purposes of customer.By default, full mode is selected. This option provides the best quality, but it has a tradeoff for performance
effectParams.erode	number	no	1	Erode level for edge smoothing
effectParams.enhanceFrame	boolean	no	false	Enable lighting feature
effectParams.gamma	number	no	0	Gamma value for lighting feature
effectParams.isSkipAlpha	boolean	no	false	Enable skip alpha feature

runAlpha

Extract alpha mask that separate persona(transparent) and background(black).

const inputColor: ImageData = inputImageData
const effectParams: IPsySegEffectParams = {
    isSkipAlpha: false,
}

const result: IPsySegResponses = await psySegInstance.runAlpha(
    inputColor,
    effectParams
)

Attribute	Type	Required	Default	Description
inputColor	ImageData	yes		The ImageData of Video Frame or Image Element
effectParams	IPsySegEffectParams	no	{}	The advanced configuration
effectParams.isSkipAlpha	boolean	no	false	Enable skip alpha feature

toMask

Extract alpha mask that separate persona(white) and background(transparent).

const inputColor: ImageData = inputImageData
const effectParams: IPsySegEffectParams = {
    isSkipAlpha: false,
    dstSize: {
        width: 320,
        height: 240,
    },
    erode: 1,
}

const result: IPsySegResponses = await psySegInstance.toMask(
    inputColor,
    effectParams
)

Attribute	Type	Required	Default	Description
inputColor	ImageData	yes		The ImageData of Video Frame or Image Element
effectParams	IPsySegEffectParams	no	{}	The advanced configuration
effectParams.isSkipAlpha	boolean	no	false	Enable skip alpha feature
effectParams.dstSize	object	no	{ width: 160, height: 96 }	Destination size of output
effectParams.erode	number	no	1	Erode level for edge smoothing

shapeForeground

Ability to present user in a circle/square shape and merge into a background image

const inputColor: ImageData = inputImageData
const inputBackground: ImageData = backgroundImageData
const effectParams: IPsySegEffectParams = {
    shape: 'circle'
}

const result: IPsySegResponses = await psySegInstance.shapeForeground(
    inputColor,
    inputBackground,
    effectParams
)

Attribute	Type	Required	Default	Description
inputColor	ImageData	yes		The ImageData of Video Frame or Image Element
inputBackground	ImageData	yes		The ImageData of Image Element
effectParams	IPsySegEffectParams	no	{}	The advanced configuration
effectParams.shape	string	no	circle	Present user in a border shape, it should be "circle"/"square"

maskShapeBackground

Make a mask has circle/square shape into center of background image

const inputBackground: ImageData = backgroundImageData
const effectParams: IPsySegEffectParams = {
    shape: 'circle'
}

const result: IPsySegResponses = await psySegInstance.maskShapeBackground(
    inputBackground,
    effectParams
)

Attribute	Type	Required	Default	Description
inputBackground	ImageData	yes		The ImageData of Image Element
effectParams	IPsySegEffectParams	no	{}	The advanced configuration
effectParams.shape	string	no	circle	Make mask in a border shape, it should be "circle"/"square"

Examples

Create Image from URL

export const createImage = async (
    imgSrc: string,
    resolution: _IResolution
): Promise<HTMLImageElement> => {
    return new Promise(resolve => {
        const img = new Image(resolution.width, resolution.height)
        img.crossOrigin = 'Anonymous'
        img.onload = () => {
            resolve(img)
        }
        img.src = imgSrc
    })
}

Get ImageData from Video Frame or Image

export const getImageData = (
    sourceInput: HTMLVideoElement | HTMLImageElement | null,
    resolution: _IResolution
) => {
    const inputCanvas = document.createElement('canvas')
    inputCanvas.width = resolution.width
    inputCanvas.height = resolution.height
    const canvasContext = inputCanvas.getContext('2d')

    if (canvasContext && sourceInput) {
        canvasContext.drawImage(
            sourceInput,
            0,
            0,
            resolution.width,
            resolution.height
        )
        return canvasContext.getImageData(0, 0, resolution.width, resolution.height)
    }
}

Render the Image Data into the Canvas Output

export const drawCanvas = (
    outputBuf: ImageData | undefined,
    ctxOutput: CanvasRenderingContext2D,
    options: _IDrawnCanvasOptions
) => {
    if (outputBuf && ctxOutput) {
        const {canvasWidth, canvasHeight} = options

        ctxOutput.clearRect(0, 0, canvasWidth, canvasHeight)
        ctxOutput.putImageData(outputBuf, 0, 0)

    }
}

How to use PsySeg SDK

// Initialize PsySeg SDK instance
const psySegInstance = new PsySeg()
console.info('PsySeg Version: ', psySegInstance.version())
const psySegInstanceInit = await psySegInstance.init({
    license,
    accessToken,
    accessKey,
})
console.info('PsySeg Initiation: ', psySegInstanceInit)

// Get Background Image Data
const imageBackground = await createImage('https://psyjs-cdn.personifyinc.com/assets/psyseg2-js-demo-background.png', {
    width: 320,
    height: 240,
})
const inputBackground = getImageData(imageBackground, {
    width: 320,
    height: 240,
}) as ImageData

// Get Video Element & Canvas Element
const videoElement = document.getElementById('#video')
const canvasOutput = document.getElementById('#canvas')
canvasOutput.width = 320
canvasOutput.height = 240
const ctxOutput = canvasOutput.getContext(
    '2d'
) as CanvasRenderingContext2D

// Running function
const runEffect = async (): Promise<void> => {
    try {
        const inputColor = getImageData(
            videoElement as HTMLVideoElement,
            {width: 320, height: 240}
        ) as ImageData

        const output = await psySegInstance.replaceBackground(
            inputColor,
            inputBackground as ImageData,
            {
                mode: 'full',
                erode: 1,
                enhanceFrame: false,
                gamma: 0,
                isSkipAlpha: false,
                overlaySize: 1,
                overlayPosition: 0.5,
            }
        )

        if (output?.status) {
            drawCanvas(
                output.data,
                ctxOutput,
                {
                    canvasWidth: 320,
                    canvasHeight: 240,
                }
            )
        }

        // Timeout will affect to the FPS and CPU usage
        const timeout = 1000 / 30

        // Recursive call
        setTimeout(runEffect, timeout)
    } catch (e) {
        console.error(e)
    }
}

runEffect()

Notes: For Real-Time Messaging Protocol (RTMP), we should use insertable streams instead of setTimeout or requestAnimationFrame to keep web application working in inactive mode. Please take a look at https://developer.chrome.com/articles/mediastreamtrack-insertable-media-processing/ and https://mediastreamtrack.glitch.me/

Demo

Please take a look our demo

Contact

For getting more information, please contact PersonifyInc

Changelog

Release npm package @personifyinc/psyseg-sdk - v5.6.0

• Main Features:
  • Add a parameter to toggle the removal of the box blob
  • Ensure license validation prior to invoking APIs
  • Add an erosion value to the toMask api for the ability to adjust the edges of the body

Release npm package @personifyinc/psyseg-sdk - v5.5.1

• Bug Fixes:
  • Fix issue: module parse failed: Unexpected character '#' while integrating with react-script 4

Release npm package @personifyinc/psyseg-sdk - v5.5.0

• Main Features:
  • Present user in a circle/square shape and merge into a background image
  • Make a mask has circle/square shape into center of background image

Release npm package @personifyinc/psyseg-sdk - v5.4.0

• Main Features:
  • Reducing the SDK startup time
  • Update api replaceBackground to scale and move UE
  • Improve Psyseg model
  • Return the model size for api init()

Release npm package @personifyinc/psyseg-sdk - v5.3.0

• Main Features:
  • Re-provide the api runAlpha

Release npm package @personifyinc/psyseg-sdk - v5.2.0

• Main Features:
  • Apply license validation to the api toMask
  • Remove online license validation
  • Remove api runAlpha
  • Reduce CPU and memory usage

Release npm package @personifyinc/psyseg-sdk - v5.0.0

• Main Features:
  • Separate the tflite model from the wasm file to reduce capacity and easily develop other models with different sizes in the future
  • Update the tflite model to improve segmentation and reduce CPU & Memory usage
  • Remove unnecessary logs

Release npm package @personifyinc/psyseg-sdk - v4.0.0

• Main Features:
  • Implement API toMask to extract alpha mask that separate persona(white) and background(transparent).
  • Use reject statement instead of resolve statement for catching error
  • Update API version() to return value v4.0.0

Release npm package @personifyinc/psyseg-sdk - v3.0.0

• Main Features:
  • Restructure the entire SDK to make it simpler and easier for developers to integrate.
  • Add an init() method to wait for the SDK to initialize.
  • Remove some useless packages to reduce the size of SDK.
  • Remove some redundant code as well as some unnecessary processing to reduce CPU usage.
• Bug Fixes:
  • Fix issue: Bypass license control by blocking the api checkLicense