@zappar/zappar-babylonjs
Advanced tools
This library allows you use Zappar's best-in-class AR technology with content built using the 3D rendering platform Babylon.js.
It provides high performance (30 frames-per-second) face, image and world tracking, in the browsers already installed on your users' mobile phones.
You may also be interested in:
You can get started super-quickly using one of our bootstrap projects. They contain the basics of an AR experience for the different tracking types - no more, no less.
The JavaScript repositories, contain index.html files to get you started. Alternatively, TypeScript repositories, contain webpack setups optimized for development and deployment.
| Tracking Type | JavaScript | TypeScript |
|---|---|---|
| Image Tracking | GitHub url | GitHub url |
| Face Tracking | GitHub url | GitHub url |
| Instant Tracking | GitHub url | GitHub url |
There's a repository of example projects for your delectation over here
You can use this library by downloading a standalone zip containing the necessary files, by linking to our CDN, or by installing from NPM for use in a webpack project.
Unzip into your web project and reference from your HTML like this:
<script src="zappar-babylon.js"></script>
Reference the zappar.js library from your HTML like this:
<script src="https://libs.zappar.com/zappar-babylon/2.1.9/zappar-babylon.js"></script>
Run the following NPM command inside your project directory:
npm install --save @zappar/zappar-babylonjs
Then import the library into your JavaScript or TypeScript files:
import * as ZapparBabylon from "@zappar/zappar-babylonjs";
Please note - This library supports Webpack 5 and later.
You can integrate the Zappar library with the existing runRenderLoop loop of your Babylon.js project. A typical project may look like this. The remainder of this document goes into more detail about each of the component elements of this example.
import * as BABYLON from 'babylonjs';
// Import path to the target file ( Made using @zappar/zapworks-cli )
import targetfile from './assets/example-tracking-image.zpt';
import * as ZapparBabylon from '@zappar/zappar-babylonjs';
// Setup Babylon.js in the usual way.
const canvasHolder = document.querySelector('#canvas-holder') || document.createElement('div');
const canvas = document.createElement('canvas');
canvasHolder.appendChild(canvas);
const engine = new BABYLON.Engine(canvas, true, { preserveDrawingBuffer: true, stencil: true });
const scene = new BABYLON.Scene(engine);
const light = new BABYLON.HemisphericLight('light1', new BABYLON.Vector3(0, 1, 0), scene);
// Create a Zappar AR camera
const camera = new ZapparBabylon.Camera('camera', scene);
// Request camera permissions and start the camera
ZapparBabylon.permissionRequestUI().then((granted) => {
if (granted) camera.start();
else ZapparBabylon.permissionDeniedUI();
});
// Set up a tracker, in this case an image tracker
const imageTracker = new ZapparBabylon.ImageTrackerLoader().load(targetfile);
const trackerTransformNode = new ZapparBabylon.ImageAnchorTransformNode('tracker', camera, imageTracker, scene);
// Child any 3D content you'd like tracked from the image into the trackerTransformNode
const box = BABYLON.Mesh.CreateBox('box', 1, scene, false);
box.parent = trackerTransformNode;
engine.runRenderLoop(() => {
camera.updateFrame();
scene.render();
});
Due to browser restrictions surrounding use of the camera, you must use HTTPS to view or preview your site, even if doing so locally from your computer. If you're using webpack, consider using webpack-dev-server which has an https option to enable this.
Alternatively you can use the ZapWorks command-line tool to serve a folder over HTTPS for access on your local computer, like this:
zapworks serve .
The command also lets you serve the folder for access by other devices on your local network, like this:
zapworks serve . --lan
This library works well on the browsers that enjoy the vast majority of mobile market-share. That said, there are a number of web browsers available across the mobile and desktop device landscape.
Best support:
Functional but not our primary support target (support quality):
Known to not work:
* Browsers without motion sensor access (e.g desktop browsers) don't support instant world tracking or attitude-based camera poses.
To make it easy to detect if your page is running in a browser that's not supported, we've provided the browserIncompatible() and browserIncompatibleUI() functions:
if (ZapparBabylon.browserIncompatible()) {
ZapparBabylon.browserIncompatibleUI();
throw new Error("Unsupported browser");
}
The browserIncompatibleUI() function shows a full-page dialog that informs the user they're using an unsupported browser, and provides a button to 'copy' the current page URL so they can 'paste' it into the address bar of a compatible alternative.
Once you've built your site, you have a number of options for hosting your site:
Head over to the ZapWorks Publishing and Hosting article to learn more about these options.
This wrapper library is MIT licensed, but relies on our proprietary computer vision library, @zappar/zappar-cv, for which you must maintain an activate subscription at ZapWorks. To learn more about licensing, click here.
The source code for this wrapper library is available freely for your viewing pleasure over at GitHub: https://github.com/zappar-xr/zappar-babylonjs/
The library provides a camera object that you can use instead of Babylon's Camera. It's constructed like this:
const camera = new ZapparBabylon.Camera('camera', scene);
Options may be passed into the ZapparBabylon.Camera constructor.
const camera = new ZapparBabylon.Camera(
"camera",
scene,
{
rearCameraSource:'csO9c0YpAf274OuCPUA53CNE0YHlIr2yXCi+SqfBZZ8=',
userCameraSource: 'RKxXByjnabbADGQNNZqLVLdmXlS0YkETYCIbg+XxnvM='
}
);
const camera = new ZapparBabylon.Camera(
"camera",
scene,
{ // These values are defaults.
zNear: 0.1,
zFar: 100
}
);
Call the following function once an animation frame (e.g. during your engine.runRenderLoop function) in order to process incoming camera frames:
camera.updateFrame();
Alternatively, you may allow the camera to update itself (no longer requiring the step above) by constructing it as so:
const camera = new ZapparBabylon.Camera('camera', scene, true); // Self updating camera
The SDK provides an automatically generated environment map that's useful if you're using materials that support reflections (e.g. PBRMetallicRoughnessMaterial, StandardMaterial). The map uses the camera feed to create an approximate environment that can add some realism to your scene.
To use the map, first construct an instance:
const envMap = new ZapparBabylon.CameraEnvironmentMap(camera, engine);
Attach it to specific materials:
material.environmentTexture = envMap.environmentMap;
Finally, call update() on the map each frame, between updating the camera and rendering the scene:
engine.runRenderLoop(() => {
envMap.update();
scene.render();
});
The library needs to ask the user for permission to access the camera and motion sensors on the device.
To do this, you can use the following function to show a built-in UI informing the user of the need and providing a button to trigger the browser's permission prompts. The function returns a promise that lets you know if the user granted the permissions or not.
// Show Zappar's built-in UI to request camera permissions
ZapparBabylon.permissionRequestUI().then(granted => {
if (granted) {
// User granted the permissions so start the camera
camera.start();
} else {
// User denied the permissions so show Zappar's built-in 'permission denied' UI
ZapparBabylon.permissionDeniedUI();
}
});
If you'd rather show your own permissions UI, you can use the following function to trigger the browser's permission prompts directly. The function returns a promise that resolves to true if the user granted all the necessary permissions, otherwise false. Please note - due to browser restrictions, this function must be called from within a user event, e.g. in the event handler of a button click.
ZapparBabylon.permissionRequest().then(granted => {
if (granted) {
// User granted the permissions so start the camera
} else {
// User denied the permissions
// You can show your own 'permission denied' UI here or use Zappar's built-in one
ZapparBabylon.permissionDeniedUI();
}
});
Once the user has granted the necessary permissions, you can start the rear-facing camera on the device with the following function on your camera object:
camera.start();
If you'd like to start the user-facing 'selfie' camera, pass true:
camera.start(true);
To switch between the front- and rear-facing cameras during your experience, just call camera.start(...) again passing either true or false as appropriate.
User-facing cameras are normally shown mirrored to users and so the camera object provides a number of modes to support this:
ZapparBabylon.CameraMirrorMode.Poses: this mode mirrors the background camera texture and ensures content still appears correctly tracked. In this mode your content itself isn't flipped, so any text in your tracked content doesn't appear mirrored. This is the default mode for the user-facing camera.ZapparBabylon.CameraMirrorMode.CSS: in this mode, the Zappar camera applies a scaleX(-1) CSS transform to your whole canvas. This way both the camera and your content appear mirrored.ZapparBabylon.CameraMirrorMode.None: no mirroring is performed. This is the default mode for the rear-facing camera.The userCameraMirrorMode and rearCameraMirrorMode parameters set which mode the camera object will use for each camera:
camera.userCameraMirrorMode = ZapparBabylon.CameraMirrorMode.CSS;
The Zappar library provides multiple modes for the camera to move around in the Babylon.js scene. You can set this mode with the poseMode parameter of your camera object. There are the following options:
ZapparBabylon.CameraPoseMode.Default: in this mode the camera stays at the origin of the scene, pointing down the negative Z axis. Any tracked anchors will move around in your scene as the user moves the physical camera and real-world tracked objects.ZapparBabylon.CameraPoseMode.Attitude: the camera stays at the origin of the scene, but rotates as the user rotates the physical device. When the Zappar library initializes, the negative Z axis of world space points forward in front of the user.ZapparBabylon.CameraPoseMode.AnchorOrigin: the origin of the scene is the center of the anchor specified by the camera's poseAnchorOrigin parameter. In this case the camera moves and rotates in world space around the anchor at the origin.The correct choice of camera pose will depend on your given use case and content. Here are some examples you might like to consider when choosing which is best for you:
ZapparBabylon.CameraPoseMode.Attitude and place a light shining down the negative Y axis is world space.ZapparBabylon.CameraPoseMode.AnchorOrigin would allow the blocks to rest on a tracked image regardless of how the image is held by the user, while using ZapparBabylon.CameraPoseMode.Attitude would allow the user to tip the blocks off the image by tilting it.The Zappar library offers three types of tracking for you to use to build augmented reality experiences:
To track content from a flat image in the camera view, create a new ImageTracker object:
const imageTracker = new ZapparBabylon.ImageTracker();
ImageTrackers use a special 'target file' that's been generated from the source image you'd like to track. You can generate them using the ZapWorks command-line utility like this:
zapworks train myImage.png
The resulting file can be loaded into an image tracker object by passing it to the loadTarget(...) function as either a URL or an ArrayBuffer. The function returns a promise that resolves when the target file has been loaded successfully.
const imageTracker = new ZapparBabylon.ImageTracker();
imageTracker.loadTarget("myImage.zpt").then(() => {
// Image target has been loaded
});
Alternatively the library provides a loader for loading a tracker and target file:
const imageTracker = new ZapparBabylon.ImageTrackerLoader().load("myImage.zpt");
Each ImageTracker exposes anchors for images detected and tracked in the camera view. At this time, ImageTrackers only track one image in view at a time.
Anchors have the following parameters:
id: a string that's unique for this anchorvisible: a boolean indicating if this anchor is visible in the current camera frameonVisible and onNotVisible: event handlers that emit when the anchor becomes visible, or disappears in the camera view. These events are emitted during your call to camera.updateFrame().You can access the anchors of a tracker using its anchors parameter - it's a JavaScript Map keyed with the IDs of the anchors. Trackers will reuse existing non-visible anchors for new images that appear and thus, until ImageTracker supports tracking more than one image at a time, there is never more than one anchor managed by each ImageTracker. Each tracker also exposes a JavaScript Set of anchors visible in the current camera frame as its visible parameter.
To attach 3D content (e.g. Babylon.js objects or models) to an ImageTracker or an ImageAnchor, the library provides ImageAnchorTransformNode. It's a Babylon.js TransformNode that will follow the supplied anchor (or, in the case of a supplied ImageTracker, the anchor most recently visible in that tracker) in the 3D view:
const imageAnchorTransformNode = new ZapparBabylon.ImageAnchorTransformNode('tracker', camera, imageTracker, scene);
// Add in any 3D objects you'd like to track to this image
myModel.parent = imageAnchorTransformNode;
The TransformNode provides a coordinate system that has its origin at the center of the image, with positive X axis to the right, the positive Y axis towards the top and the positive Z axis coming up out of the plane of the image. The scale of the coordinate system is such that a Y value of +1 corresponds to the top of the image, and a Y value of -1 corresponds to the bottom of the image. The X axis positions of the left and right edges of the target image therefore depend on the aspect ratio of the image.
In addition to using the anchors and visible parameters, ImageTrackers expose event handlers that you can use to be notified of changes in the anchors or their visibility. The events are emitted during your call to camera.updateFrame().
onNewAnchor - emitted when a new anchor is created by the trackeronVisible - emitted when an anchor becomes visible in a camera frameonNotVisible - emitted when an anchor goes from being visible in the previous camera frame, to being not visible in the current frameHere's an example of using these events:
imageTracker.onNewAnchor.bind(anchor => {
console.log("New anchor has appeared:", anchor.id);
// You may like to create a new ImageAnchorTransformNode here for this anchor, and add it to your scene
});
imageTracker.onVisible.bind(anchor => {
console.log("Anchor is visible:", anchor.id);
});
imageTracker.onNotVisible.bind(anchor => {
console.log("Anchor is not visible:", anchor.id);
});
To place content on or around a user's face, create a new FaceTracker object when your page loads:
const faceTracker = new ZapparBabylon.FaceTracker();
The face tracking algorithm requires a model file of data in order to operate - you can call loadDefaultModel() to load the one that's included by default with the library. The function returns a promise that resolves when the model has been loaded successfully.
const faceTracker = new ZapparBabylon.FaceTracker();
faceTracker.loadDefaultModel().then(() => {
// The model has been loaded successfully
});
Alternatively the library provides a loader for loading a tracker and model file:
const faceTracker = new ZapparBabylon.FaceTrackerLoader().load();
Each FaceTracker exposes anchors for faces detected and tracked in the camera view. By default a maximum of one face is tracked at a time, however you can change this using the maxFaces parameter:
faceTracker.maxFaces = 2;
Note that setting a value of two or more may impact the performance and framerate of the library, so we recommend sticking with one unless your use case requires tracking multiple faces.
Anchors have the following parameters:
id: a string that's unique for this anchorvisible: a boolean indicating if this anchor is visible in the current camera frameidentity and expression: Float32Arrays containing data used for rendering a face-fitting mesh (see below)onVisible and onNotVisible: event handlers that emit when the anchor becomes visible, or disappears in the camera view. These events are emitted during your call to camera.updateFrame()You can access the anchors of a tracker using its anchors parameter - it's a JavaScript Map keyed with the IDs of the anchors. Trackers will reuse existing non-visible anchors for new faces that appear and thus there are never more than maxFaces anchors handled by a given tracker. Each tracker also exposes a JavaScript Set of anchors visible in the current camera frame as its visible parameter.
To attach 3D content (e.g. Babylon.js objects or models) to a FaceTracker or a FaceAnchor, the library provides FaceAnchorTransformNode. It's a Babylon.js TransformNode that will follow the supplied anchor (or, in the case of a supplied FaceTracker, the anchor most recently visible in that tracker) in the 3D view:
const faceAnchorTransformNode = new ZapparBabylon.FaceAnchorTransformNode('tracker', camera, faceTracker, scene);
// Child any 3D objects you'd like to track to this face
myModel.parent = faceAnchorTransformNode;
The TransformNode provides a coordinate system that has its origin at the center of the head, with positive X axis to the right, the positive Y axis towards the top and the positive Z axis coming forward out of the user's head.
Note that users typically expect to see a mirrored view of any user-facing camera feed. Please see the section on mirroring the camera view earlier in this document.
In addition to using the anchors and visible parameters, FaceTrackers expose event handlers that you can use to be notified of changes in the anchors or their visibility. The events are emitted during your call to camera.updateFrame(renderer).
onNewAnchor - emitted when a new anchor is created by the trackeronVisible - emitted when an anchor becomes visible in a camera frameonNotVisible - emitted when an anchor goes from being visible in the previous camera frame, to being not visible in the current frameHere's an example of using these events:
faceTracker.onNewAnchor.bind(anchor => {
console.log("New anchor has appeared:", anchor.id);
// You may like to create a new FaceAnchorTransformNode here for this anchor, and add it to your scene
});
faceTracker.onVisible.bind(anchor => {
console.log("Anchor is visible:", anchor.id);
});
faceTracker.onNotVisible.bind(anchor => {
console.log("Anchor is not visible:", anchor.id);
});
In addition to tracking the center of the head, you can use FaceLandmarkTransformNode to track content from various points on the user's face. These landmarks will remain accurate, even as the user's expression changes.
To track a landmark, construct a new FaceLandmarkTransformNode object, passing your camera, face tracker, and the name of the landmark you'd like to track:
const faceLandmarkTransformNode = new ZapparBabylon.FaceLandmarkTransformNode('tracker', camera, faceTracker, ZapparBabylon.FaceLandmarkName.CHIN, scene);
// Child any 3D objects you'd like to track to this face
myModel.parent = faceLandmarkTransformNode;
The following landmarks are available: EYE_LEFT, EYE_RIGHT, EAR_LEFT, EAR_RIGHT, NOSE_BRIDGE, NOSE_TIP, NOSE_BASE, LIP_TOP, LIP_BOTTOM, MOUTH_CENTER, CHIN, EYEBROW_LEFT, and EYEBROW_RIGHT. Note that 'left' and 'right' here are from the user's perspective.
In addition to tracking the center of the face using FaceTracker, the Zappar library provides a number of meshes that will fit to the face/head and deform as the user's expression changes. These can be used to apply a texture to the user's skin, much like face paint, or to mask out the back of 3D models so the user's head is not occluded where it shouldn't be.
To use the face mesh, first construct a new FaceMesh object and load its data file. The loadDefaultFace function returns a promise that resolves when the data file has been loaded successfully. You may wish to use to show a 'loading' screen to the user while this is taking place.
const faceMesh = new ZapparBabylon.FaceMesh();
faceMesh.loadDefaultFace().then(() => {
// Face mesh loaded
});
Alternatively the library provides a loader for loading face mesh and data file:
const faceMesh = new ZapparBabylon.FaceMeshLoader().loadFace();
While the faceMesh object lets you access the raw vertex, UV, normal and indices data for the face mesh, you may wish to use the library's FaceMeshGeometry object which wraps the data as a Babylon.js Mesh. This Mesh object must still be childed to a FaceAnchorTransformNode to appear in the correct place on-screen:
const faceTracker = new ZapparBabylon.FaceTrackerLoader().load();
const trackerTransformNode = new ZapparBabylon.FaceTrackerTransformNode('tracker', camera, faceTracker, scene);
const material = new BABYLON.StandardMaterial('mat', scene);
material.diffuseTexture = new BABYLON.Texture(faceMeshTexture, scene);
const faceMesh = new ZapparBabylon.FaceMeshGeometry('face mesh', scene);
faceMesh.parent = trackerTransformNode;
faceMesh.material = material;
targetPlane.parent = trackerTransformNode;
Each frame, after camera.updateFrame(), call one of the following functions to update the face mesh to the most recent identity and expression output from a face anchor:
// Update directly from a FaceAnchorTransformNode
faceMesh.updateFromFaceAnchorTransformNode(faceAnchorTransformNode);
// Update from a face anchor
faceMesh.updateFromFaceAnchor(myFaceAnchor);
At this time there are two meshes included with the library. The default mesh covers the user's face, from the chin at the bottom to the forehead, and from the sideburns on each side. There are optional parameters that determine if the mouth and eyes are filled or not:
loadDefaultFace(fillMouth?: boolean, fillEyeLeft?: boolean, fillEyeRight?: boolean)
The full head simplified mesh covers the whole of the user's head, including some neck. It's ideal for drawing into the depth buffer in order to mask out the back of 3D models placed on the user's head (see Head Masking below). There are optional parameters that determine if the mouth, eyes and neck are filled or not:
loadDefaultFullHeadSimplified(fillMouth?: boolean, fillEyeLeft?: boolean, fillEyeRight?: boolean, fillNeck?: boolean)
If you're placing a 3D model around the user's head, such as a helmet, it's important to make sure the camera view of the user's real face is not hidden by the back of the model. To achieve this, the library provides ZapparBabylon.HeadMaskMesh. It's a BABYLON.Mesh that fits the user's head and fills the depth buffer, ensuring that the camera image shows instead of any 3D elements behind it in the scene.
To use it, construct the object using a ZapparBabylon.HeadMaskMeshLoader and add it to your face anchor TransformNode:
const mask = new ZapparBabylon.HeadMaskMeshLoader('mask mesh').load();
faceAnchorTransformNode.add(mask);
Then, in each frame after camera.updateFrame(), call one of the following functions to update the head mesh to the most recent identity and expression output from a face anchor:
// Update directly from a FaceAnchorTransformNode
mask.updateFromFaceAnchorTransformNode(faceAnchorTransformNode);
// Update from a face anchor
mask.updateFromFaceAnchor(myFaceAnchor);
Behind the scenes the HeadMaskMesh works using a full-head ZapparBabylon.FaceMesh with the mouth, eyes and neck filled in. Its material.disableColorWrite is set to true so it fills the depth buffer but not the color buffer.
To track content from a point on a surface in front of the user, create a new InstantWorldTracker:
const instantWorldTracker = new ZapparBabylon.InstantWorldTracker();
Each InstantWorldTracker exposes a single anchor from its anchor parameter.
To choose the point in the user's environment that the anchor tracks from, use the setAnchorPoseFromCameraOffset(...) function, like this:
instantWorldTracker.setAnchorPoseFromCameraOffset(0, 0, -5);
The parameters passed in to this function correspond to the X, Y and Z coordinates (in camera space) of the point to track. Choosing a position with X and Y coordinates of zero, and a negative Z coordinate, will select a point on a surface directly in front of the center of the screen.
To attach 3D content (e.g. Babylon.js objects or models) to an InstantWorldTracker, the library provides InstantWorldAnchorTransformNode. It's a Babylon.js TransformNode that will follow the anchor in the supplied InstantWorldTracker in the 3D view:
const trackerTransformNode = new ZapparBabylon.InstantWorldAnchorTransformNode('tracker', camera, instantWorldTracker, scene);
// Child in any 3D objects you'd like to track to this point in space
myModel.parent = trackerTransformNode;
The TransformNode provides a coordinate system that has its origin at the point that's been set, with the positive Y coordinate pointing up out of the surface, and the X and Z coordinates in the plane of the surface. How far the chosen point is from the camera (i.e. how negative the Z coordinate provided to setAnchorPoseFromCameraOffset is) determines the scale of the coordinate system exposed by the anchor.
A typical application will call setAnchorPoseFromCameraOffset each frame until the user confirms their choice of placement by tapping a button, like this:
// Not shown - initialization, camera setup & permissions
const instantWorldTracker = new ZapparBabylon.InstantWorldTracker();
const instantWorldAnchorTransformNode = new ZapparBabylon.InstantWorldAnchorTransformNode('tracker', camera, instantWorldTracker, scene);
// Not shown - add content to the instantWorldAnchorTransformNode
let hasPlaced = false;
myConfirmButton.addEventListener("click", () => { hasPlaced = true });
engine.runRenderLoop(() => {
camera.updateFrame();
if (!hasPlaced) {
instantWorldTracker.setAnchorPoseFromCameraOffset(0, 0, -5);
}
scene.render();
});
FAQs
BabylonJS wrappers for Zappar's computer vision library
The npm package @zappar/zappar-babylonjs receives a total of 114 weekly downloads. As such, @zappar/zappar-babylonjs popularity was classified as not popular.
We found that @zappar/zappar-babylonjs demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 0 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.