handsfree
Advanced tools
Quickly integrate face, hand, and/or pose tracking to your frontend projects in a snap ✨👌
Quickly integrate face, hand, and/or pose tracking to your frontend projects in a snap ✨👌
Powered by:
Sorry for the confusion! I'll likely be settling on Notion but am still trying to find the best docs. Thanks!
This repo is broken into 3 main parts: The actual library itself found in /src/, the documentation for it in /docs/, and the Handsfree Browser Extension in /extension/.
Note: models loaded from the CDN may load slower on the initial page load, but should load much faster once cached by the browser.
This option is great if you don't have or need a server, or if you're prototyping on a site like CodePen. You can also just download this repo and work with one of the /boilerplate/.
<head>
<!-- Include Handsfree.js -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@8.5.1/build/lib/assets/handsfree.css" />
<script src="https://unpkg.com/handsfree@8.5.1/build/lib/handsfree.js"></script>
</head>
<body>
<!-- Your code must be inside body as it applies classes to it -->
<script>
// Let's use handtracking and show the webcam feed with wireframes
const handsfree = new Handsfree({showDebug: true, hands: true})
handsfree.start()
// Create a plugin named "logger" to show data on every frame
handsfree.use('logger', data => {
console.log(data.hands)
})
</script>
</body>
# From your projects root
npm i handsfree
// Inside your app
import Handsfree from 'handsfree'
// Let's use handtracking and enable the plugins tagged with "browser"
const handsfree = new Handsfree({showDebug: true, hands: true})
handsfree.enablePlugins('browser')
handsfree.start()
The above will load models, some over 10Mb, from the Unpkg CDN. If you'd rather host these yourself (for example, to use offline) then you can eject the models from the npm package into your project's public folder:
# Move the models into your project's public directory
# - change PUBLIC below to where you keep your project's assets
# ON WINDOWS
xcopy /e node_modules\handsfree\build\lib PUBLIC
# EVERYWHERE ELSE
cp -r node_modules/handsfree/build/lib/* PUBLIC
import Handsfree from 'handsfree'
const handsfree = new Handsfree({
hands: true,
// Set this to your where you moved the models into
assetsPath: '/PUBLIC/assets',
})
handsfree.enablePlugins('browser')
handsfree.start()
The following aims to give you a quick overview of how things work. The key takeaway is that everything is centered around hooks/plugins, which are basically named callbacks which are run on every frame and can be toggled on and off.
The following workflow demonstrates how to use all features of Handsfree.js. Check out the Guides and References to dive deeper, and feel free to post on the Google Groups or Discord if you get stuck!
// Let's enable face tracking with the default Face Pointer
const handsfree = new Handsfree({weboji: true})
handsfree.enablePlugins('browser')
// Now let's start things up
handsfree.start()
// Let's create a plugin called "logger"
// - Plugins run on every frame and is how you "plug in" to the main loop
// - "this" context is the plugin itself. In this case, handsfree.plugin.logger
handsfree.use('logger', data => {
console.log(data.weboji.morphs, data.weboji.rotation, data.weboji.pointer, data, this)
})
// Let's switch to hand tracking now. To demonstrate that you can do this live,
// let's create a plugin that switches to hand tracking when both eyebrows go up
handsfree.use('handTrackingSwitcher', {weboji} => {
if (weboji.state.browsUp) {
// Disable this plugin
// Same as handsfree.plugin.handTrackingSwitcher.disable()
this.disable()
// Turn off face tracking and enable hand tracking
handsfree.update({
weboji: false,
hands: true
})
}
})
// You can enable and disable any combination of models and plugins
handsfree.update({
// Disable weboji which is currently running
weboji: false,
// Start the pose model
pose: true,
// This is also how you configure (or pre-configure) a bunch of plugins at once
plugin: {
fingerPointer: {enabled: false},
faceScroll: {
vertScroll: {
scrollSpeed: 0.01
}
}
}
})
// Disable all plugins
handsfree.disablePlugins()
// Enable only the plugins for making music (not actually implemented yet)
handsfree.enablePlugins('music')
// Overwrite our logger to display the original model APIs
handsfree.plugin.logger.onFrame = (data) => {
console.log(handsfree.model.pose?.api, handsfree.model.weboji?.api, handsfree.model.pose?.api)
}
|
Face Pointers
|
Motion Parallax Display
|
|
Puppeteering Industrial Robots
|
Playing desktop games with face clicks
|
|
Hand Pointers
|
Use with Three.js
|
|
Playing desktop games with pinch clicks
|
Laser pointers but with your finger
|
|
Flappy Pose - Flappy Bird but where you have to flap your arms
|
If you'd like to contribute to the library or documentation then the following will get you going:
git clone https://github.com/handsfreejs/handsfreenpm i in a terminal from the project's rootlocalhost:8080 by running npm startOnce you've run the above, you can just use npm start. If you pull the latest code, remember to run npm i to get any new dependencies (this shouldn't happen often).
# Start local development on localhost:8080
npm start
# Builds the library, documentation, and extension
npm run build
# Build only the library /dist/lib/
npm run build:lib
# Build only the documentation at /dist/docs/
npm run build:docs
# Build only the extension at /dist/extension
npm run build:extension
# Publish library to NPM
npm login
npm publish
# Deploy documentation to handsfree.js.org
deploy.sh
npm run fa:build when adding new font icons so that they are copied over into the docs/.vuepress/components/FA folder/docs, this is true when changing /docs/.vuepress.config.js as wellThe Browser Extension is a designed to help you browse the web handsfree through face and/or hand gestures. The goal is to develop a "Userscript Manager" like Tampermonkey, but for handsfree-ifying web pages, games, apps, WebXR and really any other type of content found the web.
/src/background/handsfree.js checks if you've approved the webcam. If not, then it'll open the options page in src/options/stream-capture.html/src/popup/index.html/src/background/handsfree.jssrcObject to a separate video element which is the PiP'edInstall as an unpacked chrome extension.
chrome://extensions
By default, each page will get a "Face Pointer" or a set of "Palm Pointers" for you to browse pages handsfree.
However, in addition to the pointers you can add custom handsfree interactions. For example, here's a demonstration of handsfree-ifying different things:
Or try it right away with the serverless boilerplates in /boilerplate/!
The Handsfree.js core is available for free and commercial use under Apache 2.0. Each of the models are also available for free and commercial use under Apache 2.0:
I'd like to also thank the following people and projects:
FAQs
Quickly integrate face, hand, and/or pose tracking to your frontend projects in a snap ✨👌
The npm package handsfree receives a total of 558 weekly downloads. As such, handsfree popularity was classified as not popular.
We found that handsfree demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Multiple high-impact npm maintainers confirm they have been targeted in the same social engineering campaign that compromised Axios.
Security News
Axios compromise traced to social engineering, showing how attacks on maintainers can bypass controls and expose the broader software supply chain.
Security News
Node.js has paused its bug bounty program after funding ended, removing payouts for vulnerability reports but keeping its security process unchanged.