handjoints-osc

handjoints-osc

Real-time hand tracking that sends joint coordinates over OSC. Written in Python, using Google MediaPipe.

Installation

Installing via pipx is recommended, because it installs in an isolated environment.

pipx install handjoints-osc

It's of course also possible to install via pip:

pip install handjoints-osc

Usage

$ handjoints-osc --help

usage: handjoints-osc [-h] [--host HOST] [--confidence CONFIDENCE] port

positional arguments: port send OSC to this port

options: -h, --help show this help message and exit --host HOST send OSC to this host (default: localhost) --confidence CONFIDENCE, -c CONFIDENCE minimum detection confidence threshold (default: 0.5) --device DEVICE, -d DEVICE video device index or path (default: 0, i.e. the default video device) --invert, -i invert colors

For example, to start the program and send joints coordinates to SuperCollider, which typically listens for OSC on port 57120:

handjoints-osc 57120

When hands are detected, SuperCollider will start receiving OSC messages with path "/handjoints-osc". To know which value corresponds to which joint, pressing "n" will display joint numbers on the tracking window.

Example using an alternative video device:

handjoints-osc -d /dev/video2 57120

Keybindings

• press n to toggle joint numbers
• press i to invert colors (light theme)

OSC format

• path: /handjoints i *i* ...f
• [0] number of detected hands
• [1:numHands] handedness for each detected hand
• [numHands+1:..] x and y coordinates for each joint for each hand

The program detects maximum 2 hands, each hand has 21 joints, and each joint 2 coordinates.

Arguments are all in a single list, starting with the number of hands, then handedness for each hand, and following with x and y coordinates for all joints of one hand, and then the joint of each other hand.

[nHands, ...handedness, ...coordsHand0, ...cordsHand1] coords: [j0x, j0y, j1x, j1y, j2x, j2y, ...]

If only one hand is detected, numHands + handedness + coords (21 * 2) gives 44 values. If two hands are detected, there are two handedness values, so 1 + 2 + 42 + 42 = 87 values.

Development

Recommended: make a virtual environment

python -m venv .venv source .venv/bin/activate

Install requirements:

pip install -r requirements