ndx-pose is a standardized format for storing pose estimation data in NWB, such as from DeepLabCut and SLEAP. Please post an issue or PR to suggest or add support for another pose estimation tool.
This extension consists of several new neurodata types:
Skeleton which stores the relationship between the body parts (nodes and edges).Skeletons which is a container that stores multiple Skeleton objects.PoseEstimationSeries which stores the estimated positions (x, y) or (x, y, z) of a body part over time as well as
the confidence/likelihood of the estimated positions.PoseEstimation which stores the estimated position data (PoseEstimationSeries) for multiple body parts,
computed from the same video(s) with the same tool/algorithm.SkeletonInstance which stores the estimated positions and visibility of the body parts for a single frame.TrainingFrame which stores the ground truth data for a single frame. It contains SkeletonInstance objects and
references a frame of a source video (ImageSeries). The source videos can be stored internally as data arrays or
externally as files referenced by relative file path.TrainingFrames which is a container that stores multiple TrainingFrame objects.SourceVideos which is a container that stores multiple ImageSeries objects representing source videos used in training.PoseTraining which is a container thatstores the ground truth data (TrainingFrames) and source videos (SourceVideos)
used to train the pose estimation model.It is recommended to place the Skeletons, PoseEstimation, and PoseTraining objects in an NWB processing module
named "behavior", as shown below.
pip install ndx-pose
NWB files are designed to store data from a single subject and have only one root-level Subject object.
As a result, ndx-pose was designed to store pose estimates from a single subject.
Pose estimates data from different subjects should be stored in separate NWB files.
Training images can involve multiple skeletons, however. These training images may be the same across subjects, and therefore the same across NWB files. These training images should be duplicated between files, until multi-subject support is added to NWB and ndx-pose. See https://github.com/rly/ndx-pose/pull/3
Utilities to convert DLC output to/from NWB: https://github.com/DeepLabCut/DLC2NWB
PoseEstimation object
under /processing/behavior. That PoseEstimation object contains PoseEstimationSeries objects, one for each
body part, and general metadata about the pose estimation process, skeleton, and videos. The
PoseEstimationSeries objects contain the estimated positions for that body part for a particular animal.Utilities to convert SLEAP pose tracking data to/from NWB: https://github.com/talmolab/sleap-io
Keypoint MoSeq: https://github.com/dattalab/keypoint-moseq
PoseEstimation objects from NWB files.dlc2nwb described above),
SLEAP (using sleap_io described above), FicTrac, and LightningPose to NWB. It supports appending pose estimation data to an existing NWB file.Ethome: Tools for machine learning of animal behavior: https://github.com/benlansdell/ethome
PoseEstimation objects from NWB files.Related work:
Several NWB datasets use ndx-pose 0.1.1:
Several open-source conversion scripts on GitHub also use ndx-pose.
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#ffffff', "primaryBorderColor': '#144E73', 'lineColor': '#D96F32'}}}%%
classDiagram
direction LR
namespace ndx-pose {
class PoseEstimationSeries{
<<SpatialSeries>>
name : str
description : str
timestamps : array[float; dims [frame]]
data : array[float; dims [frame, [x, y]] or [frame, [x, y, z]]]
confidence : array[float; dims [frame]]
reference_frame: str
}
class PoseEstimation {
<<NWBDataInterface>>
name : str
description : str, optional
original_videos : array[str; dims [file]], optional
labeled_videos : array[str; dims [file]], optional
dimensions : array[uint, dims [file, [width, height]]], optional
scorer : str, optional
scorer_software : str, optional
scorer_software__version : str, optional
PoseEstimationSeries
Skeleton, link
Device, link
}
class Skeleton {
<<NWBDataInterface>>
name : str
nodes : array[str; dims [body part]]
edges : array[uint; dims [edge, [node, node]]]
}
}
class Device
PoseEstimation --o PoseEstimationSeries : contains 0 or more
PoseEstimation --> Skeleton : links to
PoseEstimation --> Device : links to
%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#ffffff', "primaryBorderColor': '#144E73', 'lineColor': '#D96F32'}}}%%
classDiagram
direction LR
namespace ndx-pose {
class PoseEstimationSeries{
<<SpatialSeries>>
name : str
description : str
timestamps : array[float; dims [frame]]
data : array[float; dims [frame, [x, y]] or [frame, [x, y, z]]]
confidence : array[float; dims [frame]]
reference_frame: str
}
class PoseEstimation {
<<NWBDataInterface>>
name : str
description : str, optional
original_videos : array[str; dims [file]], optional
labeled_videos : array[str; dims [file]], optional
dimensions : array[uint, dims [file, [width, height]]], optional
scorer : str, optional
scorer_software : str, optional
scorer_software__version : str, optional
PoseEstimationSeries
Skeleton, link
Device, link
}
class Skeleton {
<<NWBDataInterface>>
name : str
nodes : array[str; dims [body part]]
edges : array[uint; dims [edge, [node, node]]]
}
class TrainingFrame {
<<NWBDataInterface>>
name : str
annotator : str, optional
source_video_frame_index : uint, optional
skeleton_instances : SkeletonInstances
source_video : ImageSeries, link, optional
source_frame : Image, link, optional
}
class SkeletonInstance {
<<NWBDataInterface>>
id: uint, optional
node_locations : array[float; dims [body part, [x, y]] or [body part, [x, y, z]]]
node_visibility : array[bool; dims [body part]], optional
Skeleton, link
}
class TrainingFrames {
<<NWBDataInterface>>
TrainingFrame
}
class SkeletonInstances {
<<NWBDataInterface>>
SkeletonInstance
}
class SourceVideos {
<<NWBDataInterface>>
ImageSeries
}
class Skeletons {
<<NWBDataInterface>>
Skeleton
}
class PoseTraining {
<<NWBDataInterface>>>
training_frames : TrainingFrames, optional
source_videos : SourceVideos, optional
}
}
class Device
class ImageSeries
class Image
PoseEstimation --o PoseEstimationSeries : contains 0 or more
PoseEstimation --> Skeleton : links to
PoseEstimation --> Device : links to
PoseTraining --o TrainingFrames : contains
PoseTraining --o SourceVideos : contains
TrainingFrames --o TrainingFrame : contains 0 or more
TrainingFrame --o SkeletonInstances : contains
TrainingFrame --> ImageSeries : links to
TrainingFrame --> Image : links to
SkeletonInstances --o SkeletonInstance : contains 0 or more
SkeletonInstance --o Skeleton : links to
SourceVideos --o ImageSeries : contains 0 or more
Skeletons --o Skeleton : contains 0 or more
This extension was created using ndx-template.
FAQs
NWB extension to store pose estimation data
We found that ndx-pose demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.