Perform a markerless kinematic analysis from multiple calibrated views as a unified workflow from an OpenPose input to an OpenSim result.
News: Version 0.10.0:
OpenSim scaling and inverse kinematics are now integrated in Pose2Sim! No static trial needed.
Other recently added features: Pose estimation, Automatic camera synchronization, Multi-person analysis, Blender visualization, Marker augmentation, Batch processing.
To upgrade, type
pip install pose2sim --upgrade
Pose2Sim provides a workflow for 3D markerless kinematics, as an alternative to traditional marker-based MoCap methods.
Pose2Sim is free and open-source, low-cost but with research-grade accuracy and production-grade robustness. It gives a maximum of control on clearly explained parameters. Any combination of phones, webcams, or GoPros can be used with fully clothed subjects, so it is particularly adapted to the sports field, the doctor's office, or for outdoor 3D animation capture.
Note: For real-time analysis with a single camera, please consider Sports2D (note that the motion must lie in the sagittal or frontal plane).
Fun fact:
Pose2Sim stands for "OpenPose to OpenSim", as it originally used OpenPose inputs (2D keypoints coordinates) and lead to an OpenSim result (full-body 3D joint angles). Pose estimation is now performed with more recent models from RTMPose, and custom models (from DeepLabCut for example) can also be used.
Pose2Sim releases:
N.B.: As always, I am more than happy to welcome contributors (see How to contribute).
Optional:
Install Anaconda or Miniconda for simplicity and avoiding the risk of incompatibilities between libraries.
Once installed, open an Anaconda prompt and create a virtual environment:
conda create -n Pose2Sim python=3.9 -y
conda activate Pose2Sim
Install OpenSim:
Install the OpenSim Python API (if you do not want to install via conda, refer to this page):
conda install -c opensim-org opensim -y
Install Pose2Sim:
If you don't use Anaconda, type python -V in terminal to make sure python>=3.9 is installed.
OPTION 1: Quick install: Open a terminal.
pip install pose2sim
OPTION 2: Build from source and test the last changes: Open a terminal in the directory of your choice and Clone the Pose2Sim repository.
git clone --depth 1 https://github.com/perfanalytics/pose2sim.git
cd pose2sim
pip install .
Optional:
For faster inference, you can run on the GPU. Install pyTorch with CUDA and cuDNN support, and ONNX Runtime with GPU support (not available on MacOS).
Be aware that GPU support takes an additional 6 GB on disk. The full installation is then 10.75 GB instead of 4.75 GB.
Run nvidia-smi in a terminal. If this results in an error, your GPU is probably not compatible with CUDA. If not, note the "CUDA version": it is the latest version your driver is compatible with (more information on this post).
Then go to the ONNXruntime requirement page, note the latest compatible CUDA and cuDNN requirements. Next, go to the pyTorch website and install the latest version that satisfies these requirements (beware that torch 2.4 ships with cuDNN 9, while torch 2.3 installs cuDNN 8). For example:
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
Finally, install ONNX Runtime with GPU support:
pip uninstall onnxruntime
pip install onnxruntime-gpu
Check that everything went well within Python with these commands:
import torch; import onnxruntime as ort
print(torch.cuda.is_available(), ort.get_available_providers())
# Should print "True ['CUDAExecutionProvider', ...]"
Note on storage use:
A full installation takes up to 11 GB of storage spate. However, GPU support is not mandatory and takes about 6 GB. Moreover, marker augmentation requires Tensorflow and does not necessarily yield better results. You can save an additional 1.3 GB by uninstalling it:pip uninstall tensorflow.
A minimal installation with carefully chosen pose models and without GPU support, Tensorflow, PyQt5 would take less than 3 GB.
This demonstration provides an example experiment of a person balancing on a beam, filmed with 4 cameras.
Open a terminal, enter pip show pose2sim, report package location.
Copy this path and go to the Single participant Demo folder: cd <path>\Pose2Sim\Demo_SinglePerson.
Type ipython, and try the following code:
from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
Pose2Sim.poseEstimation()
Pose2Sim.synchronization()
Pose2Sim.personAssociation()
Pose2Sim.triangulation()
Pose2Sim.filtering()
Pose2Sim.markerAugmentation()
Pose2Sim.kinematics()
3D marker locations are stored as .trc files in each trial folder in the pose-3d directory.
3D joint angles are stored as .mot files in the kinematics directory. Scaled models are also stored in the same directory.
Note:
from Pose2Sim import Pose2Sim
Pose2Sim.runAll()
# or: Pose2Sim.runAll(do_calibration=True, do_poseEstimation=True, do_synchronization=True, do_personAssociation=True, do_triangulation=True, do_filtering=True, do_markerAugmentation=True, do_kinematics=True)
calibration_type to calculate instead of convert in Config.toml (more info there).pip uninstall tensorflow.Visualize your results and look in detail for potential areas of improvement.
Install OpenSim GUI:
Download the executable there.
Visualize results:
kinematics folder.kinematics folder.pose-3d folder.
Install the add-on:
Follow instructions on the Pose2Sim_Blender add-on page.
Visualize results:
Just play with the buttons!
Visualize camera positions, videos, triangulated keypoints, OpenSim skeleton, video overlay your results on videos, ... or let your creativity flow and create your own animations!
https://github.com/davidpagnon/Pose2Sim_Blender/assets/54667644/a2cfb75d-a2d4-471a-b6f8-8f1ee999a619
N.B.: Full install only required to import the skeleton. See instructions there.
Another person, hidden all along, will appear when multi-person analysis is activated!
Go to the Multi-participant Demo folder: cd <path>\Pose2Sim\Demo_MultiPerson.
Type ipython, and try the following code:
from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
Pose2Sim.poseEstimation()
# Pose2Sim.synchronization()
Pose2Sim.personAssociation()
Pose2Sim.triangulation()
Pose2Sim.filtering()
Pose2Sim.markerAugmentation()
Pose2Sim.kinematics()
or equivalently:
from Pose2Sim import Pose2Sim
Pose2Sim.runAll(do_synchronization=False) # Synchronization possible, but tricky with multiple persons
One .trc file per participant will be generated and stored in the pose-3d directory.
Similarly, one scaled .osim model and one joint angle .mot file per participant will be stored in the kinematicsfolder.
You can visualize your results with Blender as explained in Demonstration Part-2.
N.B.:
project > multi_person = true for each trial that contains multiple persons.markerAugmentation > participant_height and participant_mass matches the person's IDs.Run numerous analysis with different parameters and minimal friction.
Go to the Batch Demo folder: cd <path>\Pose2Sim\Demo_Batch.
Type ipython, and try the following code:
from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
Pose2Sim.poseEstimation()
Pose2Sim.synchronization()
Pose2Sim.personAssociation()
Pose2Sim.triangulation()
Pose2Sim.filtering()
Pose2Sim.markerAugmentation()
Pose2Sim.kinematics()
or equivalently:
from Pose2Sim import Pose2Sim
Pose2Sim.runAll()
The batch processing structure requires a Config.toml file in each of the trial directories. Global parameters are given in the Config.toml file of the BatchSession folder. They can be altered for specific Trials by uncommenting keys and their values in their respective Config.toml files.
Run Pose2Sim from the BatchSession folder if you want to batch process the whole session, or from a Trial folder if you want to process only a specific trial.
| SingleTrial | BatchSession |
|---|---|
SingleTrial | BatchSession |
For example, try uncommenting [project] and set frame_range = [10,99], or uncomment [pose] and set mode = 'lightweight' in the Config.toml file of Trial_2.
N.B.: If a step is not relevant for your use case (synchronization, person association, marker augmentation...), you can just skip it.
Get yourself comfy!
pip show pose2sim, report package location. cd <path>\pose2sim.Calibration and videos folders, edit the Config.toml files, and run each Pose2Sim step.Estimate 2D pose from images with RTMPose or another pose estimation solution.
RTMPose is a state-of-the-art pose estimation solution that is faster and more accurate than OpenPose. It is now included in Pose2Sim for straightforward end-to-end analysis.
Open an Anaconda prompt or a terminal in a Session or Trial folder.
Type ipython.
from Pose2Sim import Pose2Sim
Pose2Sim.poseEstimation()
N.B.: Pose estimation can be run in lightweight, balanced, or performance mode.
N.B.: The pose_model with body, feet, hands, and face is required for wrist motion but is much slower and slightly less accurate on body keypoints.
N.B.: The GPU will be used with ONNX backend if a valid CUDA installation is found (or ROCM with AMD GPUS, or MPS with MacOS), otherwise the CPU will be used with OpenVINO backend.
N.B.: Pose estimation can be dramatically sped up by increasing the value of det_frequency. In that case, the detection is only done every det_frequency frames, and bounding boxes are tracked inbetween (keypoint detection is still performed on all frames).
Coming soon
If you need to detect specific points on a human being, an animal, or an object, you can also train your own model with DeepLabCut. In this case, Pose2Sim is used as an alternative to AniPose.
DLC_to_OpenPose.py script, see Utilities). Note that the names of your camera folders must follow the same order as in the calibration file, and end with '_json':
python -m DLC_to_OpenPose -i input_h5_file
pose.CUSTOM in Config.toml, and edit the node IDs so that they correspond to the column numbers of the 2D pose file, starting from zero. Make sure you also changed the pose_model and the tracked_keypoint. config_path = r'path_to_Config.toml'
import toml, anytree
config = toml.load(config_path)
pose_model = config.get('pose').get('pose_model')
model = anytree.importer.DictImporter().import_(config.get('pose').get(pose_model))
for pre, _, node in anytree.RenderTree(model):
print(f'{pre}{node.name} id={node.id}')
N.B.: RTMlib is faster, more accurate, and easier to install than OpenPose. This is a legacy option.
N.B.: OpenPose model files are apparently not available on their website anymore. Send me an email at dp2032@bath.ac.uk if you want me to forward them to you!
The accuracy and robustness of Pose2Sim have been thoroughly assessed only with OpenPose, BODY_25B model. Consequently, we recommend using this 2D pose estimation solution. See OpenPose repository for installation and running. Windows portable demo is enough.
videos folder:
bin\OpenPoseDemo.exe --model_pose BODY_25B --video <PATH_TO_TRIAL_DIR>\videos\cam01.mp4 --write_json <PATH_TO_TRIAL_DIR>\pose\pose_cam01_json
json_display_with_img.py or json_display_with_img.py scripts (see Utilities) if you want to display 2D pose detections.N.B.: RTMlib is faster, more accurate, and easier to install than BlazePose. This is also a legacy option.
Mediapipe BlazePose is very fast, fully runs under Python, handles upside-down postures and wrist movements (but no subtalar ankle angles).
However, it is less robust and accurate than OpenPose, and can only detect a single person.
Blazepose_runsave.py (see Utilities) to run BlazePose under Python, and store the detected coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format:
python -m Blazepose_runsave -i input_file -dJs
python -m Blazepose_runsave -h for explanation on parameters.pose_model and the tracked_keypoint in the Config.toml file.N.B.: RTMlib is faster, more accurate, and easier to install than AlphaPose. This is also a legacy option.
AlphaPose is one of the main competitors of OpenPose, and its accuracy is comparable. As a top-down approach (unlike OpenPose which is bottom-up), it is faster on single-person detection, but slower on multi-person detection.
All AlphaPose models are supported (HALPE_26, HALPE_68, HALPE_136, COCO_133, COCO, MPII). For COCO and MPII, AlphaPose must be run with the flag "--format cmu".
AlphaPose_to_OpenPose.py script, see Utilities):
python -m AlphaPose_to_OpenPose -i input_alphapose_json_file
pose_model and the tracked_keypoint in the Config.toml file.Calculate camera intrinsic properties and extrinsic locations and positions.
Convert a preexisting calibration file, or calculate intrinsic and extrinsic parameters from scratch.
Open an Anaconda prompt or a terminal in a Session or Trial folder.
Type ipython.
from Pose2Sim import Pose2Sim
Pose2Sim.calibration()
If you already have a calibration file, set calibration_type type to convert in your Config.toml file.
N.B.: If the original calibration file does not provide any residual errors, they will be logged as NaN. This is not an error and can be ignored.
.toml calibration file to the Pose2Sim Calibration folder..qca.txt within QTM.Calibration Pose2Sim folder.convert_from to 'qualisys' in your Config.toml file. Change binning_factor to 2 if you film in 540p.Calib.toml file, but TT_CameraOrientationMatrix first needs to be converted to a Rodrigues vector with OpenCV. See instructions here.Calib.toml file as is and do not run Pose2Sim.calibration().xcp Vicon calibration file to the Pose2Sim Calibration folder.convert_from to 'vicon' in your Config.toml file. No other setting is needed..pickle OpenCap calibration files to the Pose2Sim Calibration folder.convert_from to 'opencap' in your Config.toml file. No other setting is needed.intri.yml and extri.yml files to the Pose2Sim Calibration folder.convert_from to 'easymocap' in your Config.toml file. No other setting is needed.Calibration folder.convert_from to 'biocv' in your Config.toml file. No other setting is needed.Calculate calibration parameters with a checkerboard, with measurements on the scene, or automatically with detected keypoints.
Take heart, it is not that complicated once you get the hang of it!
N.B.: Try the calibration tool on the Demo by changing
calibration_typetocalculatein Config.toml.
For the sake of practicality, there are voluntarily few board images for intrinsic calibration, and few points to click for extrinsic calibration. In spite of this, your reprojection error should be under 1-2 cm, which does not hinder the quality of kinematic results in practice.).
Calculate intrinsic parameters with a checkerboard:
N.B.: Intrinsic parameters: camera properties (focal length, optical center, distortion), usually need to be calculated only once in their lifetime. In theory, cameras with same model and same settings will have identical intrinsic parameters.
N.B.: If you already calculated intrinsic parameters earlier, you can skip this step by settingoverwrite_intrinsicsto false.
Calibration\intrinsics folder.
Intrinsic calibration error should be below 0.5 px.
Calculate extrinsic parameters:
N.B.: Extrinsic parameters: camera placement in space (position and orientation), need to be calculated every time a camera is moved. Can be calculated from a board, or from points in the scene with known coordinates.
N.B.: If there is no measurable item in the scene, you can temporarily bring something in (a table, for example), perform calibration, and then remove it before you start capturing motion.
Calibration\extrinsics folder.object_coords_3d by these coordinates in Config.toml.
Extrinsic calibration error should be below 1 cm, but depending on your application, results will still be potentially acceptable up to 2.5 cm.
2D points can be triangulated only if they represent the same body position across all cameras: therefore, views need to be synchronized.
For each camera, the algorithm computes mean vertical speed for the chosen keypoints, and synchronizes by finding the time offset for which the correlation is highest.
N.B.: Skip this step if your cameras are natively synchronized.
Open an Anaconda prompt or a terminal in a Session or Trial folder.
Type ipython.
from Pose2Sim import Pose2Sim
Pose2Sim.synchronization()
In multi_person mode, a video will pop up to let the user choose on which person to synchronize.
If results are not satisfying, edit your Config.toml file:
All keypoints can be taken into account, or a subset of them.
The whole capture can be used for synchronization, or you can choose a time range when the participant is roughly horizontally static but with a clear vertical motion (set approx_time_maxspeed and time_range_around_maxspeed accordingly).
N.B.: Works best when:
N.B.: Alternatively, synchronize cameras using a flashlight, a clap, or a clear event. GoPro cameras can also be synchronized with a timecode, by GPS (outdoors), or with their app (slightly less reliable).
If
multi_personis set tofalse, the algorithm chooses the person for whom the reprojection error is smallest.
Ifmulti_personis set totrue, it associates across views the people for whom the distances between epipolar lines are the smallest. People are then associated across frames according to their displacement speed.
N.B.: Skip this step if only one person is in the field of view.
Open an Anaconda prompt or a terminal in a Session or Trial folder.
Type ipython.
from Pose2Sim import Pose2Sim
Pose2Sim.personAssociation()
Check printed output. If results are not satisfying, try and release the constraints in the Config.toml file.
Triangulate your 2D coordinates in a robust way.
The triangulation is weighted by the likelihood of each detected 2D keypoint, provided that they this likelihood is above a threshold.
If the reprojection error is above another threshold, right and left sides are swapped; if it is still above, cameras are removed until the threshold is met. If more cameras are removed than a predefined number, triangulation is skipped for this point and this frame. In the end, missing values are interpolated.
Open an Anaconda prompt or a terminal in a Session or Trial folder.
Type ipython.
from Pose2Sim import Pose2Sim
Pose2Sim.triangulation()
Check printed output, and visualize your trc in OpenSim: File -> Preview experimental data.
If your triangulation is not satisfying, try and release the constraints in the Config.toml file.
Filter your 3D coordinates.
Butterworth, Kalman, Butterworth on speed, Gaussian, LOESS, Median filters are available and can be tuned accordingly.
Open an Anaconda prompt or a terminal in a Session or Trial folder.
Type ipython.
from Pose2Sim import Pose2Sim
Pose2Sim.filtering()
Check your filtration with the displayed figures, and visualize your .trc file in OpenSim. If your filtering is not satisfying, try and change the parameters in the Config.toml file.
Output:
Use the Stanford LSTM model to estimate the position of 47 virtual markers.
Note that inverse kinematic results are not necessarily better after marker augmentation. Skip if results are not convincing.
N.B.: Marker augmentation tends to give a more stable, but less precise output. In practice, it is mostly beneficial when using less than 4 cameras.
Make sure that participant_height is correct in your Config.toml file. participant_mass is mostly optional for IK.
Only works with models estimating at least the following keypoints (e.g., not COCO):
["Neck", "RShoulder", "LShoulder", "RHip", "LHip", "RKnee", "LKnee",
"RAnkle", "LAnkle", "RHeel", "LHeel", "RSmallToe", "LSmallToe",
"RBigToe", "LBigToe", "RElbow", "LElbow", "RWrist", "LWrist"]
Will not work properly if missing values are not interpolated (i.e., if there are Nan value in the .trc file).
Open an Anaconda prompt or a terminal in a Session or Trial folder.
Type ipython.
from Pose2Sim import Pose2Sim
Pose2Sim.markerAugmentation()
Obtain a scaled model and 3D joint angles.
This can be either done fully automatically within Pose2Sim, or manually within OpenSim GUI.
Scaling and inverse kinematics are performed in a fully automatic way for each trc file.
No need for a static trial!
Note that automatic scaling is not recommended when the participant is mostly crouching or sitting. In this case, scale manually on a standing trial (see next section).
Model scaling is done according to the mean of the segment lengths, across a subset of frames. We remove the 10% fastest frames (potential outliers), the frames where the speed is 0 (person probably out of frame), the frames where the average knee and hip flexion angles are above 45° (pose estimation is not precise when the person is crouching) and the 20% most extreme segment values after the previous operations (potential outliers).
In your Config.toml file, set use_augmentation = false is you don't want to use the results with augmented marker (this is sometimes better).
Set right_left_symmetry = false if you have good reasons to think the participant is not symmetrical (e.g. if they wear a prosthetic limb).
Open an Anaconda prompt or a terminal in a Session or Trial folder.
Type ipython.
from Pose2Sim import Pose2Sim
Pose2Sim.kinematics()
Once you have the scaled model and the joint angles, you are free to go further! Inverse dynamics, muscle analysis, etc. (make sure previously add muscles from the Pose2Sim model with muscles).
If you are not fully satisfied with the results or on sitting or crouching trials, you can perform scaling and inverse kinematics in a more traditional way, with (or without) a static trial.
Scaling
Model_Pose2Sim_LSTM.osim model from Pose2Sim/OpenSim_Setup.Scaling_Setup_Pose2Sim_LSTM.xml scaling file.Inverse kinematics
IK_Setup_Pose2Sim_LSTM.xml scaling file from Pose2Sim/OpenSim_Setup.Alternatively, you can use command-line tools:
import opensim
opensim.ScaleTool("<PATH TO YOUR SCALING OR IK SETUP FILE>.xml").run()
opensim.InverseKinematicsTool("<PATH TO YOUR SCALING OR IK SETUP FILE>.xml").run()
N.B.: You'll need to adjust the time_range, output_motion_file, and enter the absolute path (NOT the relative path) to the input and output .osim, .trc, and .mot files in your setup file.
You can also run other API commands. See there for more instructions on how to use it.
A list of standalone tools (see Utilities), which can be either run as scripts, or imported as functions. Check usage in the docstring of each Python file. The figure below shows how some of these tools can be used to further extend Pose2Sim usage.
Blazepose_runsave.py Runs BlazePose on a video, and saves coordinates in OpenPose (json) or DeepLabCut (h5 or csv) format.
DLC_to_OpenPose.py Converts a DeepLabCut (h5) 2D pose estimation file into OpenPose (json) files.
AlphaPose_to_OpenPose.py Converts AlphaPose single json file to OpenPose frame-by-frame files.
calib_toml_to_easymocap.py Converts an OpenCV .toml calibration file to EasyMocap intrinsic and extrinsic .yml calibration files.
calib_easymocap_to_toml.py Converts EasyMocap intrinsic and extrinsic .yml calibration files to an OpenCV .toml calibration file.
calib_from_checkerboard.py Calibrates cameras with images or a video of a checkerboard, saves calibration in a Pose2Sim .toml calibration file. You should probably use Pose2Sim.calibration() instead, which is much easier and better.
calib_qca_to_toml.py Converts a Qualisys .qca.txt calibration file to the Pose2Sim .toml calibration file (similar to what is used in AniPose).
calib_toml_to_qca.py Converts a Pose2Sim .toml calibration file (e.g., from a checkerboard) to a Qualisys .qca.txt calibration file.
calib_toml_to_opencap.py Converts an OpenCV .toml calibration file to OpenCap .pickle calibration files.
calib_toml_to_opencap.py To convert OpenCap calibration tiles to a .toml file, please use Pose2Sim.calibration() and set convert_from = 'opencap' in Config.toml.
json_display_with_img.py Overlays 2D detected json coordinates on original raw images. High confidence keypoints are green, low confidence ones are red.
json_display_without_img.py Plots an animation of 2D detected json coordinates.
trc_plot.py Displays X, Y, Z coordinates of each 3D keypoint of a TRC file in a different matplotlib tab.
trc_from_easymocap.py Convert EasyMocap results keypoints3d .json files to .trc.
c3d_to_trc.py Converts 3D point data from a .c3d file to a .trc file compatible with OpenSim. No analog data (force plates, emg) nor computed data (angles, powers, etc.) are retrieved.
trc_to_c3d.py Converts 3D point data from a .trc file to a .c3d file compatible with Visual3D.
trc_desample.py Undersamples a trc file.
trc_Zup_to_Yup.py Changes Z-up system coordinates to Y-up system coordinates.
trc_filter.py Filters trc files. Available filters: Butterworth, Kalman, Butterworth on speed, Gaussian, LOESS, Median.
trc_gaitevents.py Detects gait events from point coordinates according to Zeni et al. (2008).
trc_combine.py Combine two trc files, for example a triangulated DeepLabCut trc file and a triangulated OpenPose trc file.
trc_from_mot_osim.py Build a trc file from a .mot motion file and a .osim model file.
bodykin_from_mot_osim.py Converts a mot file to a .csv file with rotation and orientation of all segments.
reproj_from_trc_calib.py Reprojects 3D coordinates of a trc file to the image planes defined by a calibration file. Output in OpenPose or DeepLabCut format.
If you use this code or data, please cite Pagnon et al., 2022b, Pagnon et al., 2022a, or Pagnon et al., 2021.
@Article{Pagnon_2022_JOSS,
AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
TITLE = {Pose2Sim: An open-source Python package for multiview markerless kinematics},
JOURNAL = {Journal of Open Source Software},
YEAR = {2022},
DOI = {10.21105/joss.04362},
URL = {https://joss.theoj.org/papers/10.21105/joss.04362}
}
@Article{Pagnon_2022_Accuracy,
AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy},
JOURNAL = {Sensors},
YEAR = {2022},
DOI = {10.3390/s22072712},
URL = {https://www.mdpi.com/1424-8220/22/7/2712}
}
@Article{Pagnon_2021_Robustness,
AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 1: Robustness},
JOURNAL = {Sensors},
YEAR = {2021},
DOI = {10.3390/s21196530},
URL = {https://www.mdpi.com/1424-8220/21/19/6530}
}
I would happily welcome any proposal for new features, code improvement, and more!
If you want to contribute to Pose2Sim, please see this issue.
You will be proposed a to-do list, but please feel absolutely free to propose your own ideas and improvements.
Main to-do list
✔ Pose: Support OpenPose body_25b for more accuracy, body_135 for pronation/supination. ✔ Pose: Support BlazePose for faster inference (on mobile device). ✔ Pose: Support DeepLabCut for training on custom datasets. ✔ Pose: Support AlphaPose as an alternative to OpenPose. ✔ Pose: Define custom model in config.toml rather than in skeletons.py. ✔ Pose: Integrate pose estimation within Pose2Sim (via RTMlib). ▢ Pose: Support MMPose, SLEAP, etc. ▢ Pose: Implement RTMPoseW3D and monocular 3D kinematics ▢ Pose: Directly reading from DeepLabCut .csv or .h5 files instead of converting to .json (triangulation, person association, calibration, synchronization...) ▢ Pose: GUI help for DeepLabCut model creation.
✔ Calibration: Convert Qualisys .qca.txt calibration file. ✔ Calibration: Convert Optitrack extrinsic calibration file. ✔ Calibration: Convert Vicon .xcp calibration file. ✔ Calibration: Convert OpenCap .pickle calibration files. ✔ Calibration: Convert EasyMocap .yml calibration files. ✔ Calibration: Convert bioCV calibration files. ✔ Calibration: Easier and clearer calibration procedure: separate intrinsic and extrinsic parameter calculation, edit corner detection if some are wrongly detected (or not visible). ✔ Calibration: Possibility to evaluate extrinsic parameters from cues on scene. ▢ Calibration: Support vertical checkerboard. ▢ Calibration: Once object points have been detected or clicked once, track them for live calibration of moving cameras. Propose to click again when they are lost. ▢ Calibration: Calibrate cameras by pairs and compute average extrinsic calibration with aniposelib. ▢ Calibration: Fine-tune calibration with bundle adjustment. ▢ Calibration: Support ChArUco board detection (see there). ▢ Calibration: Calculate calibration with points rather than board. (1) SBA calibration with wand (cf Argus, see converter here). Set world reference frame in the end. ▢ Calibration: Alternatively, self-calibrate with OpenPose keypoints. Set world reference frame in the end. ▢ Calibration: Convert fSpy calibration based on vanishing point.
✔ Synchronization: Synchronize cameras on keypoint speeds. ▢ Synchronization: Synchronize in multi-person mode: click on the person to synchronize on.
✔ Person Association: Automatically choose the main person to triangulate. ✔ Person Association: Multiple persons association. 1. Triangulate all the persons whose reprojection error is below a certain threshold (instead of only the one with minimum error), and then track in time with speed cf Slembrouck 2020? or 2. Based on affinity matrices Dong 2021? or 3. Based on occupancy maps Yildiz 2012? or 4. With a neural network Huang 2023?
✔ Triangulation: Triangulation weighted with confidence. ✔ Triangulation: Set a likelihood threshold below which a camera should not be used, a reprojection error threshold, and a minimum number of remaining cameras below which triangulation is skipped for this frame. ✔ Triangulation: Interpolate missing frames (cubic, bezier, linear, slinear, quadratic) ✔ Triangulation: Show mean reprojection error in px and in mm for each keypoint. ✔ Triangulation: Show how many cameras on average had to be excluded for each keypoint. ✔ Triangulation: Evaluate which cameras were the least reliable. ✔ Triangulation: Show which frames had to be interpolated for each keypoint. ✔ Triangulation: Solve limb swapping (although not really an issue with Body_25b). Try triangulating with opposite side if reprojection error too large. Alternatively, ignore right and left sides, use RANSAC or SDS triangulation, and then choose right or left by majority voting. More confidence can be given to cameras whose plane is the most coplanar to the right/left line. ✔ Triangulation: Undistort 2D points before triangulating (and distort them before computing reprojection error). ✔ Triangulation: Offer the possibility to augment the triangulated data with the OpenCap LSTM. Create "BODY_25_AUGMENTED" model, Scaling_setup, IK_Setup. ✔ Triangulation: Multiple person kinematics (output multiple .trc coordinates files). Triangulate all persons with reprojection error above threshold, and identify them by minimizing their displacement across frames. ▢ Triangulation: Pre-compile weighted_triangulation and reprojection with @jit(nopython=True, parallel=True) for faster execution. ▢ Triangulation: Offer the possibility of triangulating with Sparse Bundle Adjustment (SBA), Extended Kalman Filter (EKF), Full Trajectory Estimation (FTE) (see AcinoSet). ▢ Triangulation: Implement normalized DLT and RANSAC triangulation, Outlier rejection (sliding z-score?), as well as a triangulation refinement step. ▢ Triangulation: Track hands and face, and add articulated OpenSim hand.
✔ Filtering: Available filtering methods: Butterworth, Butterworth on speed, Gaussian, Median, LOESS (polynomial smoothing). ✔ Filtering: Implement Kalman filter and Kalman smoother. ▢ Filtering: Implement smoothNet
✔ OpenSim: Integrate better spine from lifting fullbody model to the gait full-body model, more accurate for the knee. ✔ OpenSim: Optimize model marker positions as compared to ground-truth marker-based positions. ✔ OpenSim: Add scaling and inverse kinematics setup files. ✔ OpenSim: Add full model with contact spheres (SmoothSphereHalfSpaceForce) and full-body muscles (DeGrooteFregly2016Muscle), for Moco for example. ✔ OpenSim: Add model with ISB shoulder. ✔ OpenSim: Integrate OpenSim in Pose2Sim. ✔ OpenSim: Do not require a separate scaling trial: scale on the 10% slowest frames of the moving trial instead, or take median scaling value. ▢ OpenSim: Implement optimal fixed-interval Kalman smoothing for inverse kinematics (this OpenSim fork), or Biorbd)
✔ GUI: Blender add-on (cf MPP2SOS), Maya-Mocap and BlendOsim. ▢ GUI: App or webapp (e.g., with gradio, Streamlit, or Napari ). Also see tkinter interfaces (or Kivy if we want something nice and portable, or Google Colab). Maybe have a look at the Deeplabcut GUI for inspiration. ▢ GUI: 3D plot of cameras and of triangulated keypoints. ▢ GUI: Demo on Google Colab (see Sports2D for OpenPose and Python package installation on Google Drive).
✔ Demo: Provide Demo data for users to test the code. ✔ Demo: Add videos for users to experiment with other pose detection frameworks ✔ Demo: Time shift videos and json to demonstrate synchronization ✔ Demo: Add another virtual person to demonstrate personAssociation ▢ Tutorials: Make video tutorials. ▢ Doc: Use Sphinx, MkDocs, or github.io (maybe better) for clearer documentation.
✔ Pip package ✔ Batch processing (also enable non-batch processing) ✔ Catch errors ▢ Conda package ▢ Docker image ▢ Integrate Sports2D for OpenSim analysis from a single camera ▢ Real-time: Run Pose estimation, Person association, Triangulation, Kalman filter, IK frame by frame (instead of running each step for all frames) ▢ Config parameter for non batch peocessing
▢ Run from command line via click or typer ▢ Utilities: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...) ▢ Utilities: Create trc_to_c3d.py script
✔ Bug: calibration.py. FFMPEG error message when calibration files are images. See there. ✔ Bug: common.py, class plotWindow(). Python crashes after a few runs of
Pose2Sim.filtering()whendisplay_figures=true. See there.
Acknowledgements:
BSD 3-Clause License
Copyright (c) 2022, perfanalytics All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
FAQs
Perform a markerless kinematic analysis from multiple calibrated views as a unified workflow from an OpenPose input to an OpenSim result.
We found that pose2sim demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.
Security News
Research
Socket researchers found a malicious Maven package impersonating the legitimate ‘XZ for Java’ library, introducing a backdoor for remote code execution.