Elsevier

Pattern Recognition Letters

Volume 165, January 2023, Pages 138-145
Pattern Recognition Letters

Sticks and STONES may build my bones: Deep learning reconstruction of limb rotations in stick figures

https://doi.org/10.1016/j.patrec.2022.12.012Get rights and content
Under a Creative Commons license
open access

Highlights

  • A method for estimating missing limb rotations in animations of minimal stick figures.

  • STONES uses an RNN to convert a few sparse 3D joint points to segment rotations.

  • A RollingSticks database is provided with high quality joint data from motion capture of fitness exercises.

  • Results better than other ML approaches with average fits 98% and RMSE 1 .

  • Improved accuracy compared to video-based state-of-the-art DL methods using only a fraction of the data.

Abstract

Monitoring and analyzing physical activity is becoming an important task in both clinical and non-clinical settings. To accomplish this desideratum, stick figures are often used as abstractions of human poses and movements by representing body segments as straight lines (sticks). Despite their straightforwardness, this minimalist representation is incomplete as it lacks the segments’ longitudinal rotations, and therefore, is insufficient for applications requiring full 3D kinematic data. We introduce STONES, an advanced machine learning approach for estimating longitudinal body segment rotations of based on stick figures defined from a minimal set of body points. Our approach relies on a recurrent deep neural network, which takes 3D joint positions from a minimalist stick figure representation, such as those acquired by conventional depth camera sensors, and completes it with accurate longitudinal segment rotations. We validated our approach via a test scenario based on exergaming activities (e.g., lunges, squats, and kicks), which are becoming an emerging trend in several healthcare sectors, and our estimations show a fit above 98% and mean errors of approximately 1 . Our deep learning approach effectively surpasses other machine learning-based strategies and closely matches the accuracy of state-of-the-art motion capture systems while running at real-time speeds.

Keywords

Stick figure
Pose estimation
Longitudinal rotations
Machine learning
Neural networks

Data availability

  • Data will be made available on request.

Cited by (0)

Editor: S. Sarkar.