Skip to main content
SpringerLink
Log in

Real-time dance evaluation by markerless human pose estimation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents a unified framework that evaluates dance performance by markerless estimation of human poses. Dance involves complicated poses such as full-body rotation and self-occlusion, so we first develop a human pose estimation method that is invariant to these factors. The method uses ridge data and data pruning. Then we propose a metric to quantify the similarity (i.e., timing and accuracy) between two dance sequences. To validate the proposed dance evaluation method, we conducted several experiments to evaluate pose estimation and dance performance on the benchmark dataset EVAL, SMMC-10 and a large K-Pop dance database, respectively. The proposed methods achieved pose estimation accuracy of 0.9358 mAP, average pose error of 3.88 cm, and 98% concordance with experts’ evaluation of dance performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. The K-Pop database is available from https://goo.gl/NoVDm4.

References

  1. Alexander T, Szegedy C (2014) Deeppose: Human pose estimation via deep neural networks. In: The IEEE conference on computer vision and pattern recognition, pp 1653–1660

  2. Baak A, Müller M, Bharaj G, Seidel H-P, Theobalt C (2011) A data-driven approach for real-time full body pose reconstruction from a depth camera. In: IEEE international conference on computer vision, pp 1092–1099

  3. Cao Z, Simon T, Wei S, Sheikh Y (2017) Realtime multi-person 2D pose estimation using part affinity fields. In: The IEEE conference on computer vision and pattern recognition

  4. Comaniciu D, Ramesh V, Meer P (2003) Kernel-based object tracking. IEEE Trans Pattern Anal Mach Intell 25(5):564–577

    Article  Google Scholar 

  5. Dillencourt M, Samet H, Tamminen M (1992) A general approach to connected-component labeling for arbitrary image representations. J ACM 39(2):253–280

    Article  MathSciNet  Google Scholar 

  6. Ganapathi V, Plagemann C, Koller D, Thrun S (2010) Real time motion capture using a single time-of-flight camera. In: IEEE conference on computer vision and pattern recognition, pp 755–762

  7. Ganapathi V, Plagemann C, Koller D, Thrun S (2012) Real-time human pose tracking from range data. In: European conference on computer vision, pp 738–751

  8. Hough P (1959) Machine analysis of bubble chamber pictures. In: International conference on high energy accelerators and instrumentation, pp 73

  9. Huang J, Altamar D (2016) Pose estimation on depth images with convolutional neural network

  10. Jalal A, Kamal S, Kim D (2014) A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors 14(7):11735–11759

    Article  Google Scholar 

  11. Jalal A, Kamal S, Kim D (2015) Depth Silhouettes Context: A new robust feature for human tracking and activity recognition based on embedded HMMs. In: 12th international conference on ubiquitous robots and ambient intelligence, pp 294–299

  12. Jung H, Lee S, Heo Y, Yun I (2015) Random tree walk toward instantaneous 3D human pose estimation. In: The IEEE conference on computer vision and pattern recognition, pp 2467–2474

  13. Kim Y, Kim D (2015) Efficient body part tracking using ridge data and data pruning. In: IEEE-RAS 15th international conference on humanoid robots, pp 114–120

  14. Lee M, Nevatia R (2009) Human pose tracking in monocular sequence using multilevel structured models. IEEE Trans Pattern Anal Mach Intell 31(1):27–38

    Article  Google Scholar 

  15. Ofli F, Kurillo G, Obdrzalek S, Bajcsy R, Jimison H, Pavel M (2015) Design and evaluation of an interactive exercise coaching system for older adults: lessons learned. J Biom Heath Inform 20(1):201–212

    Article  Google Scholar 

  16. Plagemann C, Ganapathi V, Koller D, Thrun S (2010) Real-time identification and localization of body parts from depth images. In: IEEE international conference on robotics and automation, pp 3108–3113

  17. Raptis M, Kirovski D, Hoppe H (2011) Real-time classification of dance gestures from skeleton animation. In: The 2011 ACM SIGGRAPH/Eurographics symposium on computer animation 147–156

  18. Reyes M, Dominguez G, Escalera S (2011) Featureweighting in dynamic timewarping for gesture recognition in depth data. In: IEEE international conference on computer vision workshops, pp 1182–1188

  19. Schramm R, Jung C, Miranda E (2015) Dynamic time warping for music conducting gestures evaluation. IEEE Trans Multimedia 17(2):243–255

    Article  Google Scholar 

  20. Shotton J, Girshick R, Fitzgibbon A, Sharp T, Cook M, Finocchio M, Moore R, Kohli P, Criminisi A, Kipman A, Blake A (2013) Efficient human pose estimation from single depth images. IEEE Trans Pattern Anal Mach Intell 35 (12):2821–2840

    Article  Google Scholar 

  21. Sung J, Ponce C, Selman B, Saxena A (2011) Human Activity Detection from RGBD Images, plan, activity, and intent recognition, vol 64

  22. Xia L, Chen C, Aggarwal J (2012) View invariant human action recognition using histograms of 3d joints. In: IEEE computer society conference on computer vision and pattern recognition workshops, pp 20–27

  23. Yang X, Tian Y (2014) Effective 3d action recognition using eigenjoints. J Vis Commun Image Represent 25(1):2–11

    Article  Google Scholar 

  24. Ye M, Yang R (2014) Real-time simultaneous pose and shape estimation for articulated objects using a single depth camera. In: IEEE conference on computer vision and pattern recognition, pp 2345–2352

  25. Yun K, Honorio J, Chattopadhyay D, Berg T, Samaras D (2012) Two-person interaction detection using body-pose features and multiple instance learning. In: IEEE computer society conference on computer vision and pattern recognition workshops, pp 28–35

Download references

Acknowledgments

This research was partially supported by the MSIT (Ministry of Science, ICT), Korea, under the SW Starlab support program (IITP-2017-0-00897) supervised by the IITP (Institute for Information & communications Technology Promotion).

This work was partially supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (IITP-2014-0-00059, Development of Predictive Visual Intelligence Technology).

Author information

Authors and Affiliations

  1. Computer Science and Engineering, Pohang University of Science and Technology, Pohang, South Korea

    Yeonho Kim & Daijin Kim

Authors
  1. Yeonho Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daijin Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daijin Kim.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(WMV 6.81 MB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, Y., Kim, D. Real-time dance evaluation by markerless human pose estimation. Multimed Tools Appl 77, 31199–31220 (2018). https://doi.org/10.1007/s11042-018-6068-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6068-4

Keywords

Search

Navigation