Skip to main content
SpringerLink
Log in

Action recognition from depth sequence using depth motion maps-based local ternary patterns and CNN

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents a method for human action recognition from depth sequences captured by the depth camera. The main idea of the method is the action mapping image classification via convolutional neural network (CNN) based approach. Firstly, we project the raw frames onto three orthogonal Cartesian planes and stack the results into three still images (corresponding to the front, side, and top views) to form the Depth Motion Maps (DMMs). Secondly, Local Ternary Pattern (LTP) is introduced as an image filter for DMMs, thus to improve the distinguishability of similar actions. Finally, we apply CNN to action recognition by classifying corresponding LTP-encoded images. Experimental results on the popular and challenging benchmark MSR-Action 3D and MSR-Gesture dataset show the effectiveness of the presented method and meet real-time action recognition task requirements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Aggarwal JK, Xia L (2014) Human activity recognition from 3D data: a review. Pattern Recogn Lett 48(1):70–80

    Article  Google Scholar 

  2. Chen C, Jafari R, Kehtarnavaz N (2015) Action recognition from depth sequences using depth motion maps-based local binary patterns. In: IEEE Winter Conference on Applications of Computer Vision. IEEE Computer Society, Hawaii, p 1092–1099

  3. Chen C, Hou Z, Zhang B, Jiang J, Yang Y (2015) Gradient local auto-correlations and extreme learning machine for depth-based activity recognition. In: 11th international symposium on Visual Computing (ISVC'15). Springer International Publishing, Las Vegas, p 613-623

  4. Chen C, Liu K, Kehtarnavaz N (2016) Real-time human action recognition based on depth motion maps. J Real-Time Image Proc 12(1):155–163

    Article  Google Scholar 

  5. Guo P, Miao Z, Shen Y et al (2014) Continuous human action recognition in real time. Multimed Tools Appl 68(3):827–844

    Article  Google Scholar 

  6. Hattori H, Lee N, Boddeti VN et al (2018) Synthesizing a scene-specific pedestrian detector and pose estimator for static video surveillance. Int J Comput Vis 126(9):1027–1044

    Article  Google Scholar 

  7. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Computer Vision and Pattern Recognition. IEEE Computer Society, LAS Vegas, p 770-778

  8. Junsong Y, Ying W, Zicheng L, Jiang W (2012) Mining action let ensemble for action recognition with depth cameras. In: Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Providence, Rhode Island, p 1290–1297

  9. Krizhevsky A, Sutskever I, Hinton GE (2012). ImageNet classification with deep convolutional neural networks. In: International Conference on Neural Information Processing Systems. Curran Associates Inc, Doha, p 1097-1105

  10. Kurakin A, Zhang Z, Liu Z (2012) A real time system for dynamic hand gesture recognition with a depth sensor. In: European Signal Processing Conference. IEEE. Bucharest, Romania, p 1975–1979

  11. Laraba S, Brahimi M, Tilmanne J et al (2017) 3D skeleton-based action recognition by representing motion capture sequences as 2D-RGB images. Comput Anima Virt W 28(3-4):1–11

  12. Li X, Li J (2013) Lpt optimization algorithm in the nuclear environment image monitoring. J Softw 8(3):659–665

    Google Scholar 

  13. Li W, Zhang Z, Liu Z (2010) Action recognition based on a bag of 3D points. In: Computer Vision and Pattern Recognition Workshops. IEEE Computer Society, San Francisco, p 9-14

  14. Oreifej O, Liu Z (2013) HON4D: histogram of oriented 4D normals for activity recognition from depth sequences. In: Computer Vision and Pattern Recognition. IEEE Computer Society, Portland, p 716–723

  15. Reily B, Han F, Parker LE et al (2018) Skeleton-based bio-inspired human activity prediction for real-time human–robot interaction. Auton Robot 42(4):1281–1298

    Article  Google Scholar 

  16. Shen Z, Liu Z, Li J, et al. (2017) DSOD: Learning deeply supervised object detectors from scratch. In: IEEE International Conference on Computer Vision, ICCV, Venis Italy, p 1937–1945

  17. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations. ICLR, San Diego, p 1-14

  18. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

    Article  MathSciNet  MATH  Google Scholar 

  19. Wang J, Liu Z, Chorowski J, Chen Z, Wu Y (2012) Robust 3d action recognition with random occupancy patterns. In: European Conference on Computer Vision. Springer-Verlag, Florence, p 872–885

  20. Wang P, Li W, Gao Z, Tang C, Zhang J, Ogunbona P (2015) ConvNets-Based Action Recognition from Depth Maps through Virtual Cameras and Pseudo coloring. In: ACM International Conference on Multimedia. ACM, Brisbane, p 1119-1122

  21. Wang P, Li W, Gao Z, Zhang J, Tang C, Ogunbona P (2016) Action recognition from depth maps using deep convolutional neural networks. IEEE Trans Human-Mach Syst 46(4):498–509

    Article  Google Scholar 

  22. Xia L, Aggarwal JK (2013) Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera. In: Computer Vision and Pattern Recognition. IEEE Computer Society, Portland, p 2834-2841

  23. Yang X, Tian YL (2012) Eigen joints-based action recognition using naïve-bayes-nearest-neighbor. In: Computer Vision and Pattern Recognition Workshops. IEEE Computer Society, Providence, Rhode Island, p 14–19

  24. Yang R, Yang R (2015) DMM-pyramid based deep architectures for action recognition with depth cameras. In: Asian Conference on Computer Vision. Springer International Publishing, Singapore, p 37–49

  25. Yang X, Zhang C, Tian Y (2012) Recognizing actions using depth motion maps-based histograms of oriented gradients. In: Proceedings of ACM International Conference on Multimedia. ACM, Nara, Japan, p 1057-1060

  26. Yang J, Wang Y, Lv Z et al (2018) Interaction with three-dimensional gesture and character input in virtual reality: recognizing gestures in different directions and improving user input. IEEE Consum Electro 7(2):64–72

    Article  Google Scholar 

  27. Zanfir M, Leordeanu M, Sminchisescu C (2013) The moving pose: an efficient 3D kinematics descriptor for low-latency action recognition and detection. In: IEEE International Conference on Computer Vision. IEEE Computer Society, Sydney, p 2752–2759

Download references

Acknowledgments

The authors thank the anonymous reviewers for valuable comments. This work is mainly supported by grants from Zhejiang Provincial Top Key Discipline of Computer Software and Theory, National Natural Science Foundation of China (No. 61170109, 61672467), and National Science Foundation of Zhejiang Province (No. 2015C31095), China.

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Zhejiang Normal University, Jinhua, China

    Zhifei Li, Zhonglong Zheng & Feilong Lin

  2. Department of Computer Science, CITYU, Hong Kong, Hong Kong

    Zhifei Li, Howard Leung & Qing Li

Authors
  1. Zhifei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhonglong Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feilong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Howard Leung
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhifei Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Zheng, Z., Lin, F. et al. Action recognition from depth sequence using depth motion maps-based local ternary patterns and CNN. Multimed Tools Appl 78, 19587–19601 (2019). https://doi.org/10.1007/s11042-019-7356-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7356-3

Keywords

Search

Navigation