Skip to main content
SpringerLink
Log in
Book cover

International Conference on Neural Information Processing

ICONIP 2017: Neural Information Processing pp 377–385Cite as

A Spatio-Temporal Convolutional Neural Network for Skeletal Action Recognition

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10636))

Abstract

Human action recognition based on 3D skeleton data is a rapidly growing research area in computer vision. Convolutional Neural Networks (CNNs) have been proved to be the most effective representation learning in many vision tasks, but there is little work of CNNs for skeletal action recognition due to the variable-length of time sequences and lack of big skeleton datasets. In this paper, we propose a Spatio-Temporal CNN for skeleton based action recognition. A CNN architecture with two convolutional layers is used, in which the first layer is used to capture the spatial patterns and second layer for spatio-temporal patterns. Some techniques including data augmentation and segment pooling strategy are employed for long sequences. Experimental results on MSR Action3D, MSR DailyActivity3D and UT-Kinect show that our approach achieves comparable results with those of the state-of-the-art models.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Aggarwal, J.K., Ryoo, M.S.: Human activity analysis: a review. ACM Comput. Surv. 43(3), 1–43 (2011)

    Article  Google Scholar 

  2. Han, J.: Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans. Cybern. 43(5), 1318–1334 (2013)

    Article  Google Scholar 

  3. Chen, L., Wei, H., Ferryman, J.M.: A survey of human motion analysis using depth imagery. Pattern Recogn. Lett. 34, 1995–2006 (2013)

    Article  Google Scholar 

  4. Ye, M., Zhang, Q., Wang, L., Zhu, J., Yang, R., Gall, J.: A survey on human motion analysis from depth data. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. LNCS, vol. 8200, pp. 149–187. Springer, Heidelberg (2013). doi:10.1007/978-3-642-44964-2_8

    Chapter  Google Scholar 

  5. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from single depth images. IEEE Conf. Comput. Vis. Pattern Recogn. 411(1), 1297–1304 (2011)

    Google Scholar 

  6. Xia, L., Chen, C.C., Aggarwal, J.K.: View invariant human action recognition using histograms of 3D joints. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 20–27 (2012)

    Google Scholar 

  7. Yang, X., Tian, Y.L.: Effective 3D action recognition using eigenjoints. J. Vis. Commun. Image Represent. 25(1), 2–11 (2014)

    Article  MathSciNet  Google Scholar 

  8. Zanfir, M., Leordeanu, M., Sminchisescu, C.: The moving pose: an efficient 3D kinematics descriptor for low-latency action recognition and detection. In: IEEE International Conference on Computer Vision, pp. 2752–2759 (2013)

    Google Scholar 

  9. Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R., Bajcsy, R.: Sequence of the most informative joints (smij): a new representation for human skeletal action recognition. J. Vis. Commun. Image Represent. 25(1), 24–38 (2014)

    Article  Google Scholar 

  10. Hussein, M.E., Torki, M., Gowayyed, M.A., El-Saban, M.: Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations. In: The 23rd International Joint Conference on Artificial Intelligence (2013)

    Google Scholar 

  11. Sivalingam, R., Somasundaram, G., Bhatawadekar, V., Morellas, V., Papanikolopoulos, N.: Sparse representation of point trajectories for action classification. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 3601–3606 (2012)

    Google Scholar 

  12. Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1110–1118 (2015)

    Google Scholar 

  13. Zhu, W., Lan, C., Xing, J., Zeng, W., Li, Y., Shen, L., Xie, X.: Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. In: Thirtieth AAAI Conference on Artificial Intelligence (2016)

    Google Scholar 

  14. Tao, L., Vidal, R.: Moving poselets: a discriminative and interpretable skeletal motion representation for action recognition. In: IEEE Conference on Computer Vision Workshop (2015)

    Google Scholar 

  15. Wang, J., Liu, Z., Wu, Y., Yuan, J.: Mining actionlet ensemble for action recognition with depth cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1290–1297 (2012)

    Google Scholar 

  16. Wang, C., Wang, Y., Yuille, A.: An approach to pose-based action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2013)

    Google Scholar 

  17. Li, W., Zhang, Z., Liu, Z.: Action recognition based on a bag of 3D points. In: Workshop on Human Activity Understanding from 3D Data, pp. 9–14 (2010)

    Google Scholar 

  18. Zhu, Y., Chen, W., Guo, G.: Fusing spatiotemporal features and joints for 3D action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 486–491 (2013)

    Google Scholar 

  19. Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: a matlab-like environment for machine learning. In: BigLearn, NIPS Workshop (2011)

    Google Scholar 

  20. Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3D skeletons as points in a lie group. In: IEEE Conference on Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  21. Gowayyed, M.A., Torki, M., Hussein, M.E., El-Saban, M.: Histogram of oriented displacements (HOD): describing trajectories of human joints for action recognition. In: International Joint Conference on Artificial Intelligence, pp. 1351–1357 (2013)

    Google Scholar 

  22. Anirudh, R., Turaga, P., Su, J., Srivastava, A.: Elastic functional coding of human actions: from vector-fields to latent variables. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3147–3155 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, 3663 North Zhongshan Road, Shanghai, China

    Lizhang Hu & Jinhua Xu

Authors
  1. Lizhang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinhua Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinhua Xu .

Editor information

Editors and Affiliations

  1. Guangdong University of Technology, Guangzhou, China

    Derong Liu

  2. Guangdong University of Technology, Guangzhou, China

    Shengli Xie

  3. South China University of Technology, Guangzhou, China

    Yuanqing Li

  4. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Dongbin Zhao

  5. King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia

    El-Sayed M. El-Alfy

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Hu, L., Xu, J. (2017). A Spatio-Temporal Convolutional Neural Network for Skeletal Action Recognition. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10636. Springer, Cham. https://doi.org/10.1007/978-3-319-70090-8_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-70090-8_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-70089-2

  • Online ISBN: 978-3-319-70090-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation