Skip to main content
SpringerLink
Log in

Spatiotemporal Saliency Based Multi-stream Networks for Action Recognition

  • Conference paper
  • First Online:
Pattern Recognition (ACPR 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1180))

Included in the following conference series:

Abstract

Human action recognition is a challenging research topic since videos often contain clutter backgrounds, which impairs the performance of human action recognition. In this paper, we propose a novel spatiotemporal saliency based multi-stream ResNet for human action recognition, which combines three different streams: a spatial stream with RGB frames as input, a temporal stream with optical flow frames as input, and a spatiotemporal saliency stream with spatiotemporal saliency maps as input. The spatiotemporal saliency stream is responsible for capturing the spatiotemporal object foreground information from spatiotemporal saliency maps which are generated by a geodesic distance based video segmentation method. Such architecture can reduce the background interference in videos and provide the spatiotemporal object foreground information for human action recognition. Experimental results on UCF101 and HMDB51 datasets demonstrate that the complementary spatiotemporal information can further improve the performance of action recognition, and our proposed method obtains the competitive performance compared with the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems (NIPS) (2014)

    Google Scholar 

  2. Wang, L., et al.: Temporal segment networks for action recognition in videos. IEEE Trans. Pattern Anal. Mach. Intell. 41(11), 2740–2755 (2018)

    Article  Google Scholar 

  3. Feichtenhofer, C., Pinz, A., Wildes, R.P.: Spatiotemporal residual networks for video action recognition. In: Advances in Neural Information Processing Systems (NIPS) (2016)

    Google Scholar 

  4. Wang, Y., et al.: Spatiotemporal pyramid network for video action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  5. Ji, S., et al.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2013)

    Article  Google Scholar 

  6. Tran, D., et al.: Learning spatiotemporal features with 3D convolutional networks. In: IEEE International Conference on Computer Vision (ICCV) (2014)

    Google Scholar 

  7. Diba, A., et al.: Temporal 3D ConvNets: new architecture and transfer learning for video classification. arXiv preprint arXiv:1711.08200 (2017)

  8. Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3D residual networks. In: IEEE International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  9. Kar, A., et al.: AdaScan: adaptive scan pooling in deep convolutional neural networks for human action recognition in videos. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  10. Sun, S., et al.: Optical flow guided feature: a fast and robust motion representation for video action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  11. Xie, S., et al.: Rethinking spatiotemporal feature learning: speed-accuracy trade-offs in video classification. In: European Conference on Computer Vision (ECCV) (2017)

    Google Scholar 

  12. Jing, L., Ye, Y., Yang, X., Tian, Y.: 3D convolutional neural network with multi-model framework for action recognition. In IEEE International Conference on Image Processing (ICIP) (2017)

    Google Scholar 

  13. Liu, X., Yang, X.: Multi-stream with deep convolutional neural networks for human action recognition in videos. In: Cheng, L., Leung, A.C.S., Ozawa, S. (eds.) ICONIP 2018. LNCS, vol. 11301, pp. 251–262. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04167-0_23

    Chapter  Google Scholar 

  14. Wang, W., Shen, J., Porikli, F.: Saliency-aware geodesic video object segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  15. Achanta, R., et al.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2281 (2012)

    Article  Google Scholar 

  16. He, K., et al.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  17. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3

    Chapter  Google Scholar 

  18. Soomro, K., Zamir, A.R., Shah, M.: UCF101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)

  19. Deng, J., et al.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2009)

    Google Scholar 

  20. Wang, H., Schmid, C.: Action recognition with improved trajectories. In: IEEE International Conference on Computer Vision (ICCV) (2014)

    Google Scholar 

  21. Sun, L., et al.: Lattice long short-term memory for human action recognition. In: IEEE International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  22. Leordeanu, M., Sukthankar, R., Sminchisescu, C.: Efficient closed-form solution to generalized boundary detection. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 516–529. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33765-9_37

    Chapter  Google Scholar 

  23. Kuehne, H., et al.: HMDB51: a large video database for human motion recognition. In: IEEE International Conference on Computer Vision (ICCV) (2013)

    Google Scholar 

  24. Hara, K., Kataoka, H., Satoh, Y.: Learning spatio-temporal features with 3D residual networks for action recognition. In: IEEE International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  25. Lei, P., Todorovic, S.: Temporal deformable residual networks for action segmentation in videos. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  26. Ng, Y.H., et al.: Beyond short snippets: deep networks for video classification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  27. Poppe, R.: A survey on vision-based human action recognition. Image Vis. Comput. 28(6), 976–990 (2010)

    Article  Google Scholar 

  28. Huang, G., et al.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  29. Tian, C., et al.: Image denoising using deep CNN with batch renormalization. Neural Netw. 121, 461–473 (2020)

    Article  Google Scholar 

Download references

Acknowledgment

This study is supported by the National Natural Science Foundation of China (Grant No. 61562013), the Natural Science Foundation of Guangxi Province (CN) (2017GXNSFDA198025), the Study Abroad Program for Graduate Student of Guilin University of Electronic Technology (GDYX2018006), the Marsden Fund of New Zealand, the National Natural Science Foundation of China (Grant 61602407), Natural Science Foundation of Zhejiang Province (Grant LY18F020008), the China Scholarship Council (CSC) and the New Zealand China Doctoral Research Scholarships Program.

Author information

Authors and Affiliations

  1. School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, China

    Zhenbing Liu, Zeya Li, Ming Zong, Wanting Ji, Ruili Wang & Yan Tian

  2. School of Natural and Computational Sciences, Massey University, Auckland, New Zealand

    Zhenbing Liu, Zeya Li, Ming Zong, Wanting Ji, Ruili Wang & Yan Tian

  3. School of Computer and Information Engineering, Zhejiang Gongshang University, Hangzhou, China

    Zhenbing Liu, Zeya Li, Ming Zong, Wanting Ji, Ruili Wang & Yan Tian

Authors
  1. Zhenbing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zeya Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming Zong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wanting Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruili Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yan Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ming Zong .

Editor information

Editors and Affiliations

  1. University of Waikato, Hamilton, New Zealand

    Michael Cree

  2. National Ilan University, Yilan, Taiwan

    Fay Huang

  3. State University of New York at Buffalo, Buffalo, NY, USA

    Junsong Yuan

  4. Auckland University of Technology, Auckland, New Zealand

    Wei Qi Yan

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, Z., Li, Z., Zong, M., Ji, W., Wang, R., Tian, Y. (2020). Spatiotemporal Saliency Based Multi-stream Networks for Action Recognition. In: Cree, M., Huang, F., Yuan, J., Yan, W. (eds) Pattern Recognition. ACPR 2019. Communications in Computer and Information Science, vol 1180. Springer, Singapore. https://doi.org/10.1007/978-981-15-3651-9_8

Download citation

  • DOI: https://doi.org/10.1007/978-981-15-3651-9_8

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-15-3650-2

  • Online ISBN: 978-981-15-3651-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation