Abstract
Action quality assessment, which aims at evaluating the performance of specific actions, has drawn more and more attention due to its extensive demand in sports, health care, etc. Unlike action recognition, in which a few typical frames are sufficient for classification, action quality assessment requires analysis at a fine temporal granularity to discover the subtle motion difference. In this paper, we propose a novel spatio-temporal framework for action quality assessment at full-frame-rate (25fps), which consists of two steps: i.e. spatio-temporal feature extraction and temporal feature fusion, respectively. In the first step, to generate representative spatio-temporal dynamics, we utilize a spatial convolutional network (SCN) together with specially designed temporal convolutional networks (TCNs) and train them by a two-stage strategy. In the second step, we introduce an attention mechanism to fuse features in the temporal dimension according to their impact on the overall performance. Compared with existing three dimensional convolutional neural networks (3D-CNN) based methods, our model is capable of capturing more action quality relevant details. As a by-product, our model can also attend to the highlight moments in sports videos, which gives a better interpretation of the score. Extensive experiments on three public benchmarks demonstrate that the proposed method has distinct advantage in action quality assessment and achieves improvement over the state-of-the-art.
This work was supported by the National Key Research and Development Plan of China (Grant No. 2016YFB1001002) and the Foundation for Innovative Research Groups through the National Natural Science Foundation of China (Grant No. 61421003).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)
Bilen, H., Fernando, B., Gavves, E., Vedaldi, A., Gould, S.: Dynamic image networks for action recognition. In: CVPR, pp. 3034–3042 (2016)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: CVPR, pp. 6299–6308 (2017)
Chollet, F., et al.: Keras: the python deep learning library. Astrophysics Source Code Library (2018)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR, pp. 248–255 (2009)
Gregor, K., Danihelka, I., Graves, A., Rezende, D.J., Wierstra, D.: Draw: a recurrent neural network for image generation. arXiv preprint arXiv:1502.04623 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Holschneider, M., Kronland-Martinet, R., Morlet, J., Tchamitchian, P.: A real-time algorithm for signal analysis with the help of the wavelet transform. In: Combes, J.M., Grossmann, A., Tchamitchian, P. (eds.) Wavelets, pp. 286–297. Springer, Heidelberg (1990). https://doi.org/10.1007/978-3-642-75988-8_28
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NeurIPS, pp. 1097–1105 (2012)
Laptev, I.: On space-time interest points. Int. J. Comput. Vision 64(2–3), 107–123 (2005)
Laptev, I., Marszalek, M., Schmid, C., Rozenfeld, B.: Learning realistic human actions from movies. In: CVPR, pp. 1–8 (2008)
Le, Q.V., Zou, W.Y., Yeung, S.Y., Ng, A.Y.: Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In: CVPR, pp. 3361–3368 (2011)
Lea, C., Flynn, M.D., Vidal, R., Reiter, A., Hager, G.D.: Temporal convolutional networks for action segmentation and detection. In: CVPR, pp. 1003–1012 (2017)
Li, Y., Chai, X., Chen, X.: End-to-end learning for action quality assessment. In: PCM, pp. 125–134 (2018)
Li, Y., Chai, X., Chen, X.: ScoringNet: learning key fragment for action quality assessment with ranking loss in skilled sports. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11366, pp. 149–164. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20876-9_10
Park, D., Ramanan, D.: N-best maximal decoders for part models. In: ICCV, pp. 2627–2634 (2011)
Parmar, P., Morris, B.T.: Learning to score olympic events. In: CVPRW, pp. 76–84 (2017)
Pirsiavash, H., Vondrick, C., Torralba, A.: Assessing the quality of actions. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 556–571. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_36
Qiu, Z., Yao, T., Mei, T.: Learning spatio-temporal representation with pseudo-3D residual networks. In: ICCV, pp. 5534–5542 (2017)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NeurIPS, pp. 568–576 (2014)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR, pp. 2818–2826 (2016)
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: ICCV, pp. 4489–4497 (2015)
Venkataraman, V., Vlachos, I., Turaga, P.K.: Dynamical regularity for action analysis. In: BMVC, pp. 67–1 (2015)
Wang, J., Du, Z., Li, A., Wang, Y.: Atrous temporal convolutional network for video action segmentation. In: ICIP, pp. 1585–1589 (2019)
Xiang, X., Tian, Y., Reiter, A., Hager, G.D., Tran, T.D.: S3D: stacking segmental P3D for action quality assessment. In: ICIP, pp. 928–932 (2018)
Xu, B., Ye, H., Zheng, Y., Wang, H., Luwang, T., Jiang, Y.G.: Dense dilated network for few shot action recognition. In: ICMR, pp. 379–387 (2018)
Zhu, Y., Lan, Z., Newsam, S., Hauptmann, A.G.: Hidden two-stream convolutional networks for action recognition. arXiv preprint arXiv:1704.00389 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, J., Du, Z., Li, A., Wang, Y. (2020). Assessing Action Quality via Attentive Spatio-Temporal Convolutional Networks. In: Peng, Y., et al. Pattern Recognition and Computer Vision. PRCV 2020. Lecture Notes in Computer Science(), vol 12306. Springer, Cham. https://doi.org/10.1007/978-3-030-60639-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-60639-8_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60638-1
Online ISBN: 978-3-030-60639-8
eBook Packages: Computer ScienceComputer Science (R0)