Abstract
Group activity recognition is a challenging task. Group activities usually involves many actors, while only some key actors play a decisive role in group activity recognition. Therefore, extraction of key actors and their interactions is an important problem in group activity recognition. To tackle this problem, we propose a new method based on graph convolutional layers and self-attention mechanisms to extract the Subgraphs of the Actor Relation Graph (SARG). SARG is a scene representation that only contains key actors and their interactions, which is used to enhance the importance of key actors in each group activity. First, the actor relation graph was generated via the appearance and location information of the actors; it was further analyzed by using a graph convolutional layer. Second, we use the graph convolutional layer to generate the self-attention features for each participant and extract the actor relation subgraphs that can ascertain group activities. Finally, we fuse actor relation subgraphs, actor relation graphs and original features to recognize group activities. We evaluate our model over two datasets: the collective activity dataset and the volleyball dataset. SARG has an average improvement of \(4\%\), compared to 8 benchmarks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ramanathan, V., et al.: Detecting events and key actors in multi-person videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. IEEE (2015)
Wang, L., Li, W., Li, W., Van Gool, L.: Appearance-and-relation networks for video classification. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017)
Wu, J., Wang, L., Wang, L., Guo, J., Wu, G.: Learning actor relation graphs for group activity recognition. In: CVPR (2019)
Lee, J., Lee, I., Kang, J.: Self-attention graph pooling. In: 36th International Conference on Machine Learning (2019)
Ibrahim, M., Muralidharan, S., Deng, Z., Vahdat, A., Mori, G.: A hierarchical deep temporal model for group activity recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
Ibrahim, M.S., Mori, G.: Hierarchical relational networks for group activity recognition and retrieval. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 742–758. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_44
Qi, M., Jie, Q., Li, A., Wang, Y., Luo, J., Gool, L.V.: Stagnet: an attentive semantic RNN for group activity recognition. In: European Conference on Computer Vision (2018)
Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436 (2015)
Liu, J., Shahroudy, A., Xu, D., Wang, G.: Spatio-temporal LSTM with trust gates for 3d human action recognition. In: ECCV (2016)
Si, C., Chen, W., Wang, W., Wang, L., Tan, T.: An attention enhanced graph convolutional LSTM network for skeleton-based action recognition. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: NIPS (2017)
Kipf, T.N.: Max Welling. Semi-supervised classification with graph convolutional networks (2016). arXiv preprint: arXiv:1609.02907
Ying, R., You, J., Morris, C., Ren, X., Hamilton, W.L., Leskovec, J.: Hierarchical graph representation learning with differentiable pooling. In: 32th Conference on Neural Information Processing Systems (2018)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv (2014)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: IEEE, pp. 2818–2826 (2016)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. IEEE (2017)
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. Int. Conf. Learn. Represent. (2018) (accepted as poster)
Cheung, M., Shi, J., Jiang, L.Y., Wright, O., Moura, J.: Pooling in Graph Convolutional Neural Networks. IEEE (2020)
Parikh, A.P., Täckström, O., Das, D., Uszkoreit, J.: A decomposable attention model for natural language inference (2016). arXiv preprint: arXiv:1606.01933
Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. In: International Conference on Machine Learning, pp. 7354–7363. PMLR (2019)
Gao, H., Ji, S.: Graph u-nets. In: 36th International Conference on Machine Learning (2019)
Xu, K., Hu, W. Leskovec, J., Jegelka., S.: How powerful are graph neural networks? In: International Conference on Learning Representations (2019)
Choi, W., Shahid, K., Savarese, S.: What are they doing?: collective activity classification using spatio-temporal relationship among people. In: IEEE International Conference on Computer Vision Workshops (2009)
Shu, T., Todorovic, S. Zhu, S.C.: CERN: confidence-energy recurrent network for group activity recognition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Deng, Z., Vahdat, A., Hu, H., Mori, G.: Structure Inference Machines: Recurrent Neural Networks For Analyzing Relations In Group Activity Recognition. IEEE (2016)
Xin, L., Chuah, M.C.: SBGAR: semantics based group activity recognition. In: IEEE International Conference on Computer Vision (2017)
Bagautdinov, T., Alahi, A., Fleuret, F., Fua, P., Savarese, S.: Social scene understanding: end-to-end multi-person action localization and collective activity recognition. IEEE (2016)
Hu, G., Cui, B., He, Y., Yu, S.: Progressive relation learning for group activity recognin. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Duan, Y., Wang, J. (2021). Learning Key Actors and Their Interactions for Group Activity Recognition. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13022. Springer, Cham. https://doi.org/10.1007/978-3-030-88013-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-88013-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88012-5
Online ISBN: 978-3-030-88013-2
eBook Packages: Computer ScienceComputer Science (R0)