Abstract
Weakly-supervised temporal action localization seeks to localize temporal boundaries of actions while concurrently identifying their categories using only video-level category labels during training. Among the existing methods, the modal cooperation methods have achieved great success by providing pseudo supervision signals to RGB and Flow features. However, most of these methods ignore the cross-correlation between modal characteristics which can help them learn better features. By considering the cross-correlation, we propose a novel multi-head cross-modal attention mechanism to explicitly model the cross-correlation of modal features. The proposed method collaboratively enhances RGB and Flow features through a cross-correlation matrix. In this way, the enhanced features for each modality encode the inter-modal information, while preserving the exclusive and meaningful intra-modal characteristics. Experimental results on three recent methods demonstrate that the proposed Multi-head Cross-modal Attention (MCA) mechanism can significantly improve the performance of these methods, and even achieve state-of-the-art results on the THUMOS14 and ActivityNet1.2 datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: ActivityNet: a large-scale video benchmark for human activity understanding. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 961–970 (2015)
Carreira, J., Zisserman, A.: Quo Vadis, action recognition? A new model and the kinetics dataset. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
Chao, Y.W., Vijayanarasimhan, S., Seybold, B., Ross, D.A., Deng, J., Sukthankar, R.: Rethinking the faster R-CNN architecture for temporal action localization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1130–1139 (2018)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2020)
Gabeur, V., Sun, C., Alahari, K., Schmid, C.: Multi-modal transformer for video retrieval. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 214–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_13
He, B., Yang, X., Kang, L., Cheng, Z., Zhou, X., Shrivastava, A.: ASM-Loc: action-aware segment modeling for weakly-supervised temporal action localization. In: IEEE Conference on Computer Vision and Pattern Recognition (2022)
Hong, F.T., Feng, J.C., Xu, D., Shan, Y., Zheng, W.S.: Cross-modal consensus network for weakly supervised temporal action localization. In: ACM International Conference on Multimedia, pp. 1591–1599 (2021)
Huang, L., Wang, L., Li, H.: Weakly supervised temporal action localization via representative snippet knowledge propagation. In: IEEE Conference on Computer Vision and Pattern Recognition (2022)
Idrees, H., et al.: The Thumos challenge on action recognition for videos “in the wild.’’. Comput. Vis. Image Underst. 155, 1–23 (2017)
Islam, A., Long, C., Radke, R.: A hybrid attention mechanism for weakly-supervised temporal action localization. In: AAAI Conference on Artificial Intelligence, vol. 35, pp. 1637–1645 (2021)
Ji, Y., Jia, X., Lu, H., Ruan, X.: Weakly-supervised temporal action localization via cross-stream collaborative learning. In: ACM International Conference on Multimedia, pp. 853–861 (2021)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)
Lee, J.T., Jain, M., Park, H., Yun, S.: Cross-attentional audio-visual fusion for weakly-supervised action localization. In: International Conference on Learning Representations (2020)
Lee, P., Uh, Y., Byun, H.: Background suppression network for weakly-supervised temporal action localization. In: AAAI Conference on Artificial Intelligence, vol. 34, pp. 11320–11327 (2020)
Liu, D., Jiang, T., Wang, Y.: Completeness modeling and context separation for weakly supervised temporal action localization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1298–1307 (2019)
Liu, Z., et al.: ACSNet: action-context separation network for weakly supervised temporal action localization. In: AAAI Conference on Artificial Intelligence, vol. 35, pp. 2233–2241 (2021)
Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., Mei, T.: Gaussian temporal awareness networks for action localization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 344–353 (2019)
Luo, Z., et al.: Weakly-supervised action localization with expectation-maximization multi-instance learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 729–745. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_43
Ma, F., et al.: SF-Net: single-frame supervision for temporal action localization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 420–437. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_25
Narayan, S., Cholakkal, H., Khan, F.S., Shao, L.: 3C-Net: category count and center loss for weakly-supervised action localization. In: International Conference on Computer Vision, pp. 8679–8687 (2019)
Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization by sparse temporal pooling network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6752–6761 (2018)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Annual Conference on Neural Information Processing Systems, pp. 8026–8037 (2019)
Paul, S., Roy, S., Roy-Chowdhury, A.K.: W-TALC: weakly-supervised temporal activity localization and classification. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 588–607. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_35
Shi, B., Dai, Q., Mu, Y., Wang, J.: Weakly-supervised action localization by generative attention modeling. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1009–1019 (2020)
Shou, Z., Wang, D., Chang, S.F.: Temporal action localization in untrimmed videos via multi-stage CNNs. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1049–1058 (2016)
Wang, L., Xiong, Y., Lin, D., Van Gool, L.: Untrimmednets for weakly supervised action recognition and detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4325–4334 (2017)
Xu, Y., et al.: Segregated temporal assembly recurrent networks for weakly supervised multiple action detection. In: AAAI Conference on Artificial Intelligence, vol. 33, pp. 9070–9078 (2019)
Yang, W., Zhang, T., Yu, X., Qi, T., Zhang, Y., Wu, F.: Uncertainty guided collaborative training for weakly supervised temporal action detection. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 53–63 (2021)
Yu, Z., Yu, J., Cui, Y., Tao, D., Tian, Q.: Deep modular co-attention networks for visual question answering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6281–6290 (2019)
Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L 1 optical flow. In: Joint Pattern Recognition Symposium, pp. 214–223 (2007)
Zhang, C., Cao, M., Yang, D., Chen, J., Zou, Y.: CoLA: weakly-supervised temporal action localization with snippet contrastive learning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 16010–16019 (2021)
Zhang, C., et al.: Adversarial seeded sequence growing for weakly-supervised temporal action localization. In: ACM International Conference on Multimedia, pp. 738–746 (2019)
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., Lin, D.: Temporal action detection with structured segment networks. In: International Conference on Computer Vision, pp. 2914–2923 (2017)
Acknowledgments
This work was supported by National Natural Science Foundation of China under Grant No. 61732004.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ren, H., Ren, H., Ran, W., Lu, H., Jin, C. (2022). Weakly-Supervised Temporal Action Localization with Multi-Head Cross-Modal Attention. In: Khanna, S., Cao, J., Bai, Q., Xu, G. (eds) PRICAI 2022: Trends in Artificial Intelligence. PRICAI 2022. Lecture Notes in Computer Science, vol 13631. Springer, Cham. https://doi.org/10.1007/978-3-031-20868-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-20868-3_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20867-6
Online ISBN: 978-3-031-20868-3
eBook Packages: Computer ScienceComputer Science (R0)