Skip to main content
SpringerLink
Log in

SGLP-Net: Sparse Graph Label Propagation Network for Weakly-Supervised Temporal Action Localization

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14451))

Included in the following conference series:

  • 535 Accesses

Abstract

The present weakly-supervised methods for Temporal Action Localization are primarily responsible for capturing the temporal context. However, these approaches have limitations in capturing semantic context, resulting in the risk of ignoring snippets that are far apart but sharing the same action categories. To address this issue, we propose an action label propagation network utilizing sparse graph networks to effectively explore both temporal and semantic information in videos. The proposed SGLP-Net comprises two key components. One is the multi-scale temporal feature embedding module, a novel method that extracts both local and global temporal features of the videos during the initial stage using CNN and self-attention and serves as a generic module. The other is an action label propagation mechanism, which uses graph networks for feature aggregation and label propagation. To avoid the issue of excessive feature completeness, we optimize training using sparse graph convolutions. Extensive experiments are conducted on THUMOS14 and ActivityNet1.3 benchmarks, among which advanced results demonstrate the superiority of the proposed method. Code can be found at https://github.com/xyao-wu/SGLP-Net.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724–4733 (2017). https://doi.org/10.1109/CVPR.2017.502

  2. Chao, Y.W., Vijayanarasimhan, S., Seybold, B., Ross, D.A., Deng, J., Sukthankar, R.: Rethinking the faster R-CNN architecture for temporal action localization. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1130–1139 (2018). https://doi.org/10.1109/CVPR.2018.00124

  3. Chen, J., Ma, T., Xiao, C.: FastGCN: fast learning with graph convolutional networks via importance sampling. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=rytstxWAW

  4. Chen, M., Gao, J., Yang, S., Xu, C.: Dual-evidential learning for weakly-supervised temporal action localization. In: Avidan, S., Brostow, G., Cissa, M., Farinella, G.M., Hassner, T. (eds.) European Conference on Computer Vision, vol. 13664, pp. 192–208. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19772-7_12

  5. Chen, T., Li, B., Tao, Y., Wang, Y., Zhu, Y.: Class-incremental learning with multiscale distillation for weakly supervised temporal action localization. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds.) ICONIP 2022. LNCS, vol. 13623, pp. 367–378. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-19772-7_12

    Chapter  Google Scholar 

  6. Ciptadi, A., Goodwin, M.S., Rehg, J.M.: Movement pattern histogram for action recognition and retrieval. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 695–710. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10605-2_45

    Chapter  Google Scholar 

  7. Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89(1), 31–71 (1997). https://doi.org/10.1016/S0004-3702(96)00034-3. https://www.sciencedirect.com/science/article/pii/S0004370296000343

  8. Douze, M., Szlam, A., Hariharan, B., Jégou, H.: Low-shot learning with large-scale diffusion. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3349–3358 (2018). https://doi.org/10.1109/CVPR.2018.00353

  9. He, B., Yang, X., Kang, L., Cheng, Z., Zhou, X., Shrivastava, A.: ASM-Loc: action-aware segment modeling for weakly-supervised temporal action localization. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13915–13925 (2022). https://doi.org/10.1109/CVPR52688.2022.01355

  10. Heilbron, F.C., Escorcia, V., Ghanem, B., Niebles, J.C.: Activitynet: a large-scale video benchmark for human activity understanding. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961–970 (2015). https://doi.org/10.1109/CVPR.2015.7298698

  11. Idrees, H., et al.: The thumos challenge on action recognition for videos “in the wild”. Comput. Vision Image Underst. 155, 1–23 (2017). https://doi.org/10.1016/j.cviu.2016.10.018. https://www.sciencedirect.com/science/article/pii/S1077314216301710

  12. Kay, W., et al.: The kinetics human action video dataset (2017)

    Google Scholar 

  13. Lee, P., Uh, Y., Byun, H.: Background suppression network for weakly-supervised temporal action localization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 11320–11327 (2020). https://doi.org/10.1609/aaai.v34i07.6793

  14. Li, L., Kong, T., Sun, F., Liu, H.: Deep point-wise prediction for action temporal proposal. In: Gedeon, T., Wong, K.W., Lee, M. (eds.) ICONIP 2019. LNCS, vol. 11955, pp. 475–487. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-36718-3_40

    Chapter  Google Scholar 

  15. Liu, D., Jiang, T., Wang, Y.: Completeness modeling and context separation for weakly supervised temporal action localization. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1298–1307 (2019). https://doi.org/10.1109/CVPR.2019.00139

  16. Luo, W., et al.: Action unit memory network for weakly supervised temporal action localization. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9964–9974 (2021). https://doi.org/10.1109/CVPR46437.2021.00984

  17. Moniruzzaman, M., Yin, Z.: Feature weakening, contextualization, and discrimination for weakly supervised temporal action localization. IEEE Trans. Multimedia, 1–13 (2023). https://doi.org/10.1109/TMM.2023.3263965

  18. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML 2010, pp. 807–814. Omnipress, Madison (2010)

    Google Scholar 

  19. Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization by sparse temporal pooling network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017). https://doi.org/10.1109/CVPR.2018.00706

  20. Qu, S., Chen, G., Li, Z., Zhang, L., Lu, F., Knoll, A.: Acm-net: action context modeling network for weakly-supervised temporal action localization. arXiv preprint arXiv:2104.02967 (2021)

  21. Raghavan, U.N., Albert, R., Kumara, S.: Near linear time algorithm to detect community structures in large-scale networks. Phys. Rev. E 76, 036106 (2007). https://doi.org/10.1103/PhysRevE.76.036106

  22. Shi, B., Dai, Q., Mu, Y., Wang, J.: Weakly-supervised action localization by generative attention modeling. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1006–1016 (2020). https://doi.org/10.1109/CVPR42600.2020.00109

  23. Shi, D., Zhong, Y., Cao, Q., Ma, L., Li, J., Tao, D.: Tridet: temporal action detection with relative boundary modeling. arXiv:2303.07347 (2023)

  24. Su, H., Zhao, X., Lin, T., Fei, H.: Weakly supervised temporal action detection with shot-based temporal pooling network. In: Cheng, L., Leung, A.C.S., Ozawa, S. (eds.) ICONIP 2018. LNCS, vol. 11304, pp. 426–436. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04212-7_37

    Chapter  Google Scholar 

  25. Yang, W., Zhang, T., Yu, X., Qi, T., Zhang, Y., Wu, F.: Uncertainty guided collaborative training for weakly supervised temporal action detection. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 53–63 (2021). https://doi.org/10.1109/CVPR46437.2021.00012

  26. Yang, Z., Qin, J., Huang, D.: ACGNet: action complement graph network for weakly-supervised temporal action localization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 3090–3098 (2022)

    Google Scholar 

  27. Zeng, R., et al.: Graph convolutional networks for temporal action localization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7093–7102 (2019). https://doi.org/10.1109/ICCV.2019.00719

  28. Zhai, Y., Wang, L., Tang, W., Zhang, Q., Yuan, J., Hua, G.: Two-stream consensus network for weakly-supervised temporal action localization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 37–54. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_3

    Chapter  Google Scholar 

  29. Zhang, C.L., Wu, J., Li, Y.: ActionFormer: localizing moments of actions with transformers. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13664, pp. 492–510. Springer Nature Switzerland, Cham (2022). https://doi.org/10.1007/978-3-031-19772-7_29

    Chapter  Google Scholar 

  30. Zhao, C., Thabet, A., Ghanem, B.: Video self-stitching graph network for temporal action localization. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13638–13647 (2021). https://doi.org/10.1109/ICCV48922.2021.01340

Download references

Author information

Authors and Affiliations

  1. Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an, China

    Xiaoyao Wu & Yonghong Song

Authors
  1. Xiaoyao Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yonghong Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yonghong Song .

Editor information

Editors and Affiliations

  1. Central South University, Changsha, China

    Biao Luo

  2. Chinese Academy of Sciences, Beijing, China

    Long Cheng

  3. Zhejiang University, Hangzhou, China

    Zheng-Guang Wu

  4. Guangdong University of Technology, Guangzhou, China

    Hongyi Li

  5. UNSW Sydney, Sydney, NSW, Australia

    Chaojie Li

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wu, X., Song, Y. (2024). SGLP-Net: Sparse Graph Label Propagation Network for Weakly-Supervised Temporal Action Localization. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14451. Springer, Singapore. https://doi.org/10.1007/978-981-99-8073-4_12

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8073-4_12

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8072-7

  • Online ISBN: 978-981-99-8073-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation