Skip to main content
SpringerLink
Log in

Weakly-Supervised Temporal Action Localization with Regional Similarity Consistency

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13833))

Included in the following conference series:

  • 1367 Accesses

Abstract

The weakly-supervised temporal action localization task aims to train a model that can accurately locate each action instance in the video using only video-level class labels. The existing methods take into account the information of different modalities (primarily RGB and Flow), and present numerous multi-modal complementary methods. RGB features are obtained by calculating appearance information, which are easy to be disrupted by the background. On the contrary, Flow features are obtained by calculating motion information, which are usually less disrupted by the background. Based on this phenomenon, we propose a Regional Similarity Consistency (RSC) constraint between these two modalities to suppress the disturbance of background in RGB features. Specifically, we calculate the regional similarity matrices of RGB and Flow features, and impose the consistency constraint through \(L_2\) loss. To verify the effectiveness of our method, we integrate the proposed RSC constraint into three recent methods. The comprehensive experimental results show that the proposed RSC constraint can boost the performance of these methods, and achieve the state-of-the-art results on the widely-used THUMOS14 and ActivityNet1.2 datasets.

H. Ren, H. Ren—Contributed equally to this work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: ActivityNet: a large-scale video benchmark for human activity understanding. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 961–970 (2015)

    Google Scholar 

  2. Cao, M., Zhang, C., Chen, L., Shou, M.Z., Zou, Y.: Deep motion prior for weakly-supervised temporal action localization. IEEE Trans. Image Process. 31, 5203–5213 (2022)

    Article  Google Scholar 

  3. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 6299–6308 (2017)

    Google Scholar 

  4. Chao, Y.W., Vijayanarasimhan, S., Seybold, B., Ross, D.A., Deng, J., Sukthankar, R.: Rethinking the faster R-CNN architecture for temporal action localization. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 1130–1139 (2018)

    Google Scholar 

  5. Gabeur, V., Sun, C., Alahari, K., Schmid, C.: Multi-modal transformer for video retrieval. In: European Conference on Computer Vision, pp. 214–229 (2020)

    Google Scholar 

  6. He, B., Yang, X., Kang, L., Cheng, Z., Zhou, X., Shrivastava, A.: ASM-LOC: action-aware segment modeling for weakly-supervised temporal action localization. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 13925–13935 (2022)

    Google Scholar 

  7. Hong, F.T., Feng, J.C., Xu, D., Shan, Y., Zheng, W.S.: Cross-modal consensus network for weakly supervised temporal action localization. In: ACM International Conference on Multimedia, pp. 1591–1599 (2021)

    Google Scholar 

  8. Huang, L., Wang, L., Li, H.: Weakly supervised temporal action localization via representative snippet knowledge propagation. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 3272–3281 (2022)

    Google Scholar 

  9. Idrees, H., et al.: The thumos challenge on action recognition for videos in the wild. Comput. Vis. Image Underst. 155, 1–23 (2017)

    Article  Google Scholar 

  10. Islam, A., Long, C., Radke, R.: A hybrid attention mechanism for weakly-supervised temporal action localization. In: AAAI Conference on Artificial Intelligence, vol. 35, pp. 1637–1645 (2021)

    Google Scholar 

  11. Ji, Y., Jia, X., Lu, H., Ruan, X.: Weakly-supervised temporal action localization via cross-stream collaborative learning. In: ACM International Conference on Multimedia, pp. 853–861 (2021)

    Google Scholar 

  12. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)

    Google Scholar 

  13. Lee, J.T., Jain, M., Park, H., Yun, S.: Cross-attentional audio-visual fusion for weakly-supervised action localization. In: International Conference on Learning Representations (2020)

    Google Scholar 

  14. Lee, P., Uh, Y., Byun, H.: Background suppression network for weakly-supervised temporal action localization. In: AAAI Conference on Artificial Intelligence, vol. 34, pp. 11320–11327 (2020)

    Google Scholar 

  15. Lerman, P.: Fitting segmented regression models by grid search. J. Royal Stat. Soc. 29(1), 77–84 (1980)

    MathSciNet  Google Scholar 

  16. Liu, D., Jiang, T., Wang, Y.: Completeness modeling and context separation for weakly supervised temporal action localization. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 1298–1307 (2019)

    Google Scholar 

  17. Liu, Z., Wang, L., Zhang, Q., Tang, W., Yuan, J., Zheng, N., Hua, G.: ACSNet: action-context separation network for weakly supervised temporal action localization. In: AAAI Conference on Artificial Intelligence. vol. 35, pp. 2233–2241 (2021)

    Google Scholar 

  18. Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., Mei, T.: Gaussian temporal awareness networks for action localization. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 344–353 (2019)

    Google Scholar 

  19. Luo, Z., et al.: Weakly-supervised action localization with expectation-maximization multi-instance learning. In: European Conference on Computer Vision, pp. 729–745 (2020)

    Google Scholar 

  20. Ma, F., et al.: SF-net: single-frame supervision for temporal action localization. In: European Conference on Computer Vision, pp. 420–437 (2020)

    Google Scholar 

  21. Narayan, S., Cholakkal, H., Khan, F.S., Shao, L.: 3C-Net: category count and center loss for weakly-supervised action localization. In: International Conference on Computer Vision, pp. 8679–8687 (2019)

    Google Scholar 

  22. Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization by sparse temporal pooling network. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 6752–6761 (2018)

    Google Scholar 

  23. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Annual Conference on Neural Information Processing Systems, pp. 8026–8037 (2019)

    Google Scholar 

  24. Paul, S., Roy, S., Roy-Chowdhury, A.K.: W-TALC: weakly-supervised temporal activity localization and classification. In: European Conference on Computer Vision, pp. 563–579 (2018)

    Google Scholar 

  25. Shou, Z., Wang, D., Chang, S.F.: Temporal action localization in untrimmed videos via multi-stage CNNs. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 1049–1058 (2016)

    Google Scholar 

  26. Snidaro, L., Micheloni, C., Chiavedale, C.: Video security for ambient intelligence. IEEE Trans. Syst. Man Cybern. 35(1), 133–144 (2004)

    Article  Google Scholar 

  27. Wang, L., Xiong, Y., Lin, D., Van Gool, L.: UntrimmedNets for weakly supervised action recognition and detection. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 4325–4334 (2017)

    Google Scholar 

  28. Xu, Y., et al.: Segregated temporal assembly recurrent networks for weakly supervised multiple action detection. In: AAAI Conference on Artificial Intelligence, vol. 33, pp. 9070–9078 (2019)

    Google Scholar 

  29. Yang, W., Zhang, T., Yu, X., Qi, T., Zhang, Y., Wu, F.: Uncertainty guided collaborative training for weakly supervised temporal action detection. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 53–63 (2021)

    Google Scholar 

  30. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime tv-l 1 optical flow. In: Joint Pattern Recognition Symposium, pp. 214–223 (2007)

    Google Scholar 

  31. Zhai, Y., Wang, L., Tang, W., Zhang, Q., Yuan, J., Hua, G.: Two-stream consensus network for weakly-supervised temporal action localization. In: European Conference on Computer Vision, pp. 37–54 (2020)

    Google Scholar 

  32. Zhang, C., Cao, M., Yang, D., Chen, J., Zou, Y.: CoLA: weakly-supervised temporal action localization with snippet contrastive learning. In: IEEE Computer Vision and Pattern Recognition Conference, pp. 16010–16019 (2021)

    Google Scholar 

  33. Zhang, C., et al.: Adversarial seeded sequence growing for weakly-supervised temporal action localization. In: ACM International Conference on Multimedia, pp. 738–746 (2019)

    Google Scholar 

  34. Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., Lin, D.: Temporal action detection with structured segment networks. In: International Conference on Computer Vision, pp. 2914–2923 (2017)

    Google Scholar 

Download references

Acknowledgments

This work was supported by Shanghai Municipal Science and Technology Commission with Grant No. 22dz1204900.

Author information

Authors and Affiliations

  1. Shanghai Key Lab of Intelligent Information Processing, School of Computer Science, Fudan University, Shanghai, China

    Haoran Ren, Hao Ren, Hong Lu & Cheng Jin

Authors
  1. Haoran Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hong Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cheng Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong Lu .

Editor information

Editors and Affiliations

  1. University of Bergen, Bergen, Norway

    Duc-Tien Dang-Nguyen

  2. Dublin City University, Dublin, Ireland

    Cathal Gurrin

  3. Radboud University Nijmegen, Nijmegen, The Netherlands

    Martha Larson

  4. Dublin City University, Dublin, Ireland

    Alan F. Smeaton

  5. University of Amsterdam, Amsterdam, The Netherlands

    Stevan Rudinac

  6. National Institute of Information and Communications Technology, Tokyo, Japan

    Minh-Son Dao

  7. Department of Information Science and Media Studies, University of Bergen, Bergen, Norway

    Christoph Trattner

  8. La Trobe University, Melbourne, VIC, Australia

    Phoebe Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ren, H., Ren, H., Lu, H., Jin, C. (2023). Weakly-Supervised Temporal Action Localization with Regional Similarity Consistency. In: Dang-Nguyen, DT., et al. MultiMedia Modeling. MMM 2023. Lecture Notes in Computer Science, vol 13833. Springer, Cham. https://doi.org/10.1007/978-3-031-27077-2_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-27077-2_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-27076-5

  • Online ISBN: 978-3-031-27077-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation