Skip to main content
SpringerLink
Log in

Weakly-supervised temporal action localization: a survey

  • Review
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Temporal Action Localization (TAL) is an important task of various computer vision topics such as video understanding, summarization, and analysis. In the real world, the videos are long untrimmed and contain multiple actions, where the temporal boundaries annotations are required in the fully-supervised learning setting for classification and localization tasks. Since the annotation task is costly and time-consuming, the trend is moving toward the weakly-supervised setting, which depends on the video-level labels only without any additional information, and this approach is called weakly-supervised Temporal Action Localization (WTAL). In this survey, we review the concepts, strategies, and techniques related to the WTAL in order to clarify all aspects of the problem and review the state-of-the-art frameworks of WTAL according to their challenges. Furthermore, a comparison of models’ performance and results based on benchmark datasets is presented. Finally, we summarize the future works to allow the researchers to improve the model's performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Lin X, Shou Z, Chang S-F (2019) Towards train-test consistency for semi-supervised temporal action localization, [Online]. Available: http://arxiv.org/abs/1910.11285

  2. Ma F et al (2020) SF-Net: single-frame supervision for temporal action localization, [Online]. Available: http://arxiv.org/abs/2003.06845

  3. Ding X, Wang N, Gao X, Li J, Wang X, and Liu T (2020) Weakly supervised temporal action localization with segment-level labels, 1(c), [Online]. Available: http://arxiv.org/abs/2007.01598

  4. Sun C, Shetty S, Sukthankar R, and Nevatia R (2015) Temporal localization of fine-grained actions in videos by domain transfer from web images. In: MM 2015 - Proc. 2015 ACM Multimed. Conf., pp. 371–380. https://doi.org/10.1145/2733373.2806226

  5. Park J, Lee J, Jeon S, Kim S, and Sohn K (2019) Graph regularization network with semantic affinity for weakly-supervised temporal action localization. In: Proceedings - international conference on image processing, ICIP, 2019:3701–3705. https://doi.org/10.1109/ICIP.2019.8803589

  6. Nguyen P, Han B, Liu T, and Prasad G (2018) Weakly supervised action localization by sparse temporal pooling network. In: Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit, pp. 6752–6761. https://doi.org/10.1109/CVPR.2018.00706

  7. Narayan S, Cholakkal H, Khan FS, and Shao L (2019) 3C-Net: category count and center loss for weakly-supervised action localization. Proc IEEE Int Conf Comput Vis 2019: 8678–8686. https://doi.org/10.1109/ICCV.2019.00877

  8. Wang C, Cai H, Zou Y, and Xiong Y (2021) RGB stream is enough for temporal action detection, [Online]. Available: http://arxiv.org/abs/2107.04362

  9. Alwassel H, Giancola S, and Ghanem B (2020) TSP: temporally-sensitive pretraining of video encoders for localization tasks, [Online]. Available: http://arxiv.org/abs/2011.11479

  10. Nawhal M and Mori G (2021) Activity graph transformer for temporal action localization, [Online]. Available: http://arxiv.org/abs/2101.08540

  11. Alwassel H, Pardo A, Heilbron FC, Thabet A, and Ghanem B (2019) RefineLoc: iterative refinement for weakly-supervised action localization, [Online]. Available: http://arxiv.org/abs/1904.00227

  12. Bojanowski P et al (2014) Weakly supervised action labeling in videos under ordering constraints. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 8693 LNCS, no. Part 5, pp. 628–643. https://doi.org/10.1007/978-3-319-10602-1_41

  13. Huang DA, Fei-Fei L, and Niebles JC (2016) Connectionist temporal modeling for weakly supervised action labelling. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 9908 LNCS, pp. 137–153. https://doi.org/10.1007/978-3-319-46493-0_9

  14. Yang H, He X, Porikli F (2018) One-shot action localization by learning sequence matching network. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2018.00157

    Article  Google Scholar 

  15. Chéron G, Alayrac JB, Laptev I, Schmid C (2018) A flexible model for training action localization with varying levels of supervision. Adv Neural Inf Process Syst 2018:942–953

    Google Scholar 

  16. Xia H, Zhan Y (2020) A survey on temporal action localization. IEEE Access 8:70477–70487. https://doi.org/10.1109/ACCESS.2020.2986861

    Article  Google Scholar 

  17. Zhou ZH (2018) A brief introduction to weakly supervised learning. Natl Sci Rev 5(1):44–53. https://doi.org/10.1093/nsr/nwx106

    Article  Google Scholar 

  18. Kolesnikov A and Lampert CH (2016) Seed, expand and constrain: three principles for weakly-supervised image segmentation. In: Lecture notes in computer science (including subseries Lecture notes in artificial intelligence and lecture notes in bioinformatics), 9908 LNCS, pp. 695–711. https://doi.org/10.1007/978-3-319-46493-0_42

  19. Carbonneau MA, Cheplygina V, Granger E, Gagnon G (2018) Multiple instance learning: a survey of problem characteristics and applications. Pattern Recognit 77:329–353. https://doi.org/10.1016/j.patcog.2017.10.009

    Article  Google Scholar 

  20. Vanwinckelen G, Tragante do VO, Fierens D, Blockeel H (2016) Instance-level accuracy versus bag-level accuracy in multi-instance learning. Data Min Knowl Discov 30(2):313–341. https://doi.org/10.1007/s10618-015-0416-z

    Article  MathSciNet  MATH  Google Scholar 

  21. Wang L, Xiong Y, Lin D, and Van Gool L (2017) UntrimmedNets for weakly supervised action recognition and detection. In: Proc - 30th IEEE Conf Comput Vis Pattern Recognition, CVPR 2017, 2017: 6402–6411. https://doi.org/10.1109/CVPR.2017.678.

  22. Xu Y et al (2019) Segregated temporal assembly recurrent networks for weakly supervised multiple action detection. Proc AAAI Conf Artif Intell 33:9070–9078. https://doi.org/10.1609/aaai.v33i01.33019070

    Article  Google Scholar 

  23. Lee P, Uh Y, and Byun H (2019) Background suppression network for weakly-supervised temporal action localization. https://doi.org/10.1609/aaai.v34i07.6793

  24. Paul S, Roy S, and Roy-Chowdhury AK (2018) W-TALC: weakly-supervised temporal activity localization and classification. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), 11208 LNCS, pp. 588–607. https://doi.org/10.1007/978-3-030-01225-0_35

  25. Lee P, Wang J, Lu Y, and Byun H (2020) Background modeling via uncertainty estimation for weakly-supervised action localization. pp. 1–12, [Online]. Available: http://arxiv.org/abs/2006.07006

  26. Rashid M, Kjellstrom H, and Lee YJ (2020) Action graphs: weakly-supervised action localization with graph convolution networks. In: Proceedings - 2020 IEEE winter conference on applications of computer vision, WACV 2020, pp. 604–613. https://doi.org/10.1109/WACV45572.2020.9093404

  27. Shi B, Dai Q, Mu Y, and Wang J (2020) Weakly-supervised action localization by generative attention modelling. pp. 1006–1016. https://doi.org/10.1109/cvpr42600.2020.00109

  28. Schindler K and Van Gool L (2008) Action snippets: How many frames does human action recognition require?. In: 26th IEEE Conf Comput Vis Pattern Recognition, CVPR. https://doi.org/10.1109/CVPR.2008.4587730

  29. Liu D, Jiang T, and Wang Y (2019) Completeness modeling and context separation for weakly supervised temporal action localization. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, 2019:1298–1307. https://doi.org/10.1109/CVPR.2019.00139

  30. Singh KK and Lee YJ (2017) Hide-and-seek: forcing a network to be meticulous for weakly-supervised object and action localization. In: Proc IEEE Int Conf Comput Vis, 2017: 3544–3553. https://doi.org/10.1109/ICCV.2017.381

  31. Shou Z, Gao H, Zhang L, Miyazawa K, and Chang SF (2018) AutoLoc: weakly-supervised temporal action localization in untrimmed videos. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 11220 LNCS, pp. 162–179. https://doi.org/10.1007/978-3-030-01270-0_10

  32. Liu Z et al (2019) Weakly supervised temporal action localization through contrast based evaluation networks. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2019.00400

    Article  Google Scholar 

  33. Zeng R, Gan C, Chen P, Huang W, Wu Q, Tan M (2019) Breaking winner-takes-all: iterative-winners-out networks for weakly supervised temporal action localization. IEEE Trans Image Process 28(12):5797–5808. https://doi.org/10.1109/TIP.2019.2922108

    Article  MathSciNet  MATH  Google Scholar 

  34. Su H, Zhao X, and Lin T (2019) Cascaded pyramid mining network for weakly supervised temporal action localization. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 11362 LNCS, pp. 558–574. https://doi.org/10.1007/978-3-030-20890-5_36

  35. Su H, Zhao X, Lin T, and Fei H (2018) Weakly supervised temporal action detection with shot-based temporal pooling network. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), 11304 LNCS, pp. 426–436. https://doi.org/10.1007/978-3-030-04212-7_37

  36. Russakovsky O et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  37. Kay W et al (2017) The kinetics human action video dataset, [Online]. Available: http://arxiv.org/abs/1705.06950

  38. Zach C, Pock T, and Bischof H (2007) A duality based approach for realtime TV-L1 optical flow. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 4713 LNCS, pp. 214–223. https://doi.org/10.1007/978-3-540-74936-3_22

  39. Soomro K, Zamir AR, and Shah M (2012) UCF101: a dataset of 101 human actions classes from videos in the wild, [Online]. Available: http://arxiv.org/abs/1212.0402

  40. Kuehne H, Jhuang H, Garrote E, Poggio T, Serre T (2011) HMDB: a large video database for human motion recognition. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2011.6126543

    Article  Google Scholar 

  41. Simonyan K, Zisserman A (2014) Two-stream convolutional networks for action recognition in videos. Adv Neural Inf Process Syst 1(January):568–576

    Google Scholar 

  42. Wang L et al (2016) Temporal segment networks: Towards good practices for deep action recognition. Lect Notes Comput Sci (including Subser. Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 9912 LNCS, pp. 20–36. https://doi.org/10.1007/978-3-319-46484-8_2

  43. Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Li FF (2014) Large-scale video classification with convolutional neural networks. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2014.223

    Article  Google Scholar 

  44. Feichtenhofer C, Pinz A, Zisserman A (2016) Convolutional two-stream network fusion for video action recognition. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2016.213

    Article  Google Scholar 

  45. Dai X, Singh B, Zhang G, Davis LS, Chen YQ (2017) Temporal context network for activity localization in videos. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2017.610

    Article  Google Scholar 

  46. Zhong JX, Li N, Kong W, Zhang T, Li TH, and Li G (2018) Step-by-step erasion, one-by-one collection: a weakly supervised temporal action detector. In: MM 2018 - Proceedings of the 2018 ACM multimedia conference, no. 2014, pp. 35–44. https://doi.org/10.1145/3240508.3240511

  47. Huang L, Huang Y, Ouyang W, and Wang L (2020) Relational prototypical network for weakly supervised temporal action localization. Aaai

  48. Carreira J and Zisserman A (2017) Quo Vadis, action recognition? A new model and the kinetics dataset. In: Proc. - 30th IEEE Conf Comput Vis Pattern Recognition, CVPR 2017, 2017: 4724–4733. https://doi.org/10.1109/CVPR.2017.502

  49. Ioffe S and Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: 32nd Int Conf Mach Learn. ICML 2015, 1:448–456

  50. Nguyen P, Ramanan D, Fowlkes C (2019) Weakly-supervised action localization with background modeling. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2019.00560

    Article  Google Scholar 

  51. Kang Z, Wang L, Liu Z, Zhang Q, Zheng N (2019) Extracting action sensitive features to facilitate weakly-supervised action localization. IFIP Adv Inform Commun Technol. https://doi.org/10.1007/978-3-030-19823-7_15

    Article  Google Scholar 

  52. Zhai Y, Wang L, Liu Z, Zhang Q, Hua G, Zheng N (2019) Action coherence network for weakly supervised temporal action localization. Proc - Int Conf Image Process. https://doi.org/10.1109/ICIP.2019.8803447

    Article  Google Scholar 

  53. Zhang C et al (2019) Adversarial seeded sequence growing for weakly-supervised temporal action localization. In: MM 2019 - Proc 27th ACM Int Conf Multimed, pp. 738–746. https://doi.org/10.1145/3343031.3351044

  54. Yuan Y, Lyu Y, Shen X, Tsang IW, and Yeung DY (2019) Marginalized average attentional network for weakly-supervised learning. In: 7th Int Conf Learn. Represent. ICLR 2019, pp. 1–19

  55. Min K and Corso JJ (2020) Adversarial background-aware loss for weakly-supervised temporal activity localization. ECCV 2020, [Online]. Available: http://arxiv.org/abs/2007.06643

  56. Nair V and Hinton GE (2010) Rectified linear units improve restricted Boltzmann machines. In: ICML 2010 - Proceedings, 27th Int Conf Mach Learn, pp. 807–814

  57. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2016.319

    Article  Google Scholar 

  58. Narayan S, Cholakkal H, Hayat M, Khan FS, Yang MH, and Shao L (2020) D2-Net: weakly-supervised action localization via discriminative embeddings and denoised activations. arXiv, no. December

  59. Islam A and Radke RJ (2020) Weakly supervised temporal action localization using deep metric learning. In: Proceedings - 2020 IEEE winter conference on applications of computer vision, WACV 2020, pp. 536–545. https://doi.org/10.1109/WACV45572.2020.9093620

  60. Idrees H et al (2017) The THUMOS challenge on action recognition for videos ‘in the wild.’ Comput Vis Image Underst 155:1–23. https://doi.org/10.1016/j.cviu.2016.10.018

    Article  Google Scholar 

  61. Heilbron FC, Escorcia V, Ghanem B, Niebles JC (2015) ActivityNet: a large-scale video benchmark for human activity understanding. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR.2015.7298698

    Article  Google Scholar 

  62. Sigurdsson GA, Varol G, Wang X, Farhadi A, Laptev I, and Gupta A (2016) Hollywood in homes: crowdsourcing data collection for activity understanding. Lect Notes Comput Sci (including Subser. Lect Notes Artif Intell Lect Notes Bioinformatics), vol. 9905 LNCS, pp. 510–526. https://doi.org/10.1007/978-3-319-46448-0_31

  63. Zhao H, Torralba A, Torresani L, Yan Z (2019) HACS: Human action clips and segments dataset for recognition and temporal localization. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2019.00876

    Article  Google Scholar 

  64. Huang Z, Wang X, Wang JJ, Liu W, and Wang JJ (2018) Weakly-supervised semantic segmentation network with deep seeded region growing. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp. 7014–7023. https://doi.org/10.1109/CVPR.2018.00733

  65. Islam A, Long C, and Radke RJ (2021) A hybrid attention mechanism for weakly-supervised temporal action localization, no. Mil, [Online]. Available: http://arxiv.org/abs/2101.00545

  66. Ge Y, Qin X, Yang D, Jagersand M (2021) Deep snippet selective network for weakly supervised temporal action localization. Pattern Recognit 110:107686. https://doi.org/10.1016/j.patcog.2020.107686

    Article  Google Scholar 

  67. Yu T, Ren Z, Li Y, Yan E, Xu N, Yuan J (2019) Temporal structure mining for weakly supervised action detection. Proc IEEE Int Conf Comput Vis. https://doi.org/10.1109/ICCV.2019.00562

    Article  Google Scholar 

  68. Hendrycks D and Gimpel K (2016) A baseline for detecting misclassified and out-of-distribution examples in neural networks. 5th Int Conf Learn Represent ICLR 2017 – Conf Track Proc, pp. 1–12. [Online]. Available: http://arxiv.org/abs/1610.02136

  69. Hou R, Sukthankar R, and Shah M (2017) Real-time temporal action localization in untrimmed videos by sub-action discovery. Br Mach Vis Conf, BMVC . https://doi.org/10.5244/c.31.91

  70. Heidarivincheh F, Mirmehdi M, and Damen D (2019) Weakly-supervised completion moment detection using temporal attention. Proc. - 2019 Int Conf Comput Vis Work. ICCVW 2019, pp. 1188–1196. https://doi.org/10.1109/ICCVW.2019.00150

  71. Luo Z et al (2020) Weakly-supervised action localization with expectation-maximization multi-instance learning. Lect. Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), 12374 LNCS, no. Mil, pp. 729–745. https://doi.org/10.1007/978-3-030-58526-6_43

  72. Zhang XY, Li C, Shi H, Zhu X, Li P, Dong J (2020) AdapNet: adaptability decomposing encoder-decoder network for weakly supervised action recognition and localization. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2019.2962815

    Article  Google Scholar 

  73. Snell J, Swersky K, Zemel R (2017) Prototypical networks for few-shot learning. Adv Neural Inform Process Syst 2017:4078–4088

    Google Scholar 

  74. Kingma DP and J. L. Ba (2015) Adam: a method for stochastic optimization, 3rd Int Conf Learn Represent. ICLR 2015 - Conf Track Proc, pp. 1–15

  75. Zhao Y, Xiong Y, Wang L, Wu Z, Tang X, Lin D (2020) Temporal action detection with structured segment networks. Int J Comput Vis 128(1):74–95. https://doi.org/10.1007/s11263-019-01211-2

    Article  MathSciNet  MATH  Google Scholar 

  76. Defferrard M, Bresson X, and Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. Adv Neural Inform Process Syst, no. Nips, pp. 3844–3852

  77. Pang J, Cheung G (2017) Graph laplacian regularization for image denoising: analysis in the continuous domain. IEEE Trans Image Process 26(4):1770–1785. https://doi.org/10.1109/TIP.2017.2651400

    Article  MathSciNet  MATH  Google Scholar 

  78. Zhai Y, Wang L, Tang W, Zhang Q, and Yuan J (2020) Two-stream consensus network for weakly-supervised temporal action localization. In: Proc Eur. Conf Comput Vis, no. Mil, pp. 1–17

  79. Gong G, Wang X, Mu Y, Tian Q (2020) Learning temporal co-attention models for unsupervised video action localization. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit. https://doi.org/10.1109/CVPR42600.2020.00984

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported in part by the Ministry of Higher Education Malaysia for Fundamental Research Grant Scheme with Project Code: FRGS/1/2019/ICT02/USM/02/1.

Author information

Authors and Affiliations

  1. School of Computer Sciences, Universiti Sains Malaysia, 11800, Pulau Pinang, Malaysia

    AbdulRahman Baraka & Mohd Halim Mohd Noor

  2. Al-Quds Open University, Ramallah, Palestine

    AbdulRahman Baraka

Authors
  1. AbdulRahman Baraka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohd Halim Mohd Noor
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohd Halim Mohd Noor.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Baraka, A., Mohd Noor, M.H. Weakly-supervised temporal action localization: a survey. Neural Comput & Applic 34, 8479–8499 (2022). https://doi.org/10.1007/s00521-022-07102-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07102-x

Keywords

Search

Navigation