Skip to main content

Class-Incremental Learning with Multiscale Distillation for Weakly Supervised Temporal Action Localization

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2022)

Abstract

Despite recent works having made great progress in weakly supervised temporal action localization (WTAL), they still suffer from catastrophic forgetting. When only new-class videos can be utilized to update these models, their performance in old classes diminishes drastically. Even while some class-incremental learning methods are presented to assist models in continuously learning new-class knowledge, most of them focus on image classification but pay little attention to WTAL. To fill this gap, we propose a novel class-incremental learning method with multiscale distillation, which mines two separate scales of old-class information in incoming videos for updating the model. Precisely, we calculate class activation sequences (CAS) with frame-level spatio-temporal information to provide fine-grained old-class labels for the updated model. Moreover, since the high activation segments contain rich action information, we select them and construct video-level logits to constrain the updated model for maintaining the old-class knowledge further. The experimental results under various incremental learning settings on the THUMOS’14 and ActivityNet 1.3 datasets reveal that our method effectively alleviates the catastrophic forgetting problem in WTAL.

This work was supported in part by the National Innovation 2030 Major S &T Project of China under Grant 2020AAA0104203, and in part by the Nature Science Foundation of China under Grant 62006007.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: a large-scale video benchmark for human activity understanding. In: CVPR, pp. 961–970 (2015)

    Google Scholar 

  2. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR, pp. 6299–6308 (2017)

    Google Scholar 

  3. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR, pp. 580–587 (2014)

    Google Scholar 

  4. Hao, Y., Fu, Y., Jiang, Y.G., Tian, Q.: An end-to-end architecture for class-incremental object detection with knowledge distillation. In: ICME, pp. 1–6 (2019)

    Google Scholar 

  5. He, B., Yang, X., Kang, L., Cheng, Z., Zhou, X., Shrivastava, A.: Asm-loc: action-aware segment modeling for weakly-supervised temporal action localization. In: CVPR, pp. 13925–13935 (2022)

    Google Scholar 

  6. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  7. Idrees, H., Zamir, A.R., Jiang, Y.G., Gorban, A., Laptev, I., Sukthankar, R., Shah, M.: The thumos challenge on action recognition for videos “in the wild.” Computer Vision and Image Understanding 155, 1–23 (2017)

    Google Scholar 

  8. Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  9. Lee, P., Uh, Y., Byun, H.: Background suppression network for weakly-supervised temporal action localization. In: AAAI, vol. 34, pp. 11320–11327 (2020)

    Google Scholar 

  10. Lee, P., Wang, J., Lu, Y., Byun, H.: Weakly-supervised temporal action localization by uncertainty modeling. In: AAAI, vol. 35, pp. 1854–1862 (2021)

    Google Scholar 

  11. Li, B., Liu, R., Chen, T., Zhu, Y.: Weakly supervised temporal action detection with temporal dependency learning. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4474–4485 (2021)

    Google Scholar 

  12. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)

    Article  Google Scholar 

  13. Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization by sparse temporal pooling network. In: CVPR, pp. 6752–6761 (2018)

    Google Scholar 

  14. Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: CVPR, pp. 2001–2010 (2017)

    Google Scholar 

  15. Shou, Z., Wang, D., Chang, S.F.: Temporal action localization in untrimmed videos via multi-stage cnns. In: CVPR, pp. 1049–1058 (2016)

    Google Scholar 

  16. Van de Ven, G.M., Tolias, A.S.: Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 (2019)

  17. Wang, L., Xiong, Y., Lin, D., Van Gool, L.: Untrimmednets for weakly supervised action recognition and detection. In: CVPR, pp. 4325–4334 (2017)

    Google Scholar 

  18. Xu, M., Zhao, C., Rojas, D.S., Thabet, A., Ghanem, B.: G-tad: Sub-graph localization for temporal action detection. In: CVPR, pp. 10156–10165 (2020)

    Google Scholar 

  19. Yu, J., Ge, Y., Li, Z., Chen, Z., Qin, X.: Context driven network with bayes for weakly supervised temporal action localization. In: ICME, pp. 1–6 (2021)

    Google Scholar 

  20. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74936-3_22

    Chapter  Google Scholar 

  21. Zhang, C., Cao, M., Yang, D., Chen, J., Zou, Y.: Cola: weakly-supervised temporal action localization with snippet contrastive learning. In: CVPR, pp. 16010–16019 (2021)

    Google Scholar 

  22. Zhao, B., Xiao, X., Gan, G., Zhang, B., Xia, S.T.: Maintaining discrimination and fairness in class incremental learning. In: CVPR, pp. 13208–13217 (2020)

    Google Scholar 

  23. Zhu, Z., Tang, W., Wang, L., Zheng, N., Hua, G.: Enriching local and global contexts for temporal action localization. In: ICCV, pp. 13516–13525 (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yuesheng Zhu .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, T., Li, B., Tao, Y., Wang, Y., Zhu, Y. (2023). Class-Incremental Learning with Multiscale Distillation for Weakly Supervised Temporal Action Localization. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Lecture Notes in Computer Science, vol 13623. Springer, Cham. https://doi.org/10.1007/978-3-031-30105-6_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-30105-6_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-30104-9

  • Online ISBN: 978-3-031-30105-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics