Abstract
Despite recent works having made great progress in weakly supervised temporal action localization (WTAL), they still suffer from catastrophic forgetting. When only new-class videos can be utilized to update these models, their performance in old classes diminishes drastically. Even while some class-incremental learning methods are presented to assist models in continuously learning new-class knowledge, most of them focus on image classification but pay little attention to WTAL. To fill this gap, we propose a novel class-incremental learning method with multiscale distillation, which mines two separate scales of old-class information in incoming videos for updating the model. Precisely, we calculate class activation sequences (CAS) with frame-level spatio-temporal information to provide fine-grained old-class labels for the updated model. Moreover, since the high activation segments contain rich action information, we select them and construct video-level logits to constrain the updated model for maintaining the old-class knowledge further. The experimental results under various incremental learning settings on the THUMOS’14 and ActivityNet 1.3 datasets reveal that our method effectively alleviates the catastrophic forgetting problem in WTAL.
This work was supported in part by the National Innovation 2030 Major S &T Project of China under Grant 2020AAA0104203, and in part by the Nature Science Foundation of China under Grant 62006007.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: a large-scale video benchmark for human activity understanding. In: CVPR, pp. 961–970 (2015)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: CVPR, pp. 6299–6308 (2017)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR, pp. 580–587 (2014)
Hao, Y., Fu, Y., Jiang, Y.G., Tian, Q.: An end-to-end architecture for class-incremental object detection with knowledge distillation. In: ICME, pp. 1–6 (2019)
He, B., Yang, X., Kang, L., Cheng, Z., Zhou, X., Shrivastava, A.: Asm-loc: action-aware segment modeling for weakly-supervised temporal action localization. In: CVPR, pp. 13925–13935 (2022)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Idrees, H., Zamir, A.R., Jiang, Y.G., Gorban, A., Laptev, I., Sukthankar, R., Shah, M.: The thumos challenge on action recognition for videos “in the wild.” Computer Vision and Image Understanding 155, 1–23 (2017)
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)
Lee, P., Uh, Y., Byun, H.: Background suppression network for weakly-supervised temporal action localization. In: AAAI, vol. 34, pp. 11320–11327 (2020)
Lee, P., Wang, J., Lu, Y., Byun, H.: Weakly-supervised temporal action localization by uncertainty modeling. In: AAAI, vol. 35, pp. 1854–1862 (2021)
Li, B., Liu, R., Chen, T., Zhu, Y.: Weakly supervised temporal action detection with temporal dependency learning. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4474–4485 (2021)
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2017)
Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization by sparse temporal pooling network. In: CVPR, pp. 6752–6761 (2018)
Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: CVPR, pp. 2001–2010 (2017)
Shou, Z., Wang, D., Chang, S.F.: Temporal action localization in untrimmed videos via multi-stage cnns. In: CVPR, pp. 1049–1058 (2016)
Van de Ven, G.M., Tolias, A.S.: Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 (2019)
Wang, L., Xiong, Y., Lin, D., Van Gool, L.: Untrimmednets for weakly supervised action recognition and detection. In: CVPR, pp. 4325–4334 (2017)
Xu, M., Zhao, C., Rojas, D.S., Thabet, A., Ghanem, B.: G-tad: Sub-graph localization for temporal action detection. In: CVPR, pp. 10156–10165 (2020)
Yu, J., Ge, Y., Li, Z., Chen, Z., Qin, X.: Context driven network with bayes for weakly supervised temporal action localization. In: ICME, pp. 1–6 (2021)
Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74936-3_22
Zhang, C., Cao, M., Yang, D., Chen, J., Zou, Y.: Cola: weakly-supervised temporal action localization with snippet contrastive learning. In: CVPR, pp. 16010–16019 (2021)
Zhao, B., Xiao, X., Gan, G., Zhang, B., Xia, S.T.: Maintaining discrimination and fairness in class incremental learning. In: CVPR, pp. 13208–13217 (2020)
Zhu, Z., Tang, W., Wang, L., Zheng, N., Hua, G.: Enriching local and global contexts for temporal action localization. In: ICCV, pp. 13516–13525 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, T., Li, B., Tao, Y., Wang, Y., Zhu, Y. (2023). Class-Incremental Learning with Multiscale Distillation for Weakly Supervised Temporal Action Localization. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Lecture Notes in Computer Science, vol 13623. Springer, Cham. https://doi.org/10.1007/978-3-031-30105-6_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-30105-6_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30104-9
Online ISBN: 978-3-031-30105-6
eBook Packages: Computer ScienceComputer Science (R0)