Abstract
Catastrophic forgetting is a ubiquitous problem for the current generation of Artificial Neural Networks: When a network is asked to learn multiple tasks in a sequence, it fails dramatically as it tends to forget past knowledge. Little is known on how far multimodal conversational agents suffer from this phenomenon. In this paper, we study the problem of catastrophic forgetting in Visual Question Answering (VQA) and propose experiments in which we analyze pairs of tasks based on CLEVR, a dataset requiring different skills which involve visual or linguistic knowledge. Our results show that dramatic forgetting is at place in VQA, calling for studies on how multimodal models can be enhanced with continual learning methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In Experiment 1, Task A and B are trained on different amounts of data (see Sect. 3). We have trained the model on Task A with the same amount of data as Task B obtaining very similar performance: 0.566.
References
Chaudhry A, Dokania PK, Ajanthan T, Torr P (2018) Understanding forgetting and intransigence. In: ECCV, Riemannian walk for incremental learning
Das A, Kottur S, Gupta K, Singh A, Yadav D, Moura JMF, Parikh D, Batra D (2017) Visual dialog. In: CVPR
Díaz-Rodríguez N, Lomonaco V, Filliat D, Maltoni D (2018) Don’t forget, there is more than forgetting: new metrics for continual learning. In: Workshop on continual learning, NeurIPS
Johnson J, Hariharan B, van der Maaten L, Fei-Fei L, Lawrence Zitnick C, Girshick R (2017) CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning. In: CVPR
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980
Kirkpatrick J, Pascanu R, Rabinowitz N, Veness J, Desjardins G, Rusu AA, Milan K, Quan J, Ramalho T, Grabska-Barwinska A et al (2017) Overcoming catastrophic forgetting in neural networks. PNAS
Maltoni D, Lomonaco V (2018) Continuous learning in single-incremental-task scenarios. arXiv:1806.08568
McClelland JL, McNaughton BL, O’reilly RC (1995) Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol Rev 102(3)
Moradlou S, Ginzburg J (2014) Learning to understand questions. In: SemDial
Perez E, Strub F, De Vries H, Dumoulin V (2018) Visual reasoning with a general conditioning layer. In: AAAI, Film
Ring M (1997) CHILD: a first step towards continual learning. Mach Learning 28(1)
Yang Z, He X, Gao J, Deng L, Smola A (2016) Stacked attention networks for image question answering. In: CVPR
Zamir AR, Sax A, Shen W, Guibas L, Malik J, Savarese S (2018) Disentangling task transfer learning. In: CVPR, Taskonomy
Zenke F, Poole B, Ganguli S (2017) Continual learning through synaptic intelligence. In: ICML
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Greco, C., Plank, B., Fernández, R., Bernardi, R. (2021). Measuring Catastrophic Forgetting in Visual Question Answering. In: Marchi, E., Siniscalchi, S.M., Cumani, S., Salerno, V.M., Li, H. (eds) Increasing Naturalness and Flexibility in Spoken Dialogue Interaction. Lecture Notes in Electrical Engineering, vol 714. Springer, Singapore. https://doi.org/10.1007/978-981-15-9323-9_35
Download citation
DOI: https://doi.org/10.1007/978-981-15-9323-9_35
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-9322-2
Online ISBN: 978-981-15-9323-9
eBook Packages: EngineeringEngineering (R0)