Abstract
A classifier with the ability to handle continuous streams of opinion information on the Internet should have good lifelong learning ability. However, deep neural networks face the challenge of catastrophic forgetting when continuously incorporating new domain data, resulting in the loss of previously learned information. At the same time, using the knowledge of old tasks to help the learning of new tasks is also a challenge faced by lifelong learning. In this paper, we propose a novel lifelong sentiment classification method based on adaptive parameter update, which effectively prevent the catastrophic forgetting and promote knowledge transfer among tasks. In our method, we use the uncertainty regularization parameter update strategy to prevent the forgetting of old domain information, and propose an effective parameter updating strategy for sub-network, which can be used to realize the knowledge transfer among tasks. Extensive experiments on 16 popular review corpora demonstrate that the proposed method significantly outperforms the strong baselines for lifelong sentiment classification.
This work is supported by National Science and Technology Major Project (2020AAA0109703), National Natural Science Foundation of China (62076167, U23B2029).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075 (2015)
Wang, J., Yu, L.-C., Robert Lai, K., Zhang, X.: Dimensional sentiment analysis using a regional CNN-LSTM model. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol. 2: Short papers, pp. 225–230 (2016)
Wen, S., et al.: Memristive LSTM network for sentiment analysis. IEEE Trans. Syst. Man Cybern. Syst. 51(3), 1794–1804 (2019)
Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019)
Biesialska, M., Biesialska, K., Costa-Jussa, M.R.: Continual lifelong learning in natural language processing: a survey. arXiv preprint arXiv:2012.09823 (2020)
Zhang, L., Wang, S., Yuan, F., Geng, B., Yang, M.: Lifelong language learning with adaptive uncertainty regularization. Inf. Sci. 622, 794–807 (2023)
Ahn, H., Cha, S., Lee, D., Moon, T.: Uncertainty-based continual learning with adaptive regularization. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Zhai, Y., et al.: Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309.10313 (2023)
Chen, Z., Ma, N., Liu, B.: Lifelong learning for sentiment classification. arXiv preprint arXiv:1801.02808 (2018)
Wang, S., Liu, B., Chen, E., Zhang, K.: Sentiment classification by leveraging the shared knowledge from a sequence of domains. In: Proceedings of the 24th International Conference on Database Systems for Advanced Applications (DASFAA) (2019)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, June (2019). Association for Computational Linguistics
Ke, Z., Shao, Y., Lin, H., Xu, H., Shu, L., Liu, B.: Adapting a language model while preserving its general knowledge (2023)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.B.: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies, volume 1 (long and short papers). Minneapolis, MN: Association for Computational Linguistics, pp. 4171–4186 (2019)
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)
Radford, A., Narasimhan, K., Sutskever, I., et al.: Improving language understanding by generative pre-training, Tim Salimans (2018)
Zhuang, L., Wayne, L., Ya, S., Jun, Z: A robustly optimized BERT pre-training approach with post-training. In: Proceedings of the 20th Chinese National Conference on Computational Linguistics, pp. 1218–1227 (2021)
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: a lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019)
Glorot, X., Bordes, A., Bengio, Y.: Domain adaptation for large-scale sentiment classification: a deep learning approach. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 513–520 (2011)
Yuan, F., Zhang, G., Karatzoglou, A., Jose, J., Kong, B., Li, Y.: One person, one model, one world: learning continual user representation without forgetting. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 696–705 (2021)
Ziser, Y., Reichart, R.: Pivot based language modeling for improved neural domain adaptation. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1241–1251 (2018)
Mermillod, M., Bugaiska, A., Bonin, P.: The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol. 4, 54654 (2013)
Wang, Z., et al. Rehearsal-free continual language learning via efficient parameter isolation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10933–10946 (2023)
Wortsman, M., et al.: Supermasks in superposition. Adv. Neural. Inf. Process. Syst. 33, 15173–15184 (2020)
Sun, F.-K., Ho, C.-H., Lee, H.-Y.: LAMOL: language modeling for lifelong language learning. arXiv preprint arXiv:1909.03329 (2019)
Xin, S., Guo, S., Tan, T., Chen, F.: Generative memory for lifelong learning. IEEE Trans. Neural Netw. Learn. Syst. 31(6), 1884–1898 (2019)
Geng, B., Yang, M., Yuan, F., Wang, S., Ao, X., Xu, R.: Iterative network pruning with uncertainty regularization for lifelong sentiment classification. In: Proceedings of the 44th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 1229–1238 (2021)
De Lange, M., van de Ven, G., Tuytelaars, T.: Continual evaluation for lifelong learning: identifying the stability gap. arXiv preprint arXiv:2205.13452 (2022)
Mallya, A., Davis, D., Lazebnik, S.: Piggyback: adapting a single network to multiple tasks by learning to mask weights. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 67–82 (2018)
Maas, A., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150 (2011)
Pang, B., Lee, L.: Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales (2005)
Loshchilov, I., Hutter, F.: Fixing weight decay regularization in Adam (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Z., Wang, J., Nie, K., Wang, X., Liu, J. (2024). Lifelong Sentiment Classification Based on Adaptive Parameter Updating. In: Wand, M., Malinovská, K., Schmidhuber, J., Tetko, I.V. (eds) Artificial Neural Networks and Machine Learning – ICANN 2024. ICANN 2024. Lecture Notes in Computer Science, vol 15022. Springer, Cham. https://doi.org/10.1007/978-3-031-72350-6_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-72350-6_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72349-0
Online ISBN: 978-3-031-72350-6
eBook Packages: Computer ScienceComputer Science (R0)