Skip to main content

Lifelong Sentiment Classification Based on Adaptive Parameter Updating

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2024 (ICANN 2024)

Abstract

A classifier with the ability to handle continuous streams of opinion information on the Internet should have good lifelong learning ability. However, deep neural networks face the challenge of catastrophic forgetting when continuously incorporating new domain data, resulting in the loss of previously learned information. At the same time, using the knowledge of old tasks to help the learning of new tasks is also a challenge faced by lifelong learning. In this paper, we propose a novel lifelong sentiment classification method based on adaptive parameter update, which effectively prevent the catastrophic forgetting and promote knowledge transfer among tasks. In our method, we use the uncertainty regularization parameter update strategy to prevent the forgetting of old domain information, and propose an effective parameter updating strategy for sub-network, which can be used to realize the knowledge transfer among tasks. Extensive experiments on 16 popular review corpora demonstrate that the proposed method significantly outperforms the strong baselines for lifelong sentiment classification.

This work is supported by National Science and Technology Major Project (2020AAA0109703), National Natural Science Foundation of China (62076167, U23B2029).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://huggingface.co/google-bert/bert-base-uncased.

References

  1. Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075 (2015)

  2. Wang, J., Yu, L.-C., Robert Lai, K., Zhang, X.: Dimensional sentiment analysis using a regional CNN-LSTM model. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol. 2: Short papers, pp. 225–230 (2016)

    Google Scholar 

  3. Wen, S., et al.: Memristive LSTM network for sentiment analysis. IEEE Trans. Syst. Man Cybern. Syst. 51(3), 1794–1804 (2019)

    Google Scholar 

  4. Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: a review. Neural Netw. 113, 54–71 (2019)

    Google Scholar 

  5. Biesialska, M., Biesialska, K., Costa-Jussa, M.R.: Continual lifelong learning in natural language processing: a survey. arXiv preprint arXiv:2012.09823 (2020)

  6. Zhang, L., Wang, S., Yuan, F., Geng, B., Yang, M.: Lifelong language learning with adaptive uncertainty regularization. Inf. Sci. 622, 794–807 (2023)

    Article  Google Scholar 

  7. Ahn, H., Cha, S., Lee, D., Moon, T.: Uncertainty-based continual learning with adaptive regularization. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  8. Zhai, Y., et al.: Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309.10313 (2023)

  9. Chen, Z., Ma, N., Liu, B.: Lifelong learning for sentiment classification. arXiv preprint arXiv:1801.02808 (2018)

  10. Wang, S., Liu, B., Chen, E., Zhang, K.: Sentiment classification by leveraging the shared knowledge from a sequence of domains. In: Proceedings of the 24th International Conference on Database Systems for Advanced Applications (DASFAA) (2019)

    Google Scholar 

  11. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota, June (2019). Association for Computational Linguistics

    Google Scholar 

  12. Ke, Z., Shao, Y., Lin, H., Xu, H., Shu, L., Liu, B.: Adapting a language model while preserving its general knowledge (2023)

    Google Scholar 

  13. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.B.: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies, volume 1 (long and short papers). Minneapolis, MN: Association for Computational Linguistics, pp. 4171–4186 (2019)

    Google Scholar 

  14. Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)

    Article  MathSciNet  Google Scholar 

  15. Radford, A., Narasimhan, K., Sutskever, I., et al.: Improving language understanding by generative pre-training, Tim Salimans (2018)

    Google Scholar 

  16. Zhuang, L., Wayne, L., Ya, S., Jun, Z: A robustly optimized BERT pre-training approach with post-training. In: Proceedings of the 20th Chinese National Conference on Computational Linguistics, pp. 1218–1227 (2021)

    Google Scholar 

  17. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: a lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019)

  18. Glorot, X., Bordes, A., Bengio, Y.: Domain adaptation for large-scale sentiment classification: a deep learning approach. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 513–520 (2011)

    Google Scholar 

  19. Yuan, F., Zhang, G., Karatzoglou, A., Jose, J., Kong, B., Li, Y.: One person, one model, one world: learning continual user representation without forgetting. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 696–705 (2021)

    Google Scholar 

  20. Ziser, Y., Reichart, R.: Pivot based language modeling for improved neural domain adaptation. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1241–1251 (2018)

    Google Scholar 

  21. Mermillod, M., Bugaiska, A., Bonin, P.: The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol. 4, 54654 (2013)

    Article  Google Scholar 

  22. Wang, Z., et al. Rehearsal-free continual language learning via efficient parameter isolation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10933–10946 (2023)

    Google Scholar 

  23. Wortsman, M., et al.: Supermasks in superposition. Adv. Neural. Inf. Process. Syst. 33, 15173–15184 (2020)

    Google Scholar 

  24. Sun, F.-K., Ho, C.-H., Lee, H.-Y.: LAMOL: language modeling for lifelong language learning. arXiv preprint arXiv:1909.03329 (2019)

  25. Xin, S., Guo, S., Tan, T., Chen, F.: Generative memory for lifelong learning. IEEE Trans. Neural Netw. Learn. Syst. 31(6), 1884–1898 (2019)

    MathSciNet  Google Scholar 

  26. Geng, B., Yang, M., Yuan, F., Wang, S., Ao, X., Xu, R.: Iterative network pruning with uncertainty regularization for lifelong sentiment classification. In: Proceedings of the 44th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 1229–1238 (2021)

    Google Scholar 

  27. De Lange, M., van de Ven, G., Tuytelaars, T.: Continual evaluation for lifelong learning: identifying the stability gap. arXiv preprint arXiv:2205.13452 (2022)

  28. Mallya, A., Davis, D., Lazebnik, S.: Piggyback: adapting a single network to multiple tasks by learning to mask weights. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 67–82 (2018)

    Google Scholar 

  29. Maas, A., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150 (2011)

    Google Scholar 

  30. Pang, B., Lee, L.: Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales (2005)

    Google Scholar 

  31. Loshchilov, I., Hutter, F.: Fixing weight decay regularization in Adam (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jie Liu .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, Z., Wang, J., Nie, K., Wang, X., Liu, J. (2024). Lifelong Sentiment Classification Based on Adaptive Parameter Updating. In: Wand, M., Malinovská, K., Schmidhuber, J., Tetko, I.V. (eds) Artificial Neural Networks and Machine Learning – ICANN 2024. ICANN 2024. Lecture Notes in Computer Science, vol 15022. Springer, Cham. https://doi.org/10.1007/978-3-031-72350-6_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72350-6_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72349-0

  • Online ISBN: 978-3-031-72350-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics