Skip to main content
SpringerLink
Log in

A Reservoir Computing Approach to Word Sense Disambiguation

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Reservoir computing (RC) has emerged as an alternative approach for the development of fast trainable recurrent neural networks (RNNs). It is considered to be biologically plausible due to the similarity between randomly designed artificial reservoir structures and cortical structures in the brain. The paper continues our previous research on the application of a member of the family of RC approaches—the echo state network (ESN)—to the natural language processing (NLP) task of Word Sense Disambiguation (WSD). A novel deep bi-directional ESN (DBiESN) structure is proposed, as well as a novel approach for exploiting reservoirs’ steady states. The models also make use of ESN-enhanced word embeddings. The paper demonstrates that our DBiESN approach offers a good alternative to previously tested BiESN models in the context of the word sense disambiguation task having smaller number of trainable parameters. Although our DBiESN-based model achieves similar accuracy to other popular RNN architectures, we could not outperform the state of the art. However, due to the smaller number of trainable parameters in the reservoir models, in contrast to fully trainable RNNs, it is to be expected that they would have better generalization properties as well as higher potential to increase their accuracy, which should justify further exploration of such architectures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Wang P, Qian Y, Soong FK, He L, Zhao H. 2015. Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural Network. arXiv:1510.06168.

  2. Wang P, Qian Y, Soong FK, He L, Zhao H. 2015. A Unified Tagging Solution: Bidirectional LSTM Recurrent Neural Network with Word Embedding. arXiv:1511.00215.

  3. Huang Z, Xu W, Yu K. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv:1508.01991.

  4. Wang W, Chang B. Graph-based Dependency Parsing with Bidirectional LSTM. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers Association for Computational Linguistics, Berlin Germany; 2016. p. 2306–2315. https://doi.org/10.18653/v1/P16-1218.

  5. Popov A. Neural network models for word sense disambiguation: an overview. Cybernetics and Information Technologies 2018;18:139–151.

    Article  MathSciNet  Google Scholar 

  6. Hochreiter S, Schmidhuber J. Long short-Term memory. Neural Comput 1997;9(8):1735. https://doi.org/10.1162/neco.1997.9.8.1735.

    Article  Google Scholar 

  7. Cho K, van Merriënboer B., Bahdanau D, Bengio Y. On the Properties of Neural Machine translation: encoder–Decoder approaches. proceedings of SSST-8, Eighth Workshop on Syntax Semantics and Structure in Statistical Translation Association for Computational Linguistics Doha Qatar; 2014 . p. 103–111. https://doi.org/10.3115/v1/W14-4012.

  8. Maass W, Natschläger T, Markram H. Real-time Computing Without Stable states: A New Framework for Neural Computation Based on Perturbations. Neural Comput. 2002;14(11):2531. https://doi.org/10.1162/089976602760407955.

    Article  MATH  Google Scholar 

  9. Jaeger H. 2002. Tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the Echo State Network Approach, GMD Report 159 German National Research Center for Information Technology.

  10. Lukosevicius M, Jaeger H. Reservoir computing approaches to recurrent neural network training. Computer Science Review 2009;3:127. https://doi.org/10.1016/j.cosrev.2009.03.005.

    Article  MATH  Google Scholar 

  11. Scardapone S, Butcher J, Bianchi S, Malik Z. 2017. Advances in Biologically Inspired Reservoir Computing, Vol. 9. https://doi.org/10.1007/s12559-017-9469-1.

  12. Enel P, Procyk E, Quilodran R, Dominey PF. Reservoir Computing Properties of Neural Dynamics in Prefrontal Cortex. PLOS Computational Biology 2016;12(6):1. https://doi.org/10.1371/journal.pcbi.1004967.

    Article  Google Scholar 

  13. Heinrich S, Wermter S. Interactive natural language acquisition in a multi-modal recurrent neural architecture. Connection Science 2018;30(1):99. https://doi.org/10.1080/09540091.2017.1318357.

    Article  Google Scholar 

  14. Butcher JB, Verstraeten D, Schrauwen B, Day CR, Haycock PW. Reservoir Computing and Extreme Learning Machines for Non-linear time-Series Data Analysis. Neural Netw. 2013;38:76. https://doi.org/10.1016/j.neunet.2012.11.011.

    Article  Google Scholar 

  15. Gallicchio C, Micheli A, Pedrelli L. 2018. Comparison between Deep ESNs and Gated RNNs on Multivariate Time-Series Prediction. arXiv:1812.11527.

  16. Frank SL, Čerňanský M.P. Generalization and Systematicity in Echo State Networks, in the Annual Meeting of the Cognitive Science Society; 2008. pp. 733–738.

  17. Hinaut X, Dominey PF. Real-time Parallel Processing of Grammatical Structure in the fronto-Striatal system: A Recurrent Network Simulation Study Using Reservoir Computing. PLOS ONE 2013;8(2):1. https://doi.org/10.1371/journal.pone.0052946.

    Article  Google Scholar 

  18. Twiefel J, Hinaut X, Wermter S. Semantic Role Labelling for Robot Instructions using Echo State Networks. 24th European Symposium on Artificial Neural Networks, ESANN 2016, Bruges, Belgium, April 27-29, 2016; 2016. http://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2016-168.pdf.

  19. Twiefel J, Hinaut X, Soares MB, Strahl E, Wermter S. Using Natural Language Feedback in a Neuro-Inspired Integrated Multimodal Robotic Architecture. 25th IEEE International Symposium on Robot and Human Interactive Communication, RO-MAN New york, NY, USA, August 26-31, 2016; 2016. p. 52–57. https://doi.org/10.1109/ROMAN.2016.7745090.

  20. Skowronski M, Harris J. 2006. Minimum mean squared error time series classification using an echo state network prediction model in 2006 IEEE international symposium on circuits and systems IEEE. https://doi.org/10.1109/ISCAS.2006.1693294.

  21. Squartini S, Cecchi S, Rossini M, Piazza F. Echo State Networks for Real-Time Audio Applications. Advances in Neural Networks – ISNN 2007, ed. by D. Liu, S. Fei, Z. Hou, H. Zhang, C. Sun Springer Berlin Heidelberg, Berlin, Heidelberg; 2007. p. 731–740. https://doi.org/10.1007/978-3-540-72395-0_90.

  22. Tong MH, Bickett AD, Christiansen EM, Cottrell GW. Learning grammatical structure with echo state networks. Neural Netw 2007;20(3):424. https://doi.org/10.1016/j.neunet.2007.04.013.

    Article  MATH  Google Scholar 

  23. Ramamurthy R, Stenzel R, Sifa R, Ladi A, Bauckhage C. Echo State Networks for Named Entity Recognition. Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions ed. by I.V. Tetko, V. Ku̇rková, P. Karpov, F. Theis Springer International Publishing, Cham; 2019. p. 110–120.

  24. Koprinkova-Hristova P, Popov A, Simov K, Osenova P. Echo State Network for Word Sense Disambiguation. Artificial intelligence: Methodology, Systems, and Applications - 18th International Conference, AIMSA 2018, Varna, Bulgaria, September 12-14, 2018 Proceedings; 2018. p. 73–82. https://doi.org/10.1007/978-3-319-99344-7_7.

  25. Popov A, Koprinkova-Hristova P, Simov K, Osenova P. Echo State vs. LSTM Networks for Word Sense Disambiguation. Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions ed. by I.V. Tetko, V. Ku̇rková, P. Karpov, F. Theis Springer International Publishing, Cham; 2019. p. 94–109.

  26. Gallicchio C, Micheli A. A Reservoir Computing Approach for Human Gesture Recognition from Kinect Data. Proceedings of the Artificial Intelligence for Ambient Assisted Living 2016 co-located with 15th International Conference of the Italian Association for Artificial Intelligence AIxIA 2016 Genova, Italy, November 28th, 2016; 2016. p. 33–42. http://ceur-ws.org/Vol-1803/paper3.pdf.

  27. Rodan A, Sheta AF, Faris H. Bidirectional Reservoir Networks Trained Using SVM + Privileged Information for Manufacturing Process Modeling. Soft Comput 2017;21(22): 6811. https://doi.org/10.1007/s00500-016-2232-9.

    Article  Google Scholar 

  28. Popov A. Networks Word Sense Disambiguation with Recurrent Neural. Proceedings of the student research workshop associated with RANLP 2017 INCOMA ltd. Varna; 2017. p. 25–34. https://doi.org/10.26615/issn.1314-9156.2017_004.

  29. Simov KI, Koprinkova-Hristova PD, Popov A, Osenova P. Word Embeddings Improvement via Echo State Networks. IEEE International Symposium on INnovations in Intelligent SysTems and Applications, INISTA 2019, Sofia, Bulgaria, July 3-5, 2019; 2019. p. 1–6. https://doi.org/10.1109/INISTA.2019.8778297.

  30. Koprinkova-Hristova PD, Tontchev N. Echo State Networks for Multi-dimensional Data Clustering. Artificial Neural Networks and Machine Learning - ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Lausanne, Switzerland, September 11-14, 2012, Proceedings, Part I; 2012. p. 571–578. https://doi.org/10.1007/978-3-642-33269-2_72.

  31. Gallicchio C, Micheli A. 2016. A reservoir computing approach for human gesture recognition from kinect data inproceedings of the AI for ambient assisted living.

  32. Raganato A, Camacho-Collados J, Navigli R. Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers Association for Computational Linguistics, Valencia, Spain; 2017. p. 99–110. https://www.aclweb.org/anthology/E17-1010.

  33. Koprinkova-Hristova P. 2016. Multi-dimensional Data Clustering and Visualization via Echo State Networks Springer International Publishing, Cham. https://doi.org/10.1007/978-3-319-32192-9_3.

  34. Jaeger H. 2007. Discovering multiscale dynamical features with hierarchical echo state networks. Tech. rep., Jacobs University Bremen.

  35. Fernández S, Graves A, Schmidhuber J. Sequence Labelling in Structured Domains with Hierarchical Recurrent Neural Networks. Proceedings of the 20th International Joint Conference on Artifical Intelligence Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, IJCAI’07; 2007. p. 774–779.

  36. Triefenbach F, Jalalvand A, Schrauwen B, Pierre Martens J. Phoneme Recognition with Large Hierarchical Reservoirs. Advances in Neural Information Processing Systems 23, ed. by J.D. Lafferty, C.K.I. Williams, J. Shawe-Taylor, R.S. Zemel, A. Culotta (Curran Associates, Inc.; 2010. p. 2307–2315. http://papers.nips.cc/paper/4056-phoneme-recognition-with-large-hierarchical-reservoirs.pdf.

  37. Triefenbach F, Jalalvand A, Demuynck K, Martens J. Acoustic modeling with hierarchical reservoirs, IEEE transactions on audio. Speech, and Language Processing 2013;21(11):2439. https://doi.org/10.1109/TASL.2013.2280209.

    Article  Google Scholar 

  38. Triefenbach F, Demuynck K, Martens J. Large vocabulary continuous speech recognition with reservoir-Based acoustic models. IEEE Signal Processing Letters 2014;21(3):311. https://doi.org/10.1109/LSP.2014.2302080.

    Article  Google Scholar 

  39. Bengio Y, Lee DH, Bornschein J, Lin Z. 2015. Towards biologically plausible deep learning. arXiv:1502.04156.

  40. Fellbaum C. 2010. Wordnet Springer Netherlands. https://doi.org/10.1007/978-90-481-8847-5_10.

  41. G.A. Miller, Leacock C, Tengi R, Bunker RT. A Semantic Concordance. Proceedings of the Workshop on Human Language Technology Association for Computational Linguistics, Stroudsburg, PA, USA,HLT ’93, pp. 303–308; 1993. https://doi.org/10.3115/1075671.1075742.

  42. Edmonds P, Cotton S. SENSEVAL-2: Overview. Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems Association for Computational Linguistics, Toulouse, France; 2001. p. 1–5. https://www.aclweb.org/anthology/S01-1001.

  43. Snyder B, Palmer M. The English all-words task. Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text Association for Computational Linguistics, Barcelona, Spain; 2004. p. 41–43. https://www.aclweb.org/anthology/W04-0811.

  44. Pradhan S, Loper E, Dligach D, Palmer M. SemEval-2007 Task-17: English Lexical Sample, SRL and All Words. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007) Association for Computational Linguistics, Prague, Czech Republic; 2007. p. 87–92. https://www.aclweb.org/anthology/S07-1016.

  45. Navigli R, Jurgens D, Vannella D. SemEval-2013 Task 12: Multilingual Word Sense Disambiguation. Second Joint Conference on Lexical and Computational Semantics *SEM, Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation SemEval 2013 Association for Computational Linguistics, Atlanta, Georgia, USA; 2013 . p. 222–231. https://www.aclweb.org/anthology/S13-2040.

  46. Moro A, Navigli R. SemEval-2015 Task 13: Multilingual All-Words Sense Disambiguation and Entity Linking. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015) (Association for Computational Linguistics, Denver, Colorado); 2015. p. 288–297. https://doi.org/10.18653/v1/S15-2049.

  47. Yuxin C, Le-Ngoc T, Champagne B, Changjiang X. Recursive least squares constant modulus algorithm for blind adaptive array. IEEE Transactions on Signal Processing 2004;52(5):1452.

    Article  MathSciNet  MATH  Google Scholar 

  48. Slobodyan S, Bogomolova A, Kolyuzhnov D. Stochastic gradient versus recursive least squares learning. SRNN CERGE-EI Working Paper 2006;309:1. https://doi.org/10.2139/ssrn.1129821.

    Google Scholar 

  49. Belkin M, Hsu D, Ma S, Mandal S. 2018. Reconciling modern machine learning practice and the bias-variance trade-off arxiv: Machine Learning.

Download references

Funding

This research has been partially supported by the National Scientific Program “Information and Communication Technologies for a Single Digital Market in Science, Education and Security (ICTinSES),” financed by the Ministry of Education and Science. Alexander Popov’s contribution has been supported by the Bulgarian Ministry of Education and Science under the National Research Programme “Young scientists and postdoctoral students” approved by DCM # 577 / 17.08.2018.

Author information

Authors and Affiliations

  1. Institute of Information and Communication Technologies, Bulgarian Academy of Sciences, Sofia, Bulgaria

    Kiril Simov, Petia Koprinkova-Hristova, Alexander Popov & Petya Osenova

Authors
  1. Kiril Simov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Petia Koprinkova-Hristova
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexander Popov
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Petya Osenova
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Petia Koprinkova-Hristova.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Trends in Reservoir Computing

Guest Editors: Claudio Gallicchio, Alessio Micheli, Simone Scardapane, Miguel C. Soriano

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Simov, K., Koprinkova-Hristova, P., Popov, A. et al. A Reservoir Computing Approach to Word Sense Disambiguation. Cogn Comput 15, 1409–1418 (2023). https://doi.org/10.1007/s12559-020-09758-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-020-09758-w

Keywords

Search

Navigation