Skip to main content
SpringerLink
Log in

Distributed Representation of n-gram Statistics for Boosting Self-organizing Maps with Hyperdimensional Computing

  • Conference paper
  • First Online:
Perspectives of System Informatics (PSI 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11964))

Abstract

This paper presents an approach for substantial reduction of the training and operating phases of Self-Organizing Maps in tasks of 2-D projection of multi-dimensional symbolic data for natural language processing such as language classification, topic extraction, and ontology development. The conventional approach for this type of problem is to use n-gram statistics as a fixed size representation for input of Self-Organizing Maps. The performance bottleneck with n-gram statistics is that the size of representation and as a result the computation time of Self-Organizing Maps grows exponentially with the size of n-grams. The presented approach is based on distributed representations of structured data using principles of hyperdimensional computing. The experiments performed on the European languages recognition task demonstrate that Self-Organizing Maps trained with distributed representations require less computations than the conventional n-gram statistics while well preserving the overall performance of Self-Organizing Maps.

This work was supported by the Swedish Research Council (VR, grant 2015-04677) and the Swedish Foundation for International Cooperation in Research and Higher Education (grant IB2018-7482) for its Initiation Grant for Internationalisation, which allowed conducting the study.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Available online at http://www.statmt.org/europarl/.

References

  1. Alahakoon, D., Halgamuge, S., Srinivasan, B.: Dynamic self-organizing maps with controlled growth for knowledge discovery. IEEE Trans. Neural Netw. 11(3), 601–614 (2000)

    Article  Google Scholar 

  2. Appiah, K., Hunter, A., Dickinson, P., Meng, H.: Implementation and applications of tri-state self-organizing maps on FPGA. IEEE Trans. Circ. Syst. Video Technol. 22(8), 1150–1160 (2012)

    Article  Google Scholar 

  3. Bandaragoda, T.R., De Silva, D., Alahakoon, D.: Automatic event detection in microblogs using incremental machine learning. J. Assoc. Inform. Sci. Technol. 68(10), 2394–2411 (2017)

    Article  Google Scholar 

  4. Bishop, C.M., Svensén, M., Williams, C.K.: GTM: the generative topographic mapping. Neural Comput. 10(1), 215–234 (1998)

    Article  Google Scholar 

  5. De Silva, D., Alahakoon, D.: Incremental knowledge acquisition and self learning from text. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2010)

    Google Scholar 

  6. De Silva, D., Alahakoon, D., Yu, X.: A data fusion technique for smart home energy management and analysis. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 4594–4600 (2016)

    Google Scholar 

  7. De Silva, D., et al.: Machine learning to support social media empowered patients in cancer care and cancer treatment decisions. PLoS One 13(10), 1–10 (2018)

    Google Scholar 

  8. De Silva, D., Yu, X., Alahakoon, D., Holmes, G.: A data mining framework for electricity consumption analysis from meter data. IEEE Trans. Industr. Inf. 7(3), 399–407 (2011)

    Article  Google Scholar 

  9. Dittenbach, M., Merkl, D., Rauber, A.: The growing hierarchical self-organizing map. In: International Joint Conference on Neural Networks (IJCNN), vol. 6, pp. 15–19 (2000)

    Google Scholar 

  10. Eliasmith, C.: How to Build a Brain. Oxford University Press, Oxford (2013)

    Book  Google Scholar 

  11. Frady, E.P., Kleyko, D., Sommer, F.T.: A theory of sequence indexing and working memory in recurrent neural networks. Neural Comput. 30, 1449–1513 (2018)

    Article  MathSciNet  Google Scholar 

  12. Hazan, H., Saunders, D.J., Sanghavi, D.T., Siegelmann, H.T., Kozma, R.: Unsupervised learning with self-organizing spiking neural networks. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–6 (2018)

    Google Scholar 

  13. Hinton, G., McClelland, J., Rumelhart, D.: Distributed representations. In: Rumelhart, D., McClelland, J. (eds.) Parallel Distributed Processing. Explorations in the Microstructure of Cognition - Volume 1 Foundations, pp. 77–109. MIT Press, Cambridge (1986)

    Google Scholar 

  14. Jayaratne, M., Alahakoon, D., De Silva, D., Yu, X.: Bio-inspired multisensory fusion for autonomous robots. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3090–3095 (2018)

    Google Scholar 

  15. Joshi, A., Halseth, J.T., Kanerva, P.: Language geometry using random indexing. In: de Barros, J.A., Coecke, B., Pothos, E. (eds.) QI 2016. LNCS, vol. 10106, pp. 265–274. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52289-0_21

    Chapter  Google Scholar 

  16. Kanerva, P.: Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors. Cogn. Comput. 1(2), 139–159 (2009)

    Article  Google Scholar 

  17. Kaski, S., Honkela, T., Lagus, K., Kohonen, T.: WEBSOM-self-organizing maps of document collections1. Neurocomputing 21(1–3), 101–117 (1998)

    Article  Google Scholar 

  18. Kleyko, D., Osipov, E.: Brain-like classifier of temporal patterns. In: International Conference on Computer and Information Sciences (ICCOINS), pp. 104–113 (2014)

    Google Scholar 

  19. Kleyko, D., Osipov, E., De Silva, D., Wiklund, U., Alahakoon, D.: Integer self-organizing maps for digital hardware. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)

    Google Scholar 

  20. Kleyko, D., Osipov, E., Gayler, R.: Recognizing permuted words with vector symbolic architectures: a cambridge test for machines. Proc. Comput. Sci. 88, 169–175 (2016)

    Article  Google Scholar 

  21. Kleyko, D., Osipov, E., Papakonstantinou, N., Vyatkin, V.: Hyperdimensional computing in industrial systems: the use-case of distributed fault isolation in a power plant. IEEE Access 6, 30766–30777 (2018)

    Article  Google Scholar 

  22. Kleyko, D., Osipov, E., Wiklund, U.: A hyperdimensional computing framework for analysis of cardiorespiratory synchronization during paced deep breathing. IEEE Access 7, 34403–34415 (2019)

    Article  Google Scholar 

  23. Kleyko, D., Rahimi, A., Gayler, R., Osipov, E.: Autoscaling bloom filter: controlling trade-off between true and false positives. Neural Comput. Appl. 1–10 (2019)

    Google Scholar 

  24. Kleyko, D., Rahimi, A., Rachkovskij, D., Osipov, E., Rabaey, J.: Classification and recall with binary hyperdimensional computing: trade-offs in choice of density and mapping characteristic. IEEE Trans. Neural Netw. Learn. Syst. 29(12), 5880–5898 (2018)

    Article  Google Scholar 

  25. Kohonen, T.: Self-Organizing Maps. Springer Series in Information Sciences. Springer, Heidelberg (2001). https://doi.org/10.1007/978-3-642-56927-2

    Book  MATH  Google Scholar 

  26. Kohonen, T., Somervuo, P.: Self-organizing maps of symbol strings. Neurocomputing 21(1–3), 19–30 (1998)

    Article  Google Scholar 

  27. Kusiak, A.: Smart manufacturing must embrace big data. Nat. News 544(7648), 23 (2017)

    Article  Google Scholar 

  28. Nallaperuma, D., De Silva, D., Alahakoon, D., Yu, X.: Intelligent detection of driver behavior changes for effective coordination between autonomous and human driven vehicles. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3120–3125 (2018)

    Google Scholar 

  29. Nawaratne, R., Bandaragoda, T., Adikari, A., Alahakoon, D., De Silva, D., Yu, X.: Incremental knowledge acquisition and self-learning for autonomous video surveillance. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 4790–4795 (2017)

    Google Scholar 

  30. Osipov, E., Kleyko, D., Legalov, A.: Associative synthesis of finite state automata model of a controlled object with hyperdimensional computing. In: Annual Conference of the IEEE Industrial Electronics Society (IECON), pp. 3276–3281 (2017)

    Google Scholar 

  31. Plate, T.A.: Holographic Reduced Representations: Distributed Representation for Cognitive Structures. Center for the Study of Language and Information (CSLI), Stanford (2003)

    Google Scholar 

  32. Quasto, U., Richter, M., Biemann, C.: Corpus portal for search in monolingual corpora. In: Fifth International Conference on Language Resources and Evaluation (LREC), pp. 1799–1802 (2006)

    Google Scholar 

  33. Rachkovskij, D.A.: Representation and processing of structures with binary sparse distributed codes. IEEE Trans. Knowl. Data Eng. 3(2), 261–276 (2001)

    Article  Google Scholar 

  34. Rahimi, A., et al.: High-dimensional Computing as a Nanoscalable Paradigm. IEEE Trans. Circ. Syst. I: Regul. Pap. 64(9), 2508–2521 (2017)

    Google Scholar 

  35. Rahimi, A., Kanerva, P., Benini, L., Rabaey, J.M.: Efficient biosignal processing using hyperdimensional computing: network templates for combined learning and classification of ExG signals. Proc. IEEE 107(1), 123–143 (2019)

    Article  Google Scholar 

  36. Rahimi, A., Kanerva, P., Rabaey, J.: A robust and energy efficient classifier using brain-inspired hyperdimensional computing. In: IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 64–69 (2016)

    Google Scholar 

  37. Rasanen, O., Kakouros, S.: Modeling dependencies in multiple parallel data streams with hyperdimensional computing. IEEE Signal Process. Lett. 21(7), 899–903 (2014)

    Article  Google Scholar 

  38. Recchia, G., Sahlgren, M., Kanerva, P., Jones, M.: Encoding sequential information in semantic space models: comparing holographic reduced representation and random permutation. Computat. Intell. Neurosci. 1–18 (2015)

    Article  Google Scholar 

  39. Santana, A., Morais, A., Quiles, M.: An alternative approach for binary and categorical self-organizing maps. In: International Joint Conference on Neural Networks (IJCNN), pp. 2604–2610 (2017)

    Google Scholar 

  40. Shah-Hosseini, H., Safabakhsh, R.: TASOM: a new time adaptive self-organizing map. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 33(2), 271–282 (2003)

    Article  Google Scholar 

  41. Vesanto, J., Alhoniemi, E.: Clustering of the self-organizing map. IEEE Trans. Neural Netw. 11(3), 586–600 (2000)

    Article  Google Scholar 

  42. Zolotukhin, M., Hamalainen, T., Juvonen, A.: Online anomaly detection by using n-gram model and growing hierarchical self-organizing maps. In: International Wireless Communications and Mobile Computing Conference (IWCMC), pp. 47–52 (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Luleå University of Technology, Luleå, Sweden

    Denis Kleyko, Evgeny Osipov & Valeriy Vyatkin

  2. La Trobe University, Melbourne, Australia

    Daswin De Silva & Damminda Alahakoon

  3. Umeå University, Umeå, Sweden

    Urban Wiklund

Authors
  1. Denis Kleyko
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Evgeny Osipov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daswin De Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Urban Wiklund
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Valeriy Vyatkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Damminda Alahakoon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denis Kleyko .

Editor information

Editors and Affiliations

  1. Microsoft Research, Redmond, WA, USA

    Nikolaj Bjørner

  2. AP Ershov Institute of Informatics Systems, Novosibirsk, Russia

    Irina Virbitskaite

  3. School of Computer Science, University of Manchester, Manchester, UK

    Andrei Voronkov

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kleyko, D., Osipov, E., De Silva, D., Wiklund, U., Vyatkin, V., Alahakoon, D. (2019). Distributed Representation of n-gram Statistics for Boosting Self-organizing Maps with Hyperdimensional Computing. In: Bjørner, N., Virbitskaite, I., Voronkov, A. (eds) Perspectives of System Informatics. PSI 2019. Lecture Notes in Computer Science(), vol 11964. Springer, Cham. https://doi.org/10.1007/978-3-030-37487-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-37487-7_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-37486-0

  • Online ISBN: 978-3-030-37487-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation