Skip to main content
SpringerLink
Log in

The Impact of Constant Field of Attention on Properties of Contextual Neural Networks

  • Conference paper
  • First Online:
Book cover Intelligent Information and Database Systems (ACIIDS 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12034))

Included in the following conference series:

Abstract

Applications of Artificial Neural Networks are used with success in many fields such as e.g. medicine, economy, science and entertainment. But most of those models processes all input signals without attention directing mechanisms. In this paper contextual neural networks are considered which are using multi-step conditional aggregation functions to direct attention of neurons during data processing. It is verified if aggregation function with constant field of attention (CFA) can help to build classification models of higher accuracy and of lower activity of hidden connections than aggregation function with dynamic field of attention (SIF). Experiments are performed with use of the H2O machine learning framework implementing Generalized Backpropagation Algorithm for selected benchmark problems from UCI ML repository and real-life ALL-AML leukemia gene expression data. Presented analysis of results gives important clues on the most promising directions of research on contextual neural networks and indicates possible improvements of the H2O framework.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Tsai, Y.-C., et al.: FineNet: a joint convolutional and recurrent neural network model to forecast and recommend anomalous financial items. In: Proceedings of the 13th ACM Conference on Recommender Systems RecSys 2019, pp. 536–537. ACM, New York (2019)

    Google Scholar 

  2. Chen, S., Zhang, S., Shang, J., Chen, B., Zheng, N.: Brain-inspired cognitive model with attention for self-driving cars. IEEE Trans. Cogn. Dev. Syst. 11(1), 13–25 (2019)

    Article  Google Scholar 

  3. Suleymanova, I., et al.: A deep convolutional neural network approach for astrocyte detection. Sci. Rep. 8(12878), 1–7 (2018)

    Google Scholar 

  4. Guest, D., Cranmer, K., Whiteson, D.: Deep learning and its application to LHC physics. Annu. Rev. Nucl. Part. Sci. 68, 1–22 (2018)

    Article  Google Scholar 

  5. Liu, L., et al.: Automatic skin binding for production characters with deep graph networks. ACM Trans. Graph. (SIGGRAPH) 38(4), 1–12 (2019). Article 114

    Google Scholar 

  6. Gao, D., Li, X., Dong, Y., Peers, P., Xu, K., Tong, X.: Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images. ACM Trans. Graph. (SIGGRAPH) 38(4), 1–15 (2019). Article 134

    Article  Google Scholar 

  7. Gong, K., et al.: Iterative PET image reconstruction using convolutional neural network representation. IEEE Trans. Med. Imaging 38(3), 675–685 (2019)

    Article  Google Scholar 

  8. Higgins, I., et al.: Beta-VAE: learning basic visual concepts with a constrained variational framework. In: International Conference on Learning Representations, ICLR 2017, vol. 2, no. 5, pp. 1–22 (2017)

    Google Scholar 

  9. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations, ICLR 2018, pp. 1–26 (2018)

    Google Scholar 

  10. Huang, X., Tan, H., Lin, G., Tian, Y.: A LSTM-based bidirectional translation model for optimizing rare words and terminologies. In: 2018 IEEE International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 5077–5086. IEEE (2018)

    Google Scholar 

  11. Athiwaratkun, B., Stokes, J.W.: Malware classification with LSTM and GRU language models and a character-level CNN. In: Proceedings 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2482–2486. IEEE (2017)

    Google Scholar 

  12. Amato, F., Mazzocca, N., Moscato, F., Vivenzio, E.: Multilayer perceptron: an intelligent model for classification and intrusion detection. In: 31st International Conference on Advanced Information Networking and Applications Workshops (WAINA), pp. 686–691. IEEE, Taipei (2017)

    Google Scholar 

  13. Dozono, H., Niina, G., Araki, S.: Convolutional self organizing map. In: 2016 IEEE International Conference on Computational Science and Computational Intelligence (CSCI), pp. 767–771. IEEE (2016)

    Google Scholar 

  14. Mel, B.W.: The Clusteron: toward a simple abstraction for a complex neuron. In: Advances in Neural Information Processing Systems, vol. 4, pp. 35–42. Morgan Kaufmann (1992)

    Google Scholar 

  15. Spratling, M.W., Hayes, G.: Learning synaptic clusters for nonlinear dendritic processing. Neural Process. Lett. 11, 17–27 (2000)

    Article  Google Scholar 

  16. Gupta, M.: Correlative type higher-order neural units with applications. In: IEEE International Conference on Automation and Logistics, ICAL 2008, pp. 715–718 (2008). Springer Computer Science

    Google Scholar 

  17. Vanrullen, R., Koch, C.: Visual selective behavior can be triggered by a feed-forward process. J. Cogn. Neurosci. 15, 209–217 (2003)

    Article  Google Scholar 

  18. Huk, M.: Learning distributed selective attention strategies with the Sigma-if neural network. In: Akbar, M., Hussain, D. (eds.) Advances in Computer Science and IT, pp. 209–232. InTech, Vukovar (2009)

    Google Scholar 

  19. Huk, M.: Notes on the generalized backpropagation algorithm for contextual neural networks with conditional aggregation functions. J. Intell. Fuzzy Syst. 32, 1365–1376 (2017)

    Article  Google Scholar 

  20. Huk, M.: Backpropagation generalized delta rule for the selective attention sigma-if artificial neural network. Int. J. App. Math. Comp. Sci. 22, 449–459 (2012)

    Article  MathSciNet  Google Scholar 

  21. Huk, M., Pietraszko, J.: Contextual neural network based spectrum prediction for cognitive radio. In: 4th International Conference on Future Generation Communication Technology (FGCT 2015), pp. 1–5. IEEE Computer Society, London (2015)

    Google Scholar 

  22. Huk, M.: Non-uniform initialization of inputs groupings in contextual neural networks. In: Nguyen, N.T., Gaol, F.L., Hong, T.-P., Trawiński, B. (eds.) ACIIDS 2019. LNCS (LNAI), vol. 11432, pp. 420–428. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-14802-7_36

    Chapter  Google Scholar 

  23. Huk, M.: Training contextual neural networks with rectifier activation functions: role and adoption of sorting methods. J. Intell. Fuzzy Syst. 38, 1–10 (2019)

    Google Scholar 

  24. Huk, M.: Weights ordering during training of contextual neural networks with generalized error backpropagation: importance and selection of sorting algorithms. In: Nguyen, N.T., Hoang, D.H., Hong, T.-P., Pham, H., Trawiński, B. (eds.) ACIIDS 2018. LNCS (LNAI), vol. 10752, pp. 200–211. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75420-8_19

    Chapter  Google Scholar 

  25. Szczepanik, M., Jóźwiak, I.: Fingerprint recognition based on minutes groups using directing attention algorithms. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012. LNCS (LNAI), vol. 7268, pp. 347–354. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29350-4_42

    Chapter  Google Scholar 

  26. Szczepanik, M., Jóźwiak, I.: Data management for fingerprint recognition algorithm based on characteristic points’ groups. In: Pechenizkiy, M., Wojciechowski, M. (eds.) New Trends in Databases and Information Systems. Foundations of Computing and Decision Sciences, 38(2), pp. 123–130. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-32518-2_40

    Chapter  Google Scholar 

  27. Huk, M.: Context-related data processing with artificial neural networks for higher reliability of telerehabilitation systems. In: 17th International Conference on E-Health Networking, Application & Services (HealthCom), pp. 217–221. IEEE Computer Society, Boston (2015)

    Google Scholar 

  28. Huk, M., Kwiatkowski, J., Konieczny, D., Kędziora, M., Mizera-Pietraszko, J.: Context-sensitive text mining with fitness leveling genetic algorithm. In: 2015 IEEE 2nd International Conference on Cybernetics (CYBCONF), Gdynia, Poland, Electronic Publication, pp. 1–6 (2015). ISBN: 978-1-4799-8321-6

    Google Scholar 

  29. Privitera, C.M., Azzariti, M., Stark, L.W.: Locating regions-of-interest for the Mars Rover expedition. Int. J. Remote Sens. 21, 3327–3347 (2000)

    Article  Google Scholar 

  30. Dua, D., Graff, C.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine, CA (2019). http://archive.ics.uci.edu/ml

  31. Golub, T.R., et al.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999)

    Article  Google Scholar 

  32. Huk, M., Kwaśnicka, H.: The concept and properties of Sigma-if neural network. In: Ribeiro, B., Albrecht, R.F., Dobnikar, A., Pearson, D.W., Steele, N.C. (eds.) Adaptive and Natural Computing Algorithms, ICANNGA 2005, pp. 13–17. Springer Computer Science, Vienna (2005). https://doi.org/10.1007/3-211-27389-1_4

    Chapter  Google Scholar 

  33. Huk, M.: Manifestation of selective attention in Sigma-if neural network. In: 2nd International Symposium Advances in Artificial Intelligence and Applications, International Multiconference on Computer Science and Information Technology, IMCSIT/AAIA 2007, vol. 2, pp. 225–236 (2007)

    Google Scholar 

  34. Huk, M.: Sigma-if neural network as the use of selective attention technique in classification and knowledge discovery problems solving. Ann. UMCS Sectio AI Informatica 4(2), 121–131 (2006)

    Google Scholar 

  35. Raczkowski, D., Canning, A., Wang, L.: Thomas Fermi charge mixing for obtaining self-consistency in density functional calculations. Phys. Rev. B 64(12), 121101–121105 (2001)

    Article  Google Scholar 

  36. Glosser, C., Piermarocchi, C., Shanker, B.: Analysis of dense quantum dot systems using a self-consistent Maxwell-Bloch framework. In: Proceedings of 2016 IEEE International Symposium on Antennas and Propagation (USNC-URSI), Puerto Rico, pp. 1323–1324. IEEE (2016)

    Google Scholar 

  37. H2O.ai: H2O Version 3.24.0.4, Fast Scalable Machine Learning API For Smarter Applications (2019). http://h2o-release.s3.amazonaws.com/h2o/rel-yates/4/index.html

  38. Janusz, B.J., Wołk, K.: Implementing contextual neural networks in distributed machine learning framework. In: Nguyen, N.T., Hoang, D.H., Hong, T.-P., Pham, H., Trawiński, B. (eds.) ACIIDS 2018. LNCS (LNAI), vol. 10752, pp. 212–223. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75420-8_20

    Chapter  Google Scholar 

  39. Wołk, K., Burnell, E.: Implementation and analysis of contextual neural networks in H2O framework. In: Nguyen, N.T., Gaol, F.L., Hong, T.-P., Trawiński, B. (eds.) ACIIDS 2019. LNCS (LNAI), vol. 11432, pp. 429–440. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-14802-7_37

    Chapter  Google Scholar 

  40. Bouckaert, R.R., Frank, E.: Evaluating the replicability of significance tests for comparing learning algorithms. In: Dai, H., Srikant, R., Zhang, C. (eds.) PAKDD 2004. LNCS (LNAI), vol. 3056, pp. 3–12. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24775-3_3

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Science Applications International Corporation, Reston, USA

    Erik Dawid Burnell

  2. The Wroclaw Institute of Spatial Information and Artificial Intelligence, Wroclaw, Poland

    Krzysztof Wołk

  3. Faculty of Computer Science and Management, Wroclaw University of Science and Technology, Wroclaw, Poland

    Krzysztof Waliczek & Rafał Kern

Authors
  1. Erik Dawid Burnell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Krzysztof Wołk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Krzysztof Waliczek
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rafał Kern
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Krzysztof Waliczek .

Editor information

Editors and Affiliations

  1. Department of Applied Informatics, Wrocław University of Science and Technology, Wrocław, Poland

    Ngoc Thanh Nguyen

  2. King Mongkut’s Institute of Technology Ladkrabang, Bangkok, Thailand

    Kietikul Jearanaitanakij

  3. Faculty of Computer Science and Information, University Teknologi Malaysia, Kuala Lumpur, Malaysia

    Ali Selamat

  4. Department of Applied Informatics, Wrocław University of Science and Technology, Wrocław, Poland

    Bogdan Trawiński

  5. King Mongkut’s Institute of Technology Ladkrabang, Bangkok, Thailand

    Suphamit Chittayasothorn

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Burnell, E.D., Wołk, K., Waliczek, K., Kern, R. (2020). The Impact of Constant Field of Attention on Properties of Contextual Neural Networks. In: Nguyen, N., Jearanaitanakij, K., Selamat, A., Trawiński, B., Chittayasothorn, S. (eds) Intelligent Information and Database Systems. ACIIDS 2020. Lecture Notes in Computer Science(), vol 12034. Springer, Cham. https://doi.org/10.1007/978-3-030-42058-1_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-42058-1_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-42057-4

  • Online ISBN: 978-3-030-42058-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation