Skip to main content
Springer Nature Link
Log in

Min-Max Cost and Information Control in Multi-layered Neural Networks

  • Conference paper
  • First Online:
Proceedings of the Future Technologies Conference (FTC) 2022, Volume 1 (FTC 2022 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 559))

Included in the following conference series:

  • 1025 Accesses

Abstract

The present paper aims to propose a new method to minimize and maximize information and its cost, accompanied by the ordinary error minimization. All these computational procedures are operated as independently as possible from each other. This method aims to solve the contradiction in conventional computational methods in which many procedures are intertwined with each other, making it hard to compromise among them. In particular, we try to minimize information at the expense of cost, followed by information maximization, to reduce humanly biased information obtained through artificially created input variables. The new method was applied to the detection of relations between mission statements and firms’ financial performance. Though the relation between them has been considered one of the main factors for strategic planning in management, the past studies could only confirm very small positive relations between them. In addition, those results turned out to be very dependent on the operationalization and variable selection. The studies suggest that there may be some indirect and mediating variables or factors to internalize the mission statements in organizational members. If neural networks have an ability to infer those mediating variables or factors, new insight into the relation can be obtained. Keeping this in mind, the experiments were performed to infer some positive relations. The new method, based on minimizing the humanly biased effects from inputs, could produce linear, non-linear, and indirect relations, which could not be extracted by the conventional methods. Thus, this study shows a possibility for neural networks to interpret complex phenomena in human and social sciences, which, in principle, conventional models cannot deal with.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Hinton, G.E., McClelland, J.L., Rumelhart, D.E.: Distributed representations. In: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1: Foundations, pp. 77–109 (1986)

    Google Scholar 

  2. Rumelhart, D.E., Zipser, D.: Feature discovery by competitive learning. Cogn. Sci. 9, 75–112 (1985)

    Article  Google Scholar 

  3. Kohonen, T.: Self-Organization and Associative Memory. Springer, New York (1988). https://doi.org/10.1007/978-3-642-88163-3

    Book  MATH  Google Scholar 

  4. Kohonen, T.: Self-Organizing Maps. Springer, Heidelberg (1995). https://doi.org/10.1007/978-3-642-97610-0

    Book  MATH  Google Scholar 

  5. Xu, Y., Xu, L., Chow, T.W.S.: PPoSOM: a new variant of PolSOM by using probabilistic assignment for multidimensional data visualization. Neurocomputing 74(11), 2018–2027 (2011)

    Article  Google Scholar 

  6. Xu, L., Chow, T.W.S.: Multivariate data classification using PolSOM. In: Prognostics and System Health Management Conference (PHM-Shenzhen), pp. 1–4. IEEE (2011)

    Google Scholar 

  7. DeSieno, D.: Adding a conscience to competitive learning. In: IEEE International Conference on Neural Networks, vol. 1, pp. 117–124. Institute of Electrical and Electronics Engineers, New York (1988)

    Google Scholar 

  8. Lei, X.: Rival penalized competitive learning for clustering analysis, RBF net, and curve detection. IEEE Trans. Neural Netw. 4(4), 636–649 (1993)

    Article  Google Scholar 

  9. Choy, C.S., Siu, W.: A class of competitive learning models which avoids neuron underutilization problem. IEEE Trans. Neural Netw. 9(6), 1258–1269 (1998)

    Article  Google Scholar 

  10. Banerjee, A., Ghosh, J.: Frequency-sensitive competitive learning for scalable balanced clustering on high-dimensional hyperspheres. IEEE Trans. Neural Netw. 15(3), 702–719 (2004)

    Article  Google Scholar 

  11. Van Hulle, M.M.: Entropy-based kernel modeling for topographic map formation. IEEE Trans. Neural Netw. 15(4), 850–858 (2004)

    Article  Google Scholar 

  12. Hubel, D.H., Wisel, T.N.: Receptive fields, binocular interaction and functional architecture in cat’s visual cortex. J. Physiol. 160, 106–154 (1962)

    Article  Google Scholar 

  13. Bienenstock, E.L., Cooper, L.N., Munro, P.W.: Theory for the development of neuron selectivity. J. Neurosci. 2, 32–48 (1982)

    Article  Google Scholar 

  14. Schoups, A., Vogels, R., Qian, N., Orban, G.: Practising orientation identification improves orientation coding in V1 neurons. Nature 412(6846), 549–553 (2001)

    Article  Google Scholar 

  15. Ukita, J.: Causal importance of low-level feature selectivity for generalization in image recognition. Neural Netw. 125, 185–193 (2020)

    Article  Google Scholar 

  16. Nguyen, A., Yosinski, J., Clune, J.: Understanding neural networks via feature visualization: a survey. In: Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., Müller, K.-R. (eds.) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. LNCS (LNAI), vol. 11700, pp. 55–76. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28954-6_4

    Chapter  Google Scholar 

  17. Montavon, G., Binder, A., Lapuschkin, S., Samek, W., Müller, K.-R.: Layer-wise relevance propagation: an overview. In: Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., Müller, K.-R. (eds.) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. LNCS (LNAI), vol. 11700, pp. 193–209. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28954-6_10

    Chapter  Google Scholar 

  18. Morcos, A.S., Barrett, D.G.T., Rabinowitz, N.C., Botvinick, M.: On the importance of single directions for generalization. Stat 1050, 15 (2018)

    Google Scholar 

  19. Leavitt, M.L., Morcos, A.: Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs. arXiv preprint arXiv:2003.01262 (2020)

  20. Arpit, D., Zhou, Y., Ngo, H., Govindaraju, V.: Why regularized auto-encoders learn sparse representation? In: International Conference on Machine Learning, pp. 136–144. PMLR (2016)

    Google Scholar 

  21. Goodfellow, I., Bengio, Y., Courville, A.: Regularization for deep learning. Deep Learn. 216–261 (2016)

    Google Scholar 

  22. Kukačka, J., Golkov, V., Cremers, D.: Regularization for deep learning: a taxonomy. arXiv preprint arXiv:1710.10686 (2017)

  23. Wu, C., Gales, M.J.F., Ragni, A., Karanasou, P., Sim, K.C.: Improving interpretability and regularization in deep learning. IEEE/ACM Trans. Audio Speech Lang. Process. 26(2), 256–265 (2017)

    Article  Google Scholar 

  24. Linsker, R.: Self-organization in a perceptual network. Computer 21(3), 105–117 (1988)

    Article  Google Scholar 

  25. Linsker, R.: Local synaptic rules suffice to maximize mutual information in a linear network. Neural Comput. 4, 691–702 (1992)

    Article  Google Scholar 

  26. Linsker, R.: Improved local learning rule for information maximization and related applications. Neural Netw. 18, 261–265 (2005)

    Article  Google Scholar 

  27. Moody, J., Hanson, S., Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. Adv. Neural Inf. Process. Syst. 4, 950–957 (1995)

    Google Scholar 

  28. Fan, F.-L., Xiong, J., Li, M., Wang, G.: On interpretability of artificial neural networks: a survey. IEEE Trans. Radiat. Plasma Med. Sci. 5(6), 741–760 (2021)

    Article  Google Scholar 

  29. Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)

    Article  Google Scholar 

  30. Hu, J., et al.: Architecture disentanglement for deep neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 672–681 (2021)

    Google Scholar 

  31. Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: learning augmentation strategies from data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113–123 (2019)

    Google Scholar 

  32. Gupta, A., Murali, A., Gandhi, D., Pinto, L.: Robot learning in homes: improving generalization and reducing dataset bias. arXiv preprint arXiv:1807.07049 (2018)

  33. Kim, B., Kim, H., Kim, K., Kim, S., Kim, J.: Learning not to learn: training deep neural networks with biased data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9012–9020 (2019)

    Google Scholar 

  34. Wang, T., Zhao, J., Yatskar, M., Chang, K.W., Ordonez, V.: Balanced datasets are not enough: estimating and mitigating gender bias in deep image representations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5310–5319 (2019)

    Google Scholar 

  35. Hendricks, L.A., Burns, K., Saenko, K., Darrell, T., Rohrbach, A.: Women also snowboard: overcoming bias in captioning models. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 771–787 (2018)

    Google Scholar 

  36. Cortés-Sánchez, J.D., Rivera, L.: Mission statements and financial performance in Latin-American firms. Verslas: Teorija ir praktika/Business Theory Pract. 20, 270–283 (2019)

    Google Scholar 

  37. Bart, C.K., Bontis, N., Taggar, S.: A model of the impact of mission statements on firm performance. Manag. Decis. 39(1), 19–35 (2001)

    Article  Google Scholar 

  38. Hirota, S., Kubo, K., Miyajima, H., Hong, P., Park, Y.W.: Corporate mission, corporate policies and business outcomes: evidence from japan. Manag. Decis. (2010)

    Google Scholar 

  39. Alegre, I., Berbegal-Mirabent, J., Guerrero, A., Mas-Machuca, M.: The real mission of the mission statement: a systematic review of the literature. J. Manag. Organ. 24(4), 456–473 (2018)

    Article  Google Scholar 

  40. Atrill, P., Omran, M., Pointon, J.: Company mission statements and financial performance. Corp. Ownersh. Control. 2(3), 28–35 (2005)

    Article  Google Scholar 

  41. Vandijck, D., Desmidt, S., Buelens, M.: Relevance of mission statements in flemish not-for-profit healthcare organizations. J. Nurs. Manag. 15(2), 131–141 (2007)

    Article  Google Scholar 

  42. Desmidt, S., Prinzie, A., Decramer, A.: Looking for the value of mission statements: a meta-analysis of 20 years of research. Manag. Decis. (2011)

    Google Scholar 

  43. Macedo, I.M., Pinho, J.C., Silva, A.M.: Revisiting the link between mission statements and organizational performance in the non-profit sector: the mediating effect of organizational commitment. Eur. Manag. J. 34(1), 36–46 (2016)

    Article  Google Scholar 

  44. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, Hoboken (1991)

    Book  Google Scholar 

  45. Buciluǎ, C., Caruana, R., Niculescu-Mizil, A.: Model compression. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535–541. ACM (2006)

    Google Scholar 

  46. Ba, J., Caruana, R.: Do deep nets really need to be deep? In: Advances in Neural Information Processing Systems, pp. 2654–2662 (2014)

    Google Scholar 

  47. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  48. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Hints for thin deep nets. In: Proceedings of ICLR, Fitnets (2015)

    Google Scholar 

  49. Luo, P., Zhu, Z., Liu, Z., Wang, X., Tang, X.: Face model compression by distilling knowledge from neurons. In: Thirtieth AAAI Conference on Artificial Intelligence (2016)

    Google Scholar 

  50. Neill, J.O.: An overview of neural network compression. arXiv preprint arXiv:2006.03669 (2020)

  51. Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: a survey. Int. J. Comput. Vis. 129(6), 1789–1819 (2021)

    Article  Google Scholar 

  52. Cheng, Y., Wang, D., Zhou, P., Zhang, T.: A survey of model compression and acceleration for deep neural networks (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Tokai University and Kumamoto Drone Technology and Development Foundation, 2880 Kamimatsuo Nishi-ku, Kumamoto, 861-5289, Japan

    Ryotaro Kamimura

  2. Tokyo Polytechnic University, 1583 Iiyama, Atsugi, Kanagawa, 243-0297, Japan

    Ryozo Kitajima

Authors
  1. Ryotaro Kamimura
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryozo Kitajima
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryotaro Kamimura .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kamimura, R., Kitajima, R. (2023). Min-Max Cost and Information Control in Multi-layered Neural Networks. In: Arai, K. (eds) Proceedings of the Future Technologies Conference (FTC) 2022, Volume 1. FTC 2022 2022. Lecture Notes in Networks and Systems, vol 559. Springer, Cham. https://doi.org/10.1007/978-3-031-18461-1_1

Download citation

Publish with us

Policies and ethics