Abstract
In this paper, we propose partial activation to simplify complex neural networks. For choosing important elements in a network, we develop a fully supervised competitive learning that can deal with any targets. This approach is an extension of competitive learning to a more general one, including supervised learning. Because competitive learning focuses on an important competitive unit, all the other competitive units are of no use. Thus, the number of connection weights to be updated can be reduced to a minimum point when we use competitive learning. We apply the method to the XOR problem to show that learning is possible with good interpretability of internal representations. Then, we apply the method to a student survey. In the problem, we try to show that the new method can produce connection weights that are more stable than those produced by BP. In addition, we show that, though connection weights are quite similar to those produced by linear regression analysis, generalization performance can be improved by changing the number of competitive units.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Rumelhart, D.E., Zipser, D.: Feature discovery by competitive learning. In: Rumelhart, D.E., G.E.H., et al. (eds.) Parallel Distributed Processing vol. 1, pp. 151–193, MIT Press, Cambridge (1986)
Grossberg, S.: Competitive learning: from interactive activation to adaptive resonance. Cognitive Science 11, 23–63 (1987)
DeSieno, D.: Adding a conscience to competitive learning. In: Proceedings of IEEE International Conference on Neural Networks, San Diego, pp. 117–124. IEEE Computer Society Press, Los Alamitos (1988)
Ahalt, S.C., Krishnamurthy, A.K., Chen, P., Melton, D.E.: Competitive learning algorithms for vector quantization. Neural Networks 3, 277–290 (1990)
Xu, L.: Rival penalized competitive learning for clustering analysis, RBF net, and curve detection. IEEE Transaction on Neural Networks 4(4), 636–649 (1993)
Luk, A., Lien, S.: Properties of the generalized lotto-type competitive learning. In: Proceedings of International conference on neural information processing, pp. 1180–1185. Morgan Kaufmann Publishers, San Mateo, CA (2000)
Hulle, M.M.V.: The formation of topographic maps that maximize the average mutual information of the output responses to noiseless input signals. Neural Computation 9(3), 595–606 (1997)
Kamimura, R.: Information theoretic competitive learning in self-adaptive multi-layered networks. Connection Science 13(4), 323–347 (2003)
Kamimura, R.: Information-theoretic competitive learning with inverse euclidean distance. Neural Processing Letters 18, 163–184 (2003)
Reed, R.: Pruning algorithms-a survey. IEEE Transactions on Neural Networks 5, 740–747 (1993)
Fritzke, B.: Growing cell structures – a self-organizing network for unsupervised and supervised learning. Neural Networks 7(9), 1441–1460 (1994)
Gatlin, L.L.: Information Theory and Living Systems. Columbia University Press (1972)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kamimura, R. (2007). Partially Activated Neural Networks by Controlling Information . In: de Sá, J.M., Alexandre, L.A., Duch, W., Mandic, D. (eds) Artificial Neural Networks – ICANN 2007. ICANN 2007. Lecture Notes in Computer Science, vol 4668. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74690-4_49
Download citation
DOI: https://doi.org/10.1007/978-3-540-74690-4_49
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74689-8
Online ISBN: 978-3-540-74690-4
eBook Packages: Computer ScienceComputer Science (R0)