Abstract
In this paper, we propose a new information theoretic competitive learning method. In realizing competition, neither the winner-take-all algorithm nor the lateral inhibition is used. Instead, the new method is based upon mutual information maximization between input patterns and competitive units. In maximizing mutual information, the entropy of competitive units is increased as much as possible. This means that all competitive units must equally be used in our framework. Thus, no under-utilized neurons (dead neurons) are generated. We applied our method to a simple artificial data problem and an actual road classification problem. In both cases, experimental results confirmed that the new method can produce the final solutions almost independently of initial conditions, and classification performance is significantly improved.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
R. Linsker, “Self-organization in a perceptual network,” Computer, vol. 21, pp. 105–117, 1988.
R. Linsker, “How to generate ordered maps by maximizing the mutual information between input and output,” Neural Computation, vol. 1, pp. 402–411, 1989.
R. Kamimura and S. Nakanishi, “Hidden information maximization for feature detection and rule discovery,” Network, vol. 6, pp. 577–622, 1995.
R. Kamimura and S. Nakanishi, “Improving generalization performance by information minimization,” IEICE Transactions on Information and Systems, vol. E78-D, no. 2, pp. 163–173, 1995.
G. Deco, W. Finnof, and H. G. Zimmermann, “Unsupervised mutual information criterion for elimination of overtraining in supervised multiplayer networks,” Neural Computation, vol. 7, pp. 86–107, 1995.
Y. Akiyama and T. Furuya, “An extension of the back-propagation learning which performs entropy maximization as well as error minimization,” Tech. Rep. NC91-6, IEICE Technical Report, 1991.
R. Kamimura, T. Kamimura, and T. R. Shultz, “Information theoretic competitive learning and linguistic rule acquistion,” Transactions of the Japanese Society for Artificial Intelligence, vol. 16, no. 2, pp. 287–298, 2001.
R. Kamimura, T. Kamimura, and O. Uchida, “Flexible feature discovery and structural information,” Connection Science, vol. 13, no. 4, pp. 323–347, 2001.
D. DeSieno, “Adding a conscience to competitive learning,” in Proceedings of IEEE International Conference on Neural Networks, (San Diego), pp. 117–124, IEEE, 1988.
S. C. Ahalt, A. K. Krishnamurthy, P. Chen, and D. E. Melton, “Competitive learning algorithms for vector quantization,” Neural Networks, vol. 3, pp. 277–290, 1990.
L. Xu, “Rival penalized competitive learning for clustering analysis, RBF net, and curve detection,” IEEE Transaction on Neural Networks, vol. 4, no. 4, pp. 636–649, 1993.
A. Luk and S. Lien, “Properties of the generalized lotto-type competitive learning,” in Proceedings of International conference on neural information processing, (San Mateo: CA), pp. 1180–1185, Morgan Kaufmann Publishers, 2000.
M. Marc and M. V. Hulle, Faithful representations and topographic maps. New York: John Wiley and Sons, Inc, 2000.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kamimura, R. (2003). Competitive Learning by Information Maximization: Eliminating Dead Neurons in Competitive Learning. In: Kaynak, O., Alpaydin, E., Oja, E., Xu, L. (eds) Artificial Neural Networks and Neural Information Processing — ICANN/ICONIP 2003. ICANN ICONIP 2003 2003. Lecture Notes in Computer Science, vol 2714. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44989-2_13
Download citation
DOI: https://doi.org/10.1007/3-540-44989-2_13
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40408-8
Online ISBN: 978-3-540-44989-8
eBook Packages: Springer Book Archive