Abstract
Combinatorial optimization problems share an interesting property with spin glass systems in that their state spaces can exhibit ultrametric structure. We use sampling methods to analyse the error surfaces of feedforward multi-layer perceptron neural networks learning encoder problems. The third order statistics of these points of attraction are examined and found to be arranged in a highly ultrametric way. This is a unique result for a finite, continuous parameter space. The implications of this result are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Chen, A. M., Lu, H. and Hecht-Nielsen, R.: On the geometry of feedforward neural network error surfaces, Neural Comput., 5(6) (1993), 910–927.
Frasconi, P., Gori, M. and Tesi, A.: Successes and failures of backpropagation: a theoretical investigation, In: O. Omidvar, and C. L. Wilson, (eds), Progress in Neural Networks, Ablex Publishing, Norwood, NJ, 1993.
Gallagher, M. and Downs, T.: On ultrametricity in feedforward neural network error surfaces, In: T. Downs, et al., (eds) Australian Conference on Neural Networks (ACNN'98), pp. 236–240. University of Queensland, 1998.
Gallagher, M., Downs, T. and Wood, I.: Ultrametric structure in autoencoder error surfaces, In: L. Niklasson et al., (eds), Proceedings International Conference on Neural Networks (ICANN'98), pp. 177–182, London, 1998, Springer.
Hamey, L. G. C.: XOR has no local minima: A case study in neural network error surface analysis, Neural Networks, 11(4) (1998), 669–681.
Hertz, J., Krogh, A. and Palmer, R. G.: Introduction to the Theory of Neural Computation, Addison-Wesley, Redwood City, CA, 1991.
Kirkpatrick, S. and Toulouse, G.: Configuration space analysis of travelling salesman problems, Journal de Physique (Paris), 46 (1985), 1277–1292.
Kruglyak, L.: How to solve the N bit encoder problem with just two hidden units, Neural Comput., 2(4) (1990), 399–401.
Lister, R.: Visualizing weight dynamics in the N-2–N encoder, In: IEEE International Conference on Neural Networks, volume 2, pp. 684–689, Piscataway, NJ, 1993. IEEE.
Lister, R.: Fractal strategies for neural network scaling, In: A. Michael Arbib, (ed), The Handbook of Brain Theory and Neural Networks, pp. 403–405. MIT Press, Cambridge, MA, 1995.
Mézard, M., Parisi, G., Sourlas, N., Toulouse, G. and Virasoro, M.: Nature of the spinglass phase, Physical Review Letters, 52(13) (1984), 1156–1159.
Parge, N. and Virasoro, M. A.: The ultrametric organization of memories in a neural network, Journal de Physique (Paris), 47(11) (1986), 1857–1864.
Rammal, R., Toulouse, G. and Virasoro, M. A.: Ultrametricity for physicists, Reviews of Modern Physics, 58(3) (1986), 765–788.
Saad, D. and Solla, S. A.: Exact solution for on-line learning in multilayer neural networks, Physical Review Letters, 74(21) (1995), 4337–4340.
Solla, S. A., Sorkin, G. B. and White, S. R.: Configuration space analysis for optimization problems, In: E. Bienenstock et al., (eds), Disordered Systems and Biological Organization, NATO ASI Series, volume F20, pp. 283–293, Berlin, New York, 1986, Springer.
Wolpert, D.: The lack of a priori distinctions between learning algorithms, Neural Comput., 8(7) (1996), 1341–1390.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Gallagher, M., Downs, T. & Wood, I. Empirical Evidence for Ultrametric Structure in Multi-layer Perceptron Error Surfaces. Neural Processing Letters 16, 177–186 (2002). https://doi.org/10.1023/A:1019956303894
Issue Date:
DOI: https://doi.org/10.1023/A:1019956303894