Abstract
This paper deals with nonlinear least-squares problems involving the fitting to data of parameterized analytic functions. For generic regression data, a general result establishes the countability, and under stronger assumptions finiteness, of the set of functions giving rise to critical points of the quadratic loss function. In the special case of what are usually called “single-hidden layer neural networks”, which are built upon the standard sigmoidal activation tanh(x) (or equivalently (1 +e −x)−1), a rough upper bound for this cardinality is provided as well.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
F. Albertini, E.D. Sontag and V. Maillot, Uniqueness of weights for neural networks, in:Artificial Neural Networks with Applications in Speech and Vision, ed. R. Mammone, Chapman and Hall, London, 1993, pp. 115–125.
E. Bierstone and P.D. Milman, Semianalytic and subanalytic sets, Inst. Hautes Études Sci. Publ. Math. 67, 1988, 5–42.
E.K. Blum, Approximation of Boolean functions by sigmoidal networks: Part I: XOR and other two-variable functions, Neural Computation 1, 1989, 532–540.
M. Brady, R. Raghavan and J. Slawny, Backpropagation fails to separate where perceptrons succeed, IEEE Trans. Circuits and Systems 36, 1989, 665–674.
J.B. Conway,Regular Algebra and Finite Machines, Chapman and Hall, London, 1971.
A. Gabrielov, Projections of semi-analytic sets, Functional Anal. Appl. 2, 1968, 282–291.
S.A. Goldman and M.J. Kearns, On the complexity of teaching,Proc. 4th ACM Workshop on Computational Learning Theory, July 1991, pp. 303–314.
M. Gori and A. Tesi, On the problem of local minima in back-propagation, Tech. Report RT-DSI 6/90, Univ. di Firenze, April 1990.
A.G. Khovanskii,Fewnomials, American Mathematical Society, Providence, RI, 1991.
J. Knight, A. Pillay and C. Steinhorn, Definable sets in ordered structures, II, Trans. Amer. Math. Soc. 295, 1986, 593–605.
A. Macintyre and E.D. Sontag, Finiteness results for sigmoidal “neural” networks, in:Proc. 25th Annual Symp. Theory Computing, San Diego, May 1993, pp. 325–334.
J.W. Milnor,Morse Theory, Princeton University Press, 1963.
R. Palais and C-l. Terng,Critical Point Theory and Submanifold Geometry, Springer, Berlin, New York, 1988.
T. Poston, C-N Lee, Y-J Choie and Y. Kwon, Local minima and backpropagation, in:Int. Joint Conf. Neural Networks, Seattle, IEEE Press, 1991, pp. 173–176.
E.D. Sontag,Mathematical Control Theory: Deterministic Finite Dimensional Systems, Springer, New York, 1990.
E.D. Sontag, Feedforward nets for interpolation and classification, J. Comp. Syst. Sci. 45, 1992, 20–48.
E.D. Sontag and H.J. Sussmann, Backpropagation can give rise to spurious local minima even for networks without hidden layers, Complex Systems 3, 1989, 91–106.
H.J. Sussmann, Real analytic desingularization and subanalytic sets: An elementary approach, Trans. Amer. Math. Soc. 317, 1990, 417–461.
H.J. Sussmann, Uniqueness of the weights for minimal feedforward nets with a given input-output map, Neural Networks, 5, 1992, 589–593.
L. van den Dries, A generalization of the Tarski-Seidenberg theorem, and some nondefinability results, Bull. AMS 15, 1986, 189–193.
L. van den Dries, Tame topology and 0-minimal structures, preprint, University of Illinois, Urbana, 1991–2.
L. van den Dries and C. Miller, On the real exponential field with restricted analytic functions, Israel J. Math., 85 1994, 19–56.
L. van den Dries, A. Macintyre and D. Marker, The elementary theory of restricted analytic fields with exponentiation, Annals of Math. 140, 1994, 183–205.
R.C. Williamson and U. Helmke, Approximation theoretic results for neural networks, in:Proceedings of the Australian Conference on Neural Networks, 1992, pp. 217–222. (Also Existence and uniqueness results for neural network approximations, IEEE Transactions on Neural Networks, 6, 1995, 2–13.)
A.J. Wilkie, Some model completeness results for expansions of the ordered field of reals by Pfaffian functions, preprint, Oxford, 1991, submitted.
A.J. Wilkie, Smooth 0-minimal theories and the model completeness of the real exponential field, preprint, Oxford, 1991, submitted.
Author information
Authors and Affiliations
Additional information
Supported in part by US Air Force Grant AFOSR-94-0293.
Rights and permissions
About this article
Cite this article
Stonag, E.D. Critical points for least-squares problems involving certain analytic functions, with applications to sigmoidal nets. Adv Comput Math 5, 245–268 (1996). https://doi.org/10.1007/BF02124746
Issue Date:
DOI: https://doi.org/10.1007/BF02124746