Skip to main content
SpringerLink
Log in

A Constructive Approach to Calculating Lower Entropy Bounds

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

This paper presents a constructive approach to estimating the size of a neural network necessary to solve a given classification problem. The results are derived using an information entropy approach in the context of limited precision integer weights. Such weights are particularly suited for hardware implementations since the area they occupy is limited, and the computations performed with them can be efficiently implemented in hardware. The considerations presented use an information entropy perspective and calculate lower bounds on the number of bits needed in order to solve a given classification problem. These bounds are obtained by approximating the classification hypervolumes with the volumes of several regular (i.e., highly symmetric) n-dimensional bodies. The bounds given here allow the user to choose the appropriate size of a neural network such that: (i) the given classification problem can be solved, and (ii) the network architecture is not oversized. All considerations presented take into account the restrictive case of limited precision integer weights, and therefore can be directly applied when designing VLSI implementations of neural networks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. M.A. Arbib (ed.), The Handbook of Brain Theory and Neural Networks, MIT Press: Cambridge, Mass., 1995.

    Google Scholar 

  2. E. Fiesler and R. Beale (eds), Handbook of Neural Computation, Oxford Univ. Press and the Inst. of Physics Publ.: New York, NY, 1996.

    Google Scholar 

  3. V. Beiu, “Entropy bounds for classification algorithms”, Neural Network World, Vol. 6, pp. 497–505, 1996.

    Google Scholar 

  4. S. Draghici and I.K. Sethi, “On the possibilities of the limited precision weights neural networks in classification problems”, in J. Mira, R. Moreno-Díaz and J. Cabestany (eds) Biological and Artificial Computation: From Neuroscience to Technology, Lecture Notes in Computer Science 1240, pp. 753–762, Springer-Verlag: Berlin, June 1997.

    Google Scholar 

  5. V. Beiu and T. De Pauw, “Tight bounds on the size of neural networks for classification problems”, in J. Mira, R. Moreno-Díaz and J. Cabestany (eds) Biological and Artificial Computation: From Neuroscience to Technology, Lecture Notes in Computer Science 1240, pp. 743–752, Springer-Verlag: Berlin, June 1997.

    Google Scholar 

  6. V. Beiu, “Digital integrated circuit implementations”, Chapter E1.4 in [2].

  7. V. Beiu, VLSI Complexity of Discrete Neural Networks, Gordon and Breach: Newark, N.J., 1998.

    Google Scholar 

  8. E.I. Neciporuk, “The synthesis of networks from threshold elements”, Soviet Mathematics - Doklady, Vol. 5, pp. 163–166, 1964 (English transl. in Automation Express, Vol. 7, pp. 27-32 and pp. 35-39, 1964).

    Google Scholar 

  9. O.B. Lupanov, “The synthesis of circuits from threshold elements”, Problemy Kibernetiki, Vol. 20, pp. 109–140, 1973.

    Google Scholar 

  10. J. Bruck and R. Smolensky, “Polynomial threshold functions, AC 0 functions and spectral norms”, SIAM J. Computing, Vol. 21, pp. 33–42, 1992.

    Google Scholar 

  11. K.-Y. Siu, V. Roychowdhury and T. Kailath, “Depth-size tradeoffs for neural computations”, IEEE Trans. on Comp., Vol. C-40, pp. 1402–1412, 1991.

    Google Scholar 

  12. E.B. Baum, “On the capabilities of multilayer perceptrons”, J. Complexity, Vol. 4, pp. 193–215, 1988.

    Google Scholar 

  13. S.-C. Huang and Y.-F. Huang, “Bounds on the number of hidden neurons of multilayer perceptrons in classification and recognition”, IEEE Trans. on Neural Networks, Vol. NN-2, pp. 47–55, 1991.

    Google Scholar 

  14. M. Arai, “Bounds on the number of hidden units in binary-valued three-layer neural networks”, Neural Networks, Vol. 6, pp. 855–860, 1993.

    Google Scholar 

  15. M.H. Hassoun, Fundamentals of Artificial Neural Networks, MIT Press: Cambridge, Mass., 1995.

    Google Scholar 

  16. H. Paugam-Moisy, “Optimisation des réseaux des neurones artificiels”, PhD dissertation, Laboratoire de l'Informatique du Parallélisme LIP-IMAG (http://www.enslyon. fr/LIP/publis.us.html), École Normale Supérieure de Lyon, 46 Allée d'Italie, 69364 Lyon, France, 1992.

  17. Y.S. Abu-Mostafa, “Connectivity versus entropy”, in D.Z. Anderson (ed.) Neural Information Processing Systems, pp. 1–8, Inst. of Physics Publ.: New York, NY, 1988.

    Google Scholar 

  18. R.C. Williamson, “ɛ-entropy and the complexity of feedforward neural networks”, in R.P. Lippmann, J.E. Moody and D.S. Touretzky (eds) Advances in Neural Information Processing Systems 3, pp. 946–952, Morgan Kaufmann: San Mateo, Calif., 1991.

    Google Scholar 

  19. V. Beiu and J.G. Taylor, “VLSI-optimal neural network learning algorithm”, in D.W. Pearson, N.C. Steele and R.F. Albrecht (eds) Artificial Neural Nets and Genetic Algorithms, pp. 61–64, Springer-Verlag: Vienna, April 1995.

    Google Scholar 

  20. V. Beiu and J.G. Taylor, “Direct synthesis of neural networks”, in Proc. of the Intl. Conf. on Microelectronics for Neural Networks, pp. 257–264, IEEE CS Press: Los Alamitos, Calif., February 1996.

    Google Scholar 

  21. A.H. Khan and E.L. Hines, “Integer-weight neural networks”, Electronic Lett., Vol. 30, pp. 1237–1238, 1994; see also: A.H. Khan, Feedforward neural networks with constrained weights, PhD dissertation, Intelligent Sys. Lab., Dept. of Eng., Warwick Univ., Coventry CV4 7AL, UK, 1996 (http://www.eng.warwick.ac.uk/~es2029).

    Google Scholar 

  22. I. Parberry, Circuit Complexity and Neural Networks, MIT Press: Cambridge, Mass., 1994.

    Google Scholar 

  23. P. Raghavan, “Learning in threshold networks: a computational model and applications”, IBM Research Center, Tech. Rep. RC 13859, 1988; also in Proc. of the 1st Workshop on Computational Learning Theory, pp. 19–27, ACM Press: New York, NY, 1988.

    Google Scholar 

  24. K.J. Lang and M.J. Witbrock, “Learning to tell two spirals apart,” in Proc. of the 1988 Connectionist Models Summer School, pp. 52–59, Morgan Kaufmann: San Mateo, Calif., 1988.

    Google Scholar 

  25. V. Beiu, “Reduced complexity constructive learning algorithm”, Neural Networks World, Vol. 8, pp. 1–38, 1998.

    Google Scholar 

  26. S. Draghici, V. Beiu and I.K. Sethi, “A VLSI optimal constructive algorithm for classification problems”, in C.H. Dagli, M. Akay, O. Ersoy, B.R. Fernández and A.E Smith (eds) Smart Engineering Systems: Neural Networks, Fuzzy Logic, Data Mining and Evolutionary Programming, pp. 145–151, ASME Press: Fairfield, N.J., November 1997.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Division NIS–1, Los Alamos National Laboratory, Mail Stop D466, Los Alamos, New Mexico, 87545, U.S.A

    Valeriu Beiu

  2. Vision and Neural Networks Laboratory, Department of Computer Science, Wayne State University, 431 State Hall, Detroit, Michigan, 48202, U.S.A.>/aff>

    Sorin Draghici

  3. Départment de Mathématique, Université Catholique de Louvain, Chemin du Cyclotron, 2, B–1348, Louvain-la-Neuve, Belgium. E-mail

    Thiery De Pauw

Authors
  1. Valeriu Beiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sorin Draghici
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thiery De Pauw
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Beiu, V., Draghici, S. & De Pauw, T. A Constructive Approach to Calculating Lower Entropy Bounds. Neural Processing Letters 9, 1–12 (1999). https://doi.org/10.1023/A:1018659009494

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1018659009494

Search

Navigation