Skip to main content

Weight discretization due to optical constraints and its influence on the generalization abilities of a simple perceptron

  • Part III: Learning: Theory and Algorithms
  • Conference paper
  • First Online:
Artificial Neural Networks — ICANN'97 (ICANN 1997)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 1327))

Included in the following conference series:

  • 303 Accesses

Abstract

Motivated from the optical implementation of NN which can be realized by storing the weights in holograms with a limited number of gray values, we focus our investigation on the dependence of the generalization and training errors of a simple perceptron with discrete weights, on the number of allowed discrete values (there are 2P allowed values for a bit precision of p) and on the training set size. Our starting point is the teacher pupil paradigm. The teacher is defined by fixing its continuous weights to random values. The pupil network that is only allowed to have discrete values was trained to learn the rule produced by the teacher with simulated annealing. For α < α s, where a encodes the training set size, weight configurations exist so that the training set can be reproduced without error whereas the generalization error is nonzero. For α > α s there is no weight configuration of the pupil which can reproduce the training set without error and for α → ∞ both training and generalization errors asymptotically converge to an εmin. We found that between a precision of 5 bit and 8 bit there was no remarkable improvement in the generalization ablitity of the pupil perceptron. This result is very useful for the optical implementation since optical constraints for storing weights in holograms restrict precision to a maximum value of 6 bit.

Special thanks to Prof.Dr.H.Horner and Priv.Doz.Dr.R.Kiihn from the Institute of Theoretical Physics at Heidelberg University for supporting our work with very helpful discussions and hints.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Engel, A.: Uniform convergence bounds for learning from examples. Modern Physics Letters B bf 8 (1994) 1683–1708

    Google Scholar 

  2. Biehl, M. Watkin, T.L.H. Rau, A.: The statistical mechanics of learning a rule. Review of Modern Physics bf 65 (1993)

    Google Scholar 

  3. Horner, H.: Dynamics of learning and generalization in a binary perceptron model. Zeitschrift für Physik B — Condensed Matter bf 87 (1992) 371–376

    Google Scholar 

  4. Horner, H.: Dynamics of learning and generalization in perceptrons with constraints. Physica A bf 200 (1993) 552–562

    Google Scholar 

  5. Lange, R.: Perfect learning in neural networks. PhD thesis Ruprecht-Karls-Universität Heidelberg (1995)

    Google Scholar 

  6. Metropolis, M. Rosenbluth,A.W. Rosenbluth, M.N. Teller, A.H. Teller, E. Equation of state calculations by fast computing machines. Journal of Chemical Physics bf 21 (1953) 1087–1092

    Google Scholar 

  7. Patel, H.-K.: Computational complexity, learning rules and storage capacities: a monte carlo study for the binary perceptron. Zeitschrift für Physik B — Condensed Matter bf 91 (1993) 257–266

    Google Scholar 

  8. Schwember, St.: Untersuchungen zur Generalisierungsfähigkeit des Simple Perzeptrons mit diskreten Gewichten. Diplomarbeit Ruprecht-Karls-Universität Heidelberg (1997)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Wulfram Gerstner Alain Germond Martin Hasler Jean-Daniel Nicoud

Rights and permissions

Reprints and permissions

Copyright information

© 1997 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Aboukassem, M., Schwember, S., Noehte, S., Männer, R. (1997). Weight discretization due to optical constraints and its influence on the generalization abilities of a simple perceptron. In: Gerstner, W., Germond, A., Hasler, M., Nicoud, JD. (eds) Artificial Neural Networks — ICANN'97. ICANN 1997. Lecture Notes in Computer Science, vol 1327. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0020173

Download citation

  • DOI: https://doi.org/10.1007/BFb0020173

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-63631-1

  • Online ISBN: 978-3-540-69620-9

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics