Skip to main content

Advertisement

Springer Nature Link
Log in

Special-purpose digital hardware for neural networks: An architectural survey

  • Published:
Journal of VLSI signal processing systems for signal, image and video technology Aims and scope Submit manuscript

Abstract

This paper presents a survey of digital systems to implement neural networks. We consider two basic options for designing these systems: parallel systems with standard digital components and parallel systems with custom processors. We describe many examples under each option, with an emphasis on commercially available systems. We report a first trend toward more general architectures and a second trend toward simple and fast structures. We discuss our experience in running a small ANN problem on two of these machines. After a reasonable programming effort, we obtain good convergence, but most of the training times are actually slower or moderately faster than on a serial workstation. We conclude that it is important to chose one's problems carefully, and that support software and in general, system integration, is only beginning to reach the level of versatility that many researchers will require.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. M. Glesner and W. Pöchmüller,An Overview of Neural Networks in VLSI, Chapman & Hall. London, 1994.

    MATH  Google Scholar 

  2. C.S. Lindsey and T. Lindblad, “Review of hardware neural networks: A user's perspective,” inProceedings of the Third Workshop on Neural Networks: From Biology to High Energy Physics, Isola d'Elba, Italy, Sept. 1993.

  3. A. König, ‘Survey and current status of neural network hard-ware,” inProceedings of the International Conference on Artificial Neural Networks, F. Fogelman-Soulie and P. Gallinari, (Eds.), Paris, Oct. 1995, pp. 391–410.

  4. S. Oteki, A. Hashimoto, T. Furuta, T. Watanabe, D.G. Stork, and H. Eguchi, “A digital neural network VLSI with on-chip learning using stochastic pulse encoding,” inProceedings of the International Joint Conference on Neural Networks, Nagoya, Japan, Oct. 1993, Vol. III, pp. 3039–45.

  5. N. Mauduit, M. Duranton, J. Gobert, and J.-A. Sirat, “Lneuro 1.0: A piece of hardware LEGO for building neural network systems,”IEEE Transactions on Neural Networks, Vol. NN-3, No. 3, pp. 414–22, May 1992.

    Article  Google Scholar 

  6. M. Yasunaga, N. Masuda, M. Yagyu, M. Asai, K. Shibata, M. Ooyama, M. Yamada, T. Sakaguchi, and M. Hashimoto, “A self-learning neural network composed of 1152 digital neurons in wafer-scale LSIs,” inProceedings of the International Joint Conference on Neural Networks, Seattle, Wash., July 1991, pp. 1844–49.

  7. C. Park, K. Buckmann, J. Diamond, U. Santoni, S.C. The, M. Holler, M. Glier, C.L. Scofield, and L. Nunez, “A radial basis function neural network with on-chip learning,” inProceedings of the International Joint Conference on Neural Networks, Nagoya, Japan, Oct. 1993, Vol. III, pp. 3035–38.

  8. P. Ienne and M.A. Viredaz, “GENES IV: A bit-serial processing element for a multi-model neural-network accelerator,”Journal of VLSI Signal Processing, Vol. 9, No. 3, pp. 257–73, Apr. 1995.

    Article  Google Scholar 

  9. N. Brüls, “Programmable VLSI array processor for neural networks and matrix-based signal processing—User description,” Technical Report, Siemens AG, Corporate Research and Development Division, Munich, Version 1.3, Oct. 1993.

    Google Scholar 

  10. M.A. Glover and W.T. Miller, III, “A massively-parallel SIMD processor for neural network and machine vision applications,” inAdvances in Neural Information Processing Systems, J.D Cowan, G. Tesauro, and J. Alspector (Eds.), Morgan Kaufmann, San Mateo, Calif., Vol. 6, pp. 843–49, 1994.

    Google Scholar 

  11. D. Hammerstrom, “A highly parallel digital architecture for neural network emulation,”VLSI for Artificial Intelligence and Neural Networks, J.G. Delgado-Frias and W.R. Moore (Eds.), Chap. 5.1, pp. 357–66, Plenum Press, New York, 1991.

    Chapter  Google Scholar 

  12. R.W. Means and L. Lisenbee, “Floating-point SIMD neurocomputer array processor,”Parallel Implementations of Neural Networks, K.W. Prztula and V.K. Prasanna (Eds.), Prentice Hall, New York, 1993.

    Google Scholar 

  13. J. Wawrzynek, K. Asanovié, B. Kingsbury, J. Beck, D. Johnson, and N. Morgan, “SPERT-II: A vector microprocessor system,”Computer, Vol. 29, No. 3, Mar. 1996.

  14. N. Morgan, J. Beck, P. Kohn, J. Bilmes, E. Allman, and J. Beer, “The Ring Array Processor: A multiprocessing peripheral for connectionist applications,”Journal of Parallel and Distributed Computing, Vol. 14, No. 3, pp. 248–59, Mar. 1992.

    Article  Google Scholar 

  15. U.A. Müller, A. Gunzinger, and W. Guggenbühl, “Fast neural net simulation with a DSP processor array,”IEEE Transactions on Neural Networks, Vol. NN-6, No. 1, pp. 203–13, Jan. 1995.

    Article  Google Scholar 

  16. D.E. Rumelhart, G.E. Hinton, and R.J. Williams, “Learning internal representations by errorpropagation,”Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D.E. Rumelhart and J.L. McClelland (Eds.), Vol. 1. Foundations, MIT Press, Cambridge, Mass., Chap. 8, 1986.

    Google Scholar 

  17. G. Kuhn and N. Herzberg, “Some variations on training of recurrent networks,”Neural Networks: Theory and Applications, R. Mammone and Y. Zeevi (Eds.), Academic Press, San Diego, Calif., Chap. 11, pp. 233–44, 1991.

    Google Scholar 

  18. R. Battiti, “First- and second-order methods for learning: Between steepest descent and Newton's method,”Neural Computation, Vol. 4, pp. 141–66, 1992.

    Article  Google Scholar 

  19. E.M. Johansson, F.U. Dowla, and D.M. Goodman, “Backpropagation learning for multilayer feed-forward neural networks using the conjugate gradient method,”International Journal of Neural Systems, Vol. 2, No. 4, pp. 291–301, 1992.

    Article  Google Scholar 

  20. A.S. Weigend, B.A. Huberman, and D.E. Rumelhart, “Predicting the future: A connectionist approach,”International Journal of Neural Systems, Vol. 1, pp. 193–209, 1990.

    Article  Google Scholar 

  21. K. Asanović, K. Beck, T. Callahan, J. Feldman, B. Irissou, B. Kingsbury, P. Kohn, J. Lazzaro, K. Morgan, D. Stoutamire, and J. Wawrzynek, “CNS-1 architecture specification: A connectionist network supercomputer,” Technical Report, TR-93-021, International Computer Science Institute, Berkeley, Calif., Apr. 1993.

    Google Scholar 

  22. A.J. De Groot and S.R. Parker, “Systolic implementation of neural networks,” inHigh Speed Computing II, K. Bromley (Ed.), SPIE—The International Society for Optical Engineering, Los Angeles, Calif., Vol. 1058. pp. 182–90, Jan. 1989.

    Chapter  Google Scholar 

  23. D.A. Pomerleau, G.L. Gusciora, D.S. Touretzky, and H.T. Kung, “Neural network simulation at warp speed: How we got 17 million connections per second,” inProceedings of the IEEE Conference on Neural Networks, San Diego, Calif., 1988, Vol. II, pp. 143–50.

  24. K. Asanović and N. Morgan, “Experimental determination of precision requirements for back-propagation training of artificial neural networks,” inProceedings of the Second International Conference on Microelectronics for Neural Networks, Munich, 1991, pp. 9–15.

  25. J.L. Holt and T.E. Baker, “Back propagation simulations using limited precision calculation,” inProceedings of the International Joint Conference on Neural Networks, Seattle, Wash., July 1991.

  26. P. Thiran, V. Peiris, P. Heim, and B. Hochet, “Quantization effects in digitally behaving circuit implementations of Kohonen networks,”IEEE Transactions on Neural Networks, Vol. NN-5, No. 3, pp. 450–58, May 1994.

    Article  Google Scholar 

  27. F. Castillo, J.A. García, J.M. Moreno, and J. Cabestany, “A coprocessor card for fast neural network emulation,” inFrom Natural to Artificial Neural Computation, J. Mira and F. Sandoval (Eds.),Lecture Notes in Computer Science, Vol. 930, pp. 752–60, Springer, Berlin, 1995.

    Chapter  Google Scholar 

  28. M. Griffin, G. Tahara, K. Knorpp, and B. Riley, “An 11-million transistor neural network execution engine,”IEEE International Conference on Solid-State Circuits, pp. 180–81, 1991.

  29. inova microelectronics corporation,N64000 Digital Neural Network Processor (Preliminary Data), Santa Clara, Calif.

  30. H. McCartor, “Back propagation implementation on the adaptive solutions neurocomputer chip,”Advances in Neural Information Processing Systems, D.S. Touretzky (Ed.), Morgan Kaufmann, San Mateo, Calif., Vol. 3, 1991.

    Google Scholar 

  31. E. Franzi, “Neural accelerator for parallelization of backpropagation algorithm,”Microprocessing and Microprogramming, Vol. 38, pp. 689–96, 1993.

    Article  Google Scholar 

  32. P. Murtagn and A.C. Tsoi, “Implementation issues of sigmoid function and its derivative for VLSI digital neural networks,”IEEE Proceedings, Vol. 139, No. 3, pp. 207–14, May 1992.

    Google Scholar 

  33. D. Hammerstrom and N. Nguyen, “An implementation of Kohonen's self-organizing map on the adaptive solutions neurocomputer,” inProceedings of the International Conference on Artificial Neural Networks, Helsinki, June 1991, Vol. I, pp. 715–20.

  34. Y. Sato, K. Shibata, M. Asai, M. Ohki, M. Sugie, T. Sakaguchi, M. Hashimoto, and Y. Kuwabara, Yoshihiro, “Development of a high-performance general purpose neuro-computer composed of 512 digital neurons,” inProceedings of the International Joint Conference on Neural Networks, Nagoya, Japan, Oct. 1993, Vol. II, pp. 1967–70.

  35. U. Ramacher, J. Beichter, W. Raab, J. Anlauf, N. Brüls, U. Hachmann, and M. Wesseling, “Design of a 1st generation neurocomputer,” inVLSI Design of Neural Networks, Kluwer Academic, Norwell, Mass., pp. 271–310, 1991.

    Chapter  Google Scholar 

  36. U. Ramacher and J. Beichter, “Systolic architectures for fast emulation of artificial neural networks,” inSystolic Array Processors, J. McCanny, J. McWhirter, and E. Swartzlander, Jr. (Eds.), Prentice Hall, New York, pp. 277–86, 1989.

    Google Scholar 

  37. U. Ramacher and P. Nachbar, “Hamiltonian dynamics of neural networks,” inProceedings of the Second International Conference on Microelectronics for Neural Networks, Munich, 1991, pp. 95–102.

  38. U. Ramacher, W. Raab, J. Anlauf, U. Hachmann, and M. Weßeling, “SYNAPSE-1—A general-purpose neurocomputer,” Technical Report, Munich, Siemens AG, Corporate Research and Development Division, Feb. 1993.

    Google Scholar 

  39. M. Viredaz, “Design and analysis of a systolic array for neural computation,” École Polytechnique Fédérale de Lausanne, Lausanne, Ph.D. Thesis No 1264, 1994.

  40. M.A. Viredaz and P. Ienne, “MANTRA I: A systolic neurocomputer,” inProceedings of the International Joint Conference on Neural Networks, Nagoya, Japan, Oct. 1993, Vol. III, pp. 3054–57.

  41. Y.I. Fet, “Vertical processing systems: A survey,”IEEE Micro, Vol. M-15, No. 1, pp. 65–75, Feb. 1995.

    Article  Google Scholar 

  42. T.G. Clarkson, D. Gorse, J.G. Taylor, and C.K. Ng, “Learning probabilistic RAM nets using VLSI structures,”IEEE Transactions on Computers, Vol. C-41, No. 12, pp. 1552–61, Dec. 1992.

    Article  Google Scholar 

  43. M. Yasunaga, N. Masuda, M. Yagyu, M. Asai, M. Yamada, and A. Masaki, “Design, fabrication and evaluation of a 5-inch wafer scale neural network LSI composed of 576 digital neurons,” inProceedings of the International Joint Conference on Neural Networks, San Diego, Calif., June 1990, Vol. II, pp. pp527–36.

  44. M. Fujita, Y. Kobayashi, K. Shiozawa, T. Takahashi, F. Mizuno, H. Hayakawa, M. Kato, S. Mori, T. Kase, and M. Yamada, “Development and fabrication of digital neural network WSIs,”IEICE Transactions on Electronics, Vol. E 76-C, No. 7, pp. 1182–90, July 1993.

    Google Scholar 

  45. J.B. Theeten, M. Duranton, N. Mauduit, and J.A. Sirat, “The Lneuro chip: A digital VLSI with on-chip learning mechanism,”Proceedings of the International Neural Networks Conference, pp. 593–96, 1990.

  46. Nestor Inc., R.I. Providence,Ni1000 Recognition Accelerator Datasheet, 1994.

  47. Intel Corp.,Ni1000 Datasheet (Preliminary), Santa Clara, Calif., 1993.

  48. C.L. Scofield and D.L. Reilly, “Into Silicon: Real time learning in a high density RBF neural network,” inProceedings of the International Joint Conference on Neural Networks, Seattle, Wash., July 1991, Vol. I, pp. 551–56.

  49. D.F. Spec, “Enhancements to probabilistic neural networks,” inProceedings of the International Joint Conference on Neural Networks, Baltimore, Md., June 1992, Vol. I, pp. 761–68.

  50. IBM.ZISC036 Data Book (Preliminary), Version 2.1, Nov. 1994.

  51. M.P. Perrone and L.N. Cooper, “The Ni1000: High speed parallel VLSI for implementing multilayer perceptrons,” inAdvances in Neural Information Processing Systems, J.D. Cowan, G. Tesauro, and J. Alspector (Eds.), Vol. 8, 1995.

  52. S.R. Jones and K.M. Sammut, “Learning in linear systolic neural network engines: Analysis and implementation,”IEEE Transactions on Neural Networks, Vol. NN-5, No. 4, pp. 584–93, July 1994.

    Article  Google Scholar 

  53. M. Duranton, “L-Neuro 2.3: A VLSI for image processing by neural networks,” inProceedings of the Fifth International Conference on Microelectronics for Neural Networks and Fuzzy Systems, Lausanne, Feb. 1996.

  54. Cray Research Inc.,CRAY T3D System Architecture Overview, HR-04033 edition, Sept. 1993.

  55. T. Kohonen,Self-Organization and Associative Memory, Springer Series in Information Sciences, Springer, Berlin, third edition, Vol. 8, 1989.

    Book  Google Scholar 

  56. T. Cornu and P. Ienne, “Performance of digital neuro-computers,” inProceedings of the Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems, Turin, Sept. 1994, pp. 87–93.

  57. P. Ienne, P. Thiran, and N. Vassilas, “Modified self-organising feature map algorithms for efficient digital hardware implementation,”IEEE Transactions on Neural Networks, Submitted paper, 1995.

  58. N. Vassilas, P. Thiran, and P. Lenne, “On modifications of Kohonen's feature map algorithm for an efficient parallel implementation,” inProceedings of the International Conference on Neural Networks, Washington, D.C., June 1996.

  59. A. Grama, A. Gupta, and V. Kumar, “Isoefficiency function: A scalability metric for parallel algorithms and architectures,”IEEE Parallel and Distributed Technology, Vol. 1, No. 3, pp. 12–21, 1993.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Microcomputing Laboratory, IN-F Ecublens, Swiss Federal Institute of Technology, CH-1015, Lausanne, Switzerland

    Paolo Ienne

  2. Mantra Centre for Neuromimetic Systems, IN-J Ecublens, Swiss Federal Institute of Technology, CH-1015, Lausanne, Switzerland

    Thierry Cornu

  3. SCR AISP Department, 755 College Road, 08540, Princeton, NJ, USA

    Gary Kuhn

Authors
  1. Paolo Ienne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thierry Cornu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gary Kuhn
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ienne, P., Cornu, T. & Kuhn, G. Special-purpose digital hardware for neural networks: An architectural survey. J VLSI Sign Process Syst Sign Image Video Technol 13, 5–25 (1996). https://doi.org/10.1007/BF00930664

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00930664

Keywords