Skip to main content
SpringerLink
Log in

Robust Training of Artificial Feedforward Neural Networks

  • Chapter
Book cover Foundations of Computational, Intelligence Volume 1

Part of the book series: Studies in Computational Intelligence ((SCI,volume 201))

Abstract

Artificial feedforward neural networks have received researchers’ great interest due to its ability to approximate functions without having a prior knowledge about the true underlying function. The most popular algorithm for training these networks is the backpropagation algorithm that is based on the minimization of the mean square error cost function. However this algorithm is not robust in the presence of outliers that may pollute the training data. In this chapter we present several methods to robustify neural network training algorithms. First, employing a family of robust statistics estimators, commonly known as M-estimators, in the backpropagation algorithm is reviewed and evaluated for the task of function approximation and dynamical model identification. As theseM-estimators sometimes do not have sufficient insensitivity to data outliers, the chapter next resorts to the statistically more robust estimator of the least median of squares, and develops a stochastic algorithm to minimize a related cost function. The reported experimental results have indeed shown the improved robustness of the new algorithm, especially compared to the standard backpropagation algorithm, on datasets with varying degrees of outlying data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Pernia-Espinoza, A.V., Ordieres-Mere, J.B., Martinez-de-Pison, F.J., Gonzalez-Marcos, A.: TAO-robust backpropagation learning algorithm. Neural Networks 18, 1–14 (2005)

    Article  Google Scholar 

  2. Annema, A.-J.: Feed-forward neural networks: Vector decomposition analysis, Modelling and Analog Implementation. Kluwer Academic Publishers, Boston (1995)

    Google Scholar 

  3. Baxt, W.G.: Use of an artificial neural network for data analysis in clinical decision making: The diagnosis of acute coronary occlusion. Neural Computation 2, 480–489 (1990)

    Article  Google Scholar 

  4. Baxt, W.G.: Use of an artificial neural network for the diagnosis of myocardial infarction. Annals of Internal Medicine 115, 843–848 (1991)

    Google Scholar 

  5. Bishop, C.M.: Neural networks for pattern recognition. Clarendon Press, Oxford (1995)

    Google Scholar 

  6. Goodall, C.: M-estimators of location: An outline of the theory. In: Hoaglin, Mosteller, Turkey (eds.) Understanding Robust and Exploratory Data Analysis, pp. 339–403 (1983)

    Google Scholar 

  7. Peterson, C., Andrson, J.: A mean field theory learning algorithm for neural networks. Complex Systems 1(1), 995–1019 (1987)

    MATH  Google Scholar 

  8. Chuang, C.C., Su, S.F., Hsiao, C.C.: The annealing robust backpropagation (ARBP) learning algorithm. IEEE Trans. on Neural Networks 11(5), 1067–1077 (2000)

    Article  Google Scholar 

  9. Churchland, P.S., Sejnowski, T.J.: The computational brain in deutscher sprache. In: Comutational intelligence. Vieweg Verlag (1997)

    Google Scholar 

  10. Corana, A., Marchesi, M., Martini, C., Ridella, S.: Minimizing multimodal functions of continuous variables with the simulated annealing algorithm. ACM Trans. on Mathematical Software 13(3), 262–280 (1987)

    Article  MATH  MathSciNet  Google Scholar 

  11. Cowan, J.D.: Neural networks: The early days. In: Touretzky, D. (ed.) Advances in neural information processing systems 2 (NIPS), pp. 828–842. Morgan Kaufmann, San Francisco (1990)

    Google Scholar 

  12. Rumelhart, D.E., McClelland, J.L.: Parallel distributed processing. MIT Press, Cambridge (1986)

    Google Scholar 

  13. Van den Bout, D.E., Miller, T.K.: Graph partitioning using annealed networks. IEEE Trans. on Neural Networks 1, 192–203 (1990)

    Article  Google Scholar 

  14. Aarts, E.H., Korst, J.: Simulated annealing and Boltzmann machines: stochastic approach to combinatorial optimization and neural computing. John Wiley and Sons, Inc., New York (1989)

    MATH  Google Scholar 

  15. Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust Statistics, The approach based on influence functions. Wiley, NewYork (1986)

    MATH  Google Scholar 

  16. Dreyfus, G.: Neural networks methodology and applications. Springer, Heidelberg (2005)

    MATH  Google Scholar 

  17. Bibro, G.L., Synder, W.E., Garnier, S.J., Gault, J.W.: Mean field annealing: A formalism for constructing GNC-like algorithms. IEEE Trans. on Neural Networks 3, 131–138 (1992)

    Article  Google Scholar 

  18. Goffe, W.L., Ferrier, G.D., Rogers, J.: Global optimization of statistical functions with simulated annealing. Journal of Econometrics 60, 65–99 (1994)

    Article  MATH  Google Scholar 

  19. Gupta, P., Sinha, N.: An improved approach for nonlinear system identification using neural networks. Journal of the Franklin Institute 336(4), 721–734 (1999)

    Article  MATH  Google Scholar 

  20. Haykin: Neural Networks: A comprehensive foundation, 2nd edn. Macmillan College Publishing, New York (1994)

    MATH  Google Scholar 

  21. Hertz, J., Krogh, A., Palmer, R.G.: Introduction to the theory of Neural Computation. Addison Vesley, New York (1991)

    Google Scholar 

  22. Hornik, K.: Multi-layer Feed-Forward Networks are Universal Approximators. In: White, H., et al. (eds.) Artificial Neural Networks: approximation and Learning Theory. Blackwell publishers, Cambridge (1992)

    Google Scholar 

  23. Hutchinson, J.M.: A radial basis function approach to financial time series analysis. Ph.D. dissertation, Massachusetts Institute of Technology (1994)

    Google Scholar 

  24. Moody, J., Darken, C.: Fast learning in networks of locally-tuned processing units. Neural Computa 1, 281–284 (1989)

    Article  Google Scholar 

  25. Liano, K.: Robust error measure for supervised neural network learning with outliers. IEEE Trans. Neural Networks 7, 246–250 (1996)

    Article  Google Scholar 

  26. Kenneth, D., Kahng, A.B.: Simulated annealing of neural networks: The cooling strategy reconsidered. Technical report CA90024, UCLA Computer science Dept., Los Angeles (1965)

    Google Scholar 

  27. Kumpati, S., Narendra: Identification and control of dynamical systems using neural networks. IEEE Trans. on Neural Networks 1(1), 4–27 (1990)

    Article  Google Scholar 

  28. Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Networks 4(2), 251–257 (1991)

    Article  Google Scholar 

  29. Huang, L., Zhang, B.L., Huang, Q.: Robust interval regression analysis using neural network. Fuzzy Sets Syst., 337–347 (1998)

    Google Scholar 

  30. Le Cun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems, vol. 2, pp. 248–257 (1990)

    Google Scholar 

  31. Leung, M.T., Engeler, W.E., Frank, P.: Fingerprint processing using backpropagation neural networks. In: Proceedings of the International Joint Conference on Neural Networks I, pp. 15–20 (1990)

    Google Scholar 

  32. Masters, T.: Advanced algorithms for neural networks: A C++ source book. John Wiley and Sons, Inc., New York (1995)

    Google Scholar 

  33. Hassoun, M.H.: Fundamentals of artificial neural networks. MIT Press, Cambridge (1995)

    MATH  Google Scholar 

  34. Huber, P.J.: Robust Statistics. John Wiley and Sons, New York (1981)

    Book  MATH  Google Scholar 

  35. Rousseeuw, P.J., Leroy, A.M.: Robust regression and outlier detection. Wiley, New York (1987)

    Book  MATH  Google Scholar 

  36. Pomerleau, D.A.: Neural network perception for mobile robot guidance. Kluwer, Boston (1993)

    Google Scholar 

  37. Rosenblatt, F.: The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review 65, 386–408 (1959)

    Article  Google Scholar 

  38. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by propagating errors. Nature 323, 533–546 (1986)

    Article  Google Scholar 

  39. Welch, R.M., Sengupta, S.K., Goroch, A.K., Rabindra, P., Rangaraj, N., Navar, M.S.: Polar cloud and surface classification using AVHRR imagery: An intercomparison of methods. Journal of Applied meteorology 31, 405–420 (1992)

    Article  Google Scholar 

  40. William, J.J.: Introduction to robust and quasi-robust statistical methods. Springer, Heidelberg (1983)

    MATH  Google Scholar 

  41. Zamarreno, J.M., Vega, P.: State space neural network: Properties and application. Neural Networks 11(6), 1099–1112 (1998)

    Article  Google Scholar 

  42. Zaprains, A.D., Refenes, A.P.: Principles of neural model identification, selection, and adequacy with applications to financial econometrics. In: Perspective in Neuro Computing. Springer, London (1999)

    Google Scholar 

  43. Zhang, Z.: Parameter estimation techniques: A tutorial with application to conic fitting. Image and Vision Computing Journal 15(1), 59–76 (1997)

    Article  Google Scholar 

  44. Rusiecki, A.L.: Robust learning algorithm with the variable learning rate. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Żurada, J.M. (eds.) ICAISC 2006. LNCS, vol. 4029, pp. 83–90. Springer, Heidelberg (2006)

    Google Scholar 

  45. Rusiecki, A.L.: Robust LTS backpropagation learning algorithm. In: Sandoval, F., Prieto, A.G., Cabestany, J., Graña, M. (eds.) IWANN 2007. LNCS, vol. 4507, pp. 102–109. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  46. Chuang, C., Jeng, J., Lin, P.: Annealing robust radial basis function networks for function approximation with outliers. Neurocomputing 56, 123–139 (2004)

    Article  Google Scholar 

  47. Lee, C., Chung, P., Tsai, J., Chang, C.: Robust radial basis function neural networks. IEEE Trans. Systems, Man, and Cybernetics – Part B: Cybernetics 29(6), 674–685 (1999)

    Article  Google Scholar 

  48. Rusiecki, A.L.: Robust MCD-based backpropagation learning algorithm. In: Rutkowski, et al. (eds.) ICAISC 2008. LNCS, vol. 5097, pp. 154–163. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electrical Engineering Department, Assiut University, Assiut, 71516, Egypt

    Moumen T. El-Melegy & Amer A. Ali

  2. Electrical Engineering Department, Al-Azhar University, Qena, 83513, Egypt

    Mohammed H. Essai

Authors
  1. Moumen T. El-Melegy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammed H. Essai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amer A. Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. College of Business Administration, Quantitative and Information System Department, Kuwait University, P.O. Box 5486, 13055, Safat, Kuwait

    Aboul-Ella Hassanien

  2. Center of Excellence for Quantifiable, Quality of Service, Norwegian University of Science & Technology, O.S. Bragstads plass 2E, 7491, Trondheim, Norway

    Ajith Abraham

  3. Department of Computer and Telecommunications Engineering, University ofWestern Macedonia, Agios Dimitrios Park, 50 100, Kozani, Greece

    Athanasios V. Vasilakos

  4. Dept. Electrical and Computer Engineering, University of Alberta, T6J 2V4, Edmonton,Alberta, Canada

    Witold Pedrycz

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

El-Melegy, M.T., Essai, M.H., Ali, A.A. (2009). Robust Training of Artificial Feedforward Neural Networks. In: Hassanien, AE., Abraham, A., Vasilakos, A.V., Pedrycz, W. (eds) Foundations of Computational, Intelligence Volume 1. Studies in Computational Intelligence, vol 201. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-01082-8_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-01082-8_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-01081-1

  • Online ISBN: 978-3-642-01082-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation