Skip to main content
SpringerLink
Log in

Training neural networks with ant colony optimization algorithms for pattern classification

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Feed-forward neural networks are commonly used for pattern classification. The classification accuracy of feed-forward neural networks depends on the configuration selected and the training process. Once the architecture of the network is decided, training algorithms, usually gradient descent techniques, are used to determine the connection weights of the feed-forward neural network. However, gradient descent techniques often get trapped in local optima of the search landscape. To address this issue, an ant colony optimization (ACO) algorithm is applied to train feed-forward neural networks for pattern classification in this paper. In addition, the ACO training algorithm is hybridized with gradient descent training. Both standalone and hybrid ACO training algorithms are evaluated on several benchmark pattern classification problems, and compared with other swarm intelligence, evolutionary and traditional training algorithms. The experimental results show the efficiency of the proposed ACO training algorithms for feed-forward neural networks for pattern classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Ozturk and Karaboga (2009) performed only the first cross-validation of our fourfold cross-validation experiments. Therefore, the results of the proposed ACO refer only to the first cross-validation dataset division.

References

  • Alba E, Chicano J (2004) Training neural networks with ga hybrid algorithms. In: Deb K (ed) Proceedings of the 2004 Genetic and Evolutionary Computation Conference, vol 3102. LNCS, Springer, Berlin, pp 852–863

    Chapter  Google Scholar 

  • Alba E, Marti R (eds) (2006) Metaheuristic procedures for training neural networks. Springer, New York

  • Bache K, Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml

  • Bennett KP, Mangasarian OL (1992) Robust linear programming discrimination of two linearly inseparable sets. Optim Methods Softw 1(1):23–34

    Article  Google Scholar 

  • Bishop C (1995) Neural networks for pattern recognition. Oxford University Press, Oxford

    Google Scholar 

  • Blum C, Socha K (2005) Training feed-forward neural networks with ant colony optimization: an application to pattern classification. In: 2005. HIS ’05. Fifth International Conference on Hybrid Intelligent Systems

  • Bonabeau E, Dorigo M, Theraulaz G (eds) (1997) Swarm intelligence: from natural to artificial systems. Oxford University Press, New York

    Google Scholar 

  • Bullinaria J (2005) Evolving neural networks: is it really worth the effort? In: Proceedings of the European Symposium on Artificial Neural Networks, pp 267–272

  • Bullnheimer B, Hartl R, Strauss C (1999) A new rank-based version of the ant system: a computational study. Cent Eur J Oper Res Econ 7(1):25–38

    MATH  MathSciNet  Google Scholar 

  • Cantu-Paz E, Kamath C (2005) An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Trans Syst, Man, Cybern-Part B: Cybern 35(5):915–927

    Article  Google Scholar 

  • Carpenter G, Grossberg S (1988) The art of adaptive pattern recognition by a self-organizing neural network. IEEE Comput 21:77–88

    Article  Google Scholar 

  • Carvalho M, Ludermir T (2006) Hybrid training of feed-forward neural networks with particle swarm optimization. In: King I, Wang J, Chan LW, Wang D (eds) Neural information processing, vol 4233. LNCS, Springer, Berlin, pp 1061–1070

    Chapter  Google Scholar 

  • Cotta C, Alba E, Sagarna R, Larrañaga P (2001) Adjusting weights in artificial neural networks using evolutionary algorithms. Estimation of distribution algorithms: a new tool for evolutionary computation. pp 361–378

  • Dayhoff J (ed) (1990) Neural-network architectures: an introduction, 1st edn. Van Nostrand Reinhold, New York

    Google Scholar 

  • Detrano R, Janosi A, Steinbrunn W, Pfisterer M, Schmid JJ, Sandhu S, Guppy KH, Lee S, Froelicher V (1989) International application of a new probability algorithm for the diagnosis of coronary artery disease. Am J Cardiol 64(5):304–310

    Article  Google Scholar 

  • Dorigo M, Gambardella LM (1997) Ant colony system: a cooperative learning approach to the traveling salesman problem. IEEE Trans Evolut Comput 1(1):53–66

    Article  Google Scholar 

  • Dorigo M, Stützle T (eds) (2004) Ant colony optimization. MIT Press, London

    MATH  Google Scholar 

  • Dorigo M, Maniezzo V, Colorni A (1996) Ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern-Part B: Cybern 26(1):29–41

    Article  Google Scholar 

  • Dorigo M, Caro GD, Gambardella LM (1999) Ant algorithms for discrete optimization. Artif Life 5(2):137–172

    Article  Google Scholar 

  • Fels S, Hinton G (1993) Glove-talk: a neural network interface between a data-glove and a speech synthesizer. IEEE Trans Neural Netw 4:2–8

    Article  Google Scholar 

  • Gennari JH, Langley P, Fisher D (1989) Models of incremental concept formation. Artif Intell 40(13):11–61

    Article  Google Scholar 

  • Hagan M, Menhaj M (1994) Training feedforward networks with the marquardt algorithm. IEEE Trans Neural Netw 5(6):989–993

    Article  Google Scholar 

  • Hinton G (1989) Connectionist learning approaches. Artif Intell 40(1–3):185–234

    Article  Google Scholar 

  • Ilonen J, Kamarainen JK, Lampinen J (2003) Differential evolution training algorithm for feed-forward neural networks. Neural Process Lett 17(1):93–105

    Article  Google Scholar 

  • Karaboga D, Ozturk C (2009) Neural networks training by artificial bee colony algorithm on pattern classification. Neural Netw World 3:279–292

    Google Scholar 

  • Karaboga D, Akay B, Ozturk C (2007) Artificial bee colony (abc) optimization algorithm for training feed-forward neural networks. In: Torra V, Narukawa Y, Yoshida Y (eds) Modeling decisions for artificial intelligence, vol 4617. LNCS, Springer, Berlin, pp 318–329

    Chapter  Google Scholar 

  • Lang K, Waibel A, Hinton G (1990) A time-delay neural network architecture for isolated word recognition. Neural Netw 3(1):33–43

    Article  Google Scholar 

  • Levenberg K (1944) A method for solution of certain problems in least squares. Q Appl Math 2:164–168

    MATH  MathSciNet  Google Scholar 

  • Liu YP, Wu MG, Qian JX (2006) Evolving neural networks using the hybrid of ant colony optimization and bp algorithm. In: Wang J, Yi Z, Zurada J, Lu BL, Yin H (eds) Advances in Neural Networks–3rd International Symposium on Neural Networks, vol 3971. LNCS, Springer, Berlin, pp 714–722.

  • Mandischer M (2002) A comparison of evolution strategies and backpropagation for neural network training. Neurocomputing 42(1):87–117

    Article  MATH  Google Scholar 

  • Mangasarian O, Setiono R, Wolberg WH (1990) Pattern recognition via linear programming: Theory and application to medical diagnosis. In: Coleman T, Li Y (eds) Large-scale numerical optimization. SIAM Publications, Philadelphia, pp 22–31

    Google Scholar 

  • Marquardt D (1963) An algorithm for least-squares estimation of nonlinear parameters. SIAM J Appl Math 11:431–441

    Article  MATH  MathSciNet  Google Scholar 

  • Mavrovouniotis M, Yang S (2013) Evolving neural networks using ant colony optimization with pheromone trail limits. In: Proceedings of the 2013 UK Workshop on Computational Intelligence, IEEE Press, pp 16–23

  • Mehrotra K, Mohan C, Ranka S (eds) (1997) Elements of artificial neural networks. MIT Press, Cambridge

    Google Scholar 

  • Mendes R, Cortez P, Rocha M, Neves J (2002) Particle swarms for feedforward neural network training. In: Neural Networks, 2002. IJCNN ’02. Proceedings of the 2002 International Joint Conference on, vol 2, pp 1895–1899

  • Montana D, Davis L (1989) Training feedforward neural network using genetic algorithms. In: Proceedings of the 11th International Joint Conference Artificial Intelligence, Morgan Kaufmann, pp 762–767

  • Prechelt L (1994) Proben1— a set of neural network benchmark problems and benchmarking rules. Tech. Rep. 21, University Karlsruhe, Germany

  • Rakitianskaia A, Engelbrecht A (2012) Training feedforward neural networks with dynamic particle swarm optimisation. Swarm Intell 6(3):233–270

  • Rumelhart D, Hinton G, Williams R (1986) Learning representations by backpropagation errors. Nature 536:323–533

    Google Scholar 

  • Socha K (2004) Aco for continuous and mixed-variable optimization. In: Dorigo M, Birattari M, Blum C, Gambardella LM, Mondada F, Stützle T (eds) Proceedings of the 4th International Workshop on Ant Algorithms and Swarm Intelligence, vol 3172. LNCS, Springer, Berlin, pp 25–36

  • Socha K, Blum C (2007) An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training. Neural Comput Appl 16:235–247

    Article  Google Scholar 

  • Stützle T, Hoos H (1997) The max–min ant system and local search for the traveling salesman problem. In: Proceedings of the 1997 IEEE International Conference on Evolutionary Computation, IEEE Press, pp 309–314

  • Sutton R (1986) Two problems with backpropagation and other steepest-descent learning procedures for networks. In: Proceedings of the 8th Annual Conference Cognitive Science Socitey, pp 823–831

  • Whitley D, Starkweather T, Bogart C (1990) Genetic algorithms and neural networks: optimizing connections and connectivity. Parallel Comput 14(3):347–361

    Article  Google Scholar 

  • Wolberg W (1990) Cancer diagnosis via linear programming. SIAM News 23:1–18

    Google Scholar 

  • Wolberg WH, Mangasarian OL (1990) Multisurface method of pattern separation for medical diagnosis applied to breast cytology. Proc Natl Acad Sci 87(23):9193–9196

    Article  MATH  Google Scholar 

  • Yao X (1999) Evolving artificial neural networks. Proc IEEE 89(9):1423–1447

    Google Scholar 

  • Yao X, Islam MM (2008) Evolving artificial neural network ensembles. IEEE Computl Intell Mag 3(1):31–42

    Article  Google Scholar 

  • Yao X, Liu Y (1996) Ensemble structure of evolutionary artificial neural networks. In: Proceedings of 1996 International Conference on Evolutionary Computation, pp 659–664

  • Yao X, Liu Y (1998) Making use of population information in evolutionary artificial neural networks. IEEE Trans Syst, Man, Cybern-Part B: Cybern 28(3):417–425

  • Zhang G (2000) Neural networks for classification: a survey. IEEE Trans Syst, Man, Cybern-Part C: Appl Rev 30(4):451–462

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for their thoughtful suggestions and constructive comments. This work was supported by the Engineering and Physical Sciences Research Council (EPSRC) of UK under Grant EP/K001310/1.

Author information

Authors and Affiliations

  1. Centre for Computational Intelligence (CCI), School of Computer Science and Informatics, De Montfort University, The Gateway, Leicester , LE1 9BH, UK

    Michalis Mavrovouniotis & Shengxiang Yang

Authors
  1. Michalis Mavrovouniotis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shengxiang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michalis Mavrovouniotis.

Additional information

Communicated by V. Loia.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mavrovouniotis, M., Yang, S. Training neural networks with ant colony optimization algorithms for pattern classification. Soft Comput 19, 1511–1522 (2015). https://doi.org/10.1007/s00500-014-1334-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-014-1334-5

Keywords

Search

Navigation