Skip to main content
SpringerLink
Log in

Novel Neuronal Activation Functions for Feedforward Neural Networks

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Feedforward neural network structures have extensively been considered in the literature. In a significant volume of research and development studies hyperbolic tangent type of a neuronal nonlinearity has been utilized. This paper dwells on the widely used neuronal activation functions as well as two new ones composed of sines and cosines, and a sinc function characterizing the firing of a neuron. The viewpoint here is to consider the hidden layer(s) as transforming blocks composed of nonlinear basis functions, which may assume different forms. This paper considers 8 different activation functions which are differentiable and utilizes Levenberg-Marquardt algorithm for parameter tuning purposes. The studies carried out have a guiding quality based on empirical results on several training data sets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Kamruzzaman J, Aziz SM (2002) A note on activation function in multilayer feedforward learning. In: Proc. of the 2002 int. joint. conf. on Neural Networks (IJCNN ’02), 12–17 May, vol 1, pp 519–523

  2. Hara K ,Nakayamma K (1994) Comparison of activation functions in multilayer neural network for pattern classification. In: IEEE world congress on computational intelligence, 1994 IEEE in conf. on neural networks, 27 June–2 July, vol 5, pp 2997–3002

  3. Wu Y, Zhao M, Ding X (1997) Beyond weights adaptation: a new neuron model with trainable activation function and its supervised learning. In: Int. conf. on neural networks, 9–12 June, vol 2, pp 1152–1157

  4. Karaköse M, Akın E (2004) Type-2 fuzzy activation function for multilayer feedforward neural networks. In: IEEE int. conf. on systems, man and cybernetics, 10–13 Oct., vol 4, pp 3762–3767

  5. Chiang C-C, Fu H-C (1992) A variant of second-order multilayer perceptron and its application to function approximations. In: Int. joint. conf. on neural networks (IJCNN ’92), 7–11 June, vol 3, pp 887–892

  6. Jang J-SR, Sun C-T, Mizutani E (1997) Neuro-fuzzy and soft computing. PTR Prentice-Hall

  7. Hagan MT, Menhaj MB (1994) Training feedforward networks with the Marquardt algorithm. IEEE Trans Neural Netw NN-5: 989–993

    Article  Google Scholar 

  8. Kretzschmar R, Büeler R, Karayiannis NB, Eggimann F (2000) Quantum neural networks versus conventional feedforward neural networks: an experimental study. In: Proc. of the 2000 IEEE signal processing society workshop on neural networks for signal processing X, Dec. 11–13, vol 1, pp 328–337

  9. Efe MÖ, Abadoğlu E, Kaynak O (1999) A novel analysis and design of a neural network assisted nonlinear controller for a bioreactor. Int J Robust Nonlinear Control 9(11): 799–815

    Article  MATH  Google Scholar 

  10. Puskorius GV, Feldkamp LA (1990) Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks. IEEE Trans Neural Netw 5(2): 279–297

    Article  Google Scholar 

  11. Ungar LH (1997) A bioreactor benchmark for adaptive-network based process control. In: Miller WT III, Sutton RS, Werbos PJ (eds) Neural networks for control. MIT Press, pp 387–402

  12. Narendra KS, Parthasarathy K (1990) Identification and control of dynamical systems using neural networks. IEEE Trans Neural Netw 1: 4–27

    Article  Google Scholar 

  13. Efe MÖ, Debiasi M, Yan P, Özbay H, Samimy M (2008) Neural network based generalization and modeling of subsonic cavity flow regimes. Int J Syst Sci 39(2): 105–117

    Article  Google Scholar 

  14. Samimy M, Debiasi M, Caraballo E, Malone J, Little J, Özbay H, Efe MÖ, Yan P, Yuan X, DeBonis J, Myatt JH, Camphouse RC (2004) Exploring strategies for closed-loop cavity flow control. In: 42nd AIAA Aerospace Sciences meeting and exhibit, January 5–8, Reno, Nevada, U.S.A., 2004 (Paper # AIAA 2004-0576)

  15. Debiasi M, Samimy M (2004) Logic-based active control of subsonic cavity-flow resonance. AIAA J 42: 1901–1909

    Article  Google Scholar 

  16. Asuncion A, Newman DJ (2007) UCI machine learning Repository [http://www.ics.uci.edu/~mlearn/MLRepository.html]. University of California, School of Information and Computer Science, Irvine, CA

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Electronics Engineering, TOBB Economics and Technology University, Söğütözü, Ankara, Turkey

    Mehmet Önder Efe

Authors
  1. Mehmet Önder Efe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mehmet Önder Efe.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Efe, M.Ö. Novel Neuronal Activation Functions for Feedforward Neural Networks. Neural Process Lett 28, 63–79 (2008). https://doi.org/10.1007/s11063-008-9082-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-008-9082-0

Keywords

Search

Navigation