Skip to main content

Advertisement

Log in

Comparison of the Convergence Rates of the New Correntropy-Based Levenberg–Marquardt (CLM) Method and the Fixed-Point Maximum Correntropy (FP-MCC) Algorithm

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

Correntropy as an efficient information theoretic (ITL) criterion has been extensively applied in many non-Gaussian applications. In order to maximize correntropy, several optimization algorithms have been proposed. Fixed-point maximum correntropy (FP-MCC) and correntropy-based Levenberg–Marquardt (CLM) are the fastest of the proposed methods. As the convergence rate of these operational methods has not been studied before, in this paper we prove the ability of their quadratic convergence from a theoretic point of view and establish the influence of bandwidth on their convergence order. Then, theoretic results are validated through numerical experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. S. Banach, Sur les operations dans les ensembles abstraits et leur application aux equations integrales. Fund. Math. 3(1), 133–181 (1922)

    Article  MATH  Google Scholar 

  2. L. Boccato, D. Fantinato, D. Silva, R. Ferrari, A. Neves, R. Attux, Analysis of ITL criteria in the context of FIR channel equalization. J. Commun. Inf. Syst. 31(1), 1–29 (2016). https://doi.org/10.14209/jcis.2016.1

    Google Scholar 

  3. R.L. Burden, J.D. Faires, Numerical Analysis. 2001 (Brooks-Cole, Pacific Grove, 2001)

    MATH  Google Scholar 

  4. L. Cao, H.M. Schwartz, Exponential convergence of the Kalman filter based parameter estimation algorithm. Int. J. Adapt. Control Signal Process. 17(10), 763–783 (2003). https://doi.org/10.1002/acs.774

    Article  MATH  Google Scholar 

  5. B. Chen, J. Wang, H. Zhao, N. Zheng, J.C. Principe, Convergence of a fixed-point algorithm under maximum correntropy criterion. IEEE Signal Process. Lett. 22(10), 1723–1727 (2015)

    Article  Google Scholar 

  6. S. Dixit, D. Nagaria, Design and analysis of cascaded LMS adaptive filters for noise cancellation. Circuits Syst. Signal Process. (2016). https://doi.org/10.1007/s00034-016-0332-5

    Google Scholar 

  7. D. Erdogmus, Information theoretic learning: Renyi’s entropy and its applications to adaptive systems training. Ph.D. Dissertation, University of Florida (2002)

  8. D. Erdogmus, R. Agrawal, J.C. Principe, A mutual information extension to the matched filter. Signal Process. 85(5 SPEC. ISS.), 927–935 (2005). https://doi.org/10.1016/j.sigpro.2004.11.018

    Article  MATH  Google Scholar 

  9. D. Erdogmus, J.C. Principe, An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems. IEEE Trans. Signal Process. 50(7), 1780–1786 (2002). https://doi.org/10.1109/TSP.2002.1011217

    Article  Google Scholar 

  10. J. Fan, The modified Levenberg–Marquardt method for nonlinear equations with cubic convergence. Math. Comput. 81(277), 447–466 (2012). https://doi.org/10.1090/S0025-5718-2011-02496-8

    Article  MathSciNet  MATH  Google Scholar 

  11. Jy Fan, Yx Yuan, On the quadratic convergence of the Levenberg–Marquardt method without nonsingularity assumption. Computing 74(1), 23–39 (2005). https://doi.org/10.1007/s00607-004-0083-1

    Article  MathSciNet  MATH  Google Scholar 

  12. M.M. Fateh, S. Khorashadizadeh, Robust control of electrically driven robots by adaptive fuzzy estimation of uncertainty. Nonlinear Dyn. 69(3), 1465–1477 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  13. A. Garde, L. Sornmo, R. Jane, B.F. Giraldo, Correntropy-based spectral characterization of respiratory patterns in patients with chronic heart failure. IEEE Trans. Biomed. Eng. 57(8), 1964–1972 (2010). https://doi.org/10.1109/TBME.2010.2044176

    Article  Google Scholar 

  14. M. Hajiabadi, G.A. Hodtani, H. Khoshbin, Adaptive multitask network based on maximum correntropy learning algorithm. Int. J. Adapt. Control Signal Process. (2017). https://doi.org/10.1002/acs.2760

    MathSciNet  MATH  Google Scholar 

  15. H. Hammouri, M. Kinnaert, E. El Yaagoubi, Observer-based approach to fault detection and isolation for nonlinear systems. IEEE Trans. Autom. Control 44(10), 1879–1884 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  16. S. Han, A family of minimum Renyi’s error entropy algorithm for information processing. Ph.D. Dissertation, University of Florida (2007)

  17. R. He, B.-G. Hu, W.-S. Zheng, X.-W. Kong, Robust principal component analysis based on maximum correntropy criterion. IEEE Trans. Image Process. 20(6), 1485–1494 (2011). https://doi.org/10.1109/TIP.2010.2103949

    Article  MathSciNet  MATH  Google Scholar 

  18. R. He, L. Wang, Z. Sun, Y. Zhang, B. Li, Information theoretic subspace clustering. IEEE Trans. Neural Netw. Learn. Syst. PP(99), 1–13 (2015). https://doi.org/10.1109/TNNLS.2015.2500600

    Google Scholar 

  19. R. He, W.S. Zheng, B.G. Hu, X.W. Kong, Two-stage nonnegative sparse representation for large-scale face recognition. IEEE Trans. Neural Netw. Learn. Syst. 24(1), 35–46 (2013). https://doi.org/10.1109/TNNLS.2012.2226471

    Article  Google Scholar 

  20. J. Heinonen, Lectures on Lipschitz analysis. Lecture Notes, pp. 1–77 (2005)

  21. A.R. Heravi, G.A. Hodtani, A new robust correntropy based Levenberg–Marquardt algorithm, in 2016 Iran Workshop on Communication and Information Theory (IWCIT) (IEEE, 2016), pp. 1–6. https://doi.org/10.1109/IWCIT.2016.7491623

  22. F. Huang, J. Zhang, S. Zhang, Adaptive filtering under a variable kernel width maximum correntropy criterion, IEEE Trans. Circuits Syst. II Express Briefs 64(10), 1247–1251 (2017)

  23. R. Izanloo, S.A. Fakoorian, H.S. Yazdi, D. Simon, Kalman filtering based on the maximum correntropy criterion in the presence of non-Gaussian noise, in 2016 Annual Conference on Information Science and Systems (CISS) (IEEE, 2016), pp. 500–505. https://doi.org/10.1109/CISS.2016.7460553

  24. R. Kalaba, A. Tishler, J. Wang, Rate of convergence of the generalized newton algorithm using the fixed-point approach. J. Optim. Theory Appl. 43(4), 543–555 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  25. E.W. Kamen, A recursive parameter estimator yielding exponential convergence under sufficient excitation. Circuits Syst. Signal Process. 8(2), 207–228 (1989). https://doi.org/10.1007/BF01599938

    Article  MathSciNet  MATH  Google Scholar 

  26. A. Khalili, A. Rastegarnia, M.K. Islam, T.Y. Rezaii, Steady-state tracking analysis of adaptive filter with maximum correntropy criterion. Circuits Syst. Signal Process. 36(4), 1725–1734 (2017)

    Article  MATH  Google Scholar 

  27. S. Khorashadizadeh, M.M. Fateh, Adaptive fourier series-based control of electrically driven robot manipulators, in 2013 3rd International Conference on Control, Instrumentation, and Automation (ICCIA) (IEEE, 2013), pp. 213–218

  28. K. Levenberg, A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2(2), 164–168 (1944)

    Article  MathSciNet  MATH  Google Scholar 

  29. H. Li, Y. Gao, P. Shi, H.K. Lam, Observer-based fault detection for nonlinear systems with sensor fault and limited communication capacity. IEEE Trans. Autom. Control 61(9), 2745–2751 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  30. W. Liu, P.P. Pokharel, J.C. Principe, Correntropy: A Localized Similarity Measure, in The 2006 IEEE International Joint Conference on Neural Network Proceedings, vol 5 (IEEE, 2006), pp. 4919–4924. https://doi.org/10.1109/IJCNN.2006.247192

  31. W. Liu, P. Pokharel, J. Principe, Error entropy, correntropy and m-estimation, in Proceedings of the 2006 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing, 2006 (IEEE, 2006), pp. 179–184

  32. W. Liu, P.P. Pokharel, J.C. Principe, Correntropy: properties and applications in non-Gaussian signal processing. IEEE Trans. Signal Process. 55(11), 1–30 (2007). https://doi.org/10.1109/TSP.2007.896065

    Article  MathSciNet  Google Scholar 

  33. L. Lu, H. Zhao, Active impulsive noise control using maximum correntropy with adaptive kernel size. Mech. Syst. Signal Process. 87, 180–191 (2017)

    Article  Google Scholar 

  34. S. Luan, T. Qiu, Y. Zhu, L. Yu, Cyclic correntropy and its spectrum in frequency estimation in the presence of impulsive noise. Signal Process. 120, 503–508 (2016). https://doi.org/10.1016/j.sigpro.2015.09.023

    Article  Google Scholar 

  35. W. Ma, B. Chen, H. Zhao, G. Gui, J. Duan, J.C. Principe, Sparse least logarithmic absolute difference algorithm with correntropy-induced metric penalty. Circuits Syst. Signal Process. 35(3), 1077–1089 (2016). https://doi.org/10.1007/s00034-015-0098-1

    Article  MathSciNet  Google Scholar 

  36. A. Mahmood, Digital communications in additive white symmetric alpha-stable noise. Ph.D Thesis, National University of Singapore (2014)

  37. D.W. Marquardt, An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  38. M. Mohammadi, G.A. Hodtani, M. Yassi, A robust correntropy-based method for analyzing multisample aCGH data. Genomics 106(5), 257–264 (2015). https://doi.org/10.1016/j.ygeno.2015.07.008

    Article  Google Scholar 

  39. E. Parzen, On estimation of a probability density function and mode. Ann. Math. Stat. 33(3), 1065–1076 (1962)

    Article  MathSciNet  MATH  Google Scholar 

  40. J.C. Principe, Information Theoretic Learning: Renyi’s Entropy and Kernel Perspectives (Springer, Berlin, 2010)

    Book  MATH  Google Scholar 

  41. J.C. Principe, D. Xu, Q. Zhao, J.W. Fisher III, Learning from examples with information theoretic criteria. J. VLSI Signal Process. 26(1–2), 61–77 (2000). https://doi.org/10.1023/A:1008143417156

    Article  MATH  Google Scholar 

  42. M. Rabinowitz, G.M. Gutt, G.F. Franklin, An adaptive Gauss–Newton algorithm for training multilayer nonlinear filters that have embedded memory. Circuits Syst. Signal Process. 18(4), 407–429 (1999). https://doi.org/10.1007/BF01200791

    Article  MATH  Google Scholar 

  43. M. Schatzman, M. Schatzman, Numerical Analysis: A Mathematical Introduction (Oxford University Press, Oxford, 2002)

    MATH  Google Scholar 

  44. J.S. Sheu, T.K. Woo, J.H. Wen, A novel convergence accelerator for the LMS adaptive filter. Circuits Syst. Signal Process. 31(1), 283–300 (2012). https://doi.org/10.1007/s00034-011-9287-8

    Article  MathSciNet  MATH  Google Scholar 

  45. A. Singh, J.C. Príncipe, A closed form recursive solution for maximum correntropy training, in 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) (IEEE, 2010), pp. 2070–2073

  46. R. Wang, B. Chen, N. Zheng, J.C. Principe, A variable step-size adaptive algorithm under maximum correntropy criterion, in 2015 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2015), pp. 1–5

  47. Z. Wu, S. Peng, B. Chen, H. Zhao, Robust hammerstein adaptive filtering under maximum correntropy criterion. Entropy 17(10), 7149–7166 (2015). https://doi.org/10.3390/e17107149

    Article  MathSciNet  Google Scholar 

  48. J. Xu, Nonlinear signal processing based on reproducing kernel, Ph.D. Dissertation, University of Florida (2007)

  49. N. Yamashita, M. Fukushima, On the rate of convergence of the Levenberg-Marquardt method, in Topics in Numerical Analysis.Analysis, ed. by G. Alefeld, X. Ehen, Computing Supplementa, vol. 15 (Springer, Berlin, 2001), pp. 239–249

  50. H. Yan, F. Qian, H. Zhang, F. Yang, G. Guo, \({H}_\infty \) fault detection for networked mechanical spring-mass systems with incomplete information. IEEE Trans. Ind. Electron. 63(9), 5622–5631 (2016)

    Article  Google Scholar 

  51. L. Yu, T. Qiu, A. Song, A time delay estimation algorithm based on the weighted correntropy spectral density. Circuits Syst. Signal Process. 36(3), 1115–1128 (2017)

    Article  MATH  Google Scholar 

  52. Y. Yu, H. Zhao, B. Chen, Z. He, Two improved normalized subband adaptive filter algorithms with good robustness against impulsive interferences. Circuits Syst. Signal Process. 35(12), 4607–4619 (2016). https://doi.org/10.1007/s00034-016-0289-4

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ghosheh Abed Hodtani.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Heravi, A.R., Hodtani, G.A. Comparison of the Convergence Rates of the New Correntropy-Based Levenberg–Marquardt (CLM) Method and the Fixed-Point Maximum Correntropy (FP-MCC) Algorithm. Circuits Syst Signal Process 37, 2884–2910 (2018). https://doi.org/10.1007/s00034-017-0694-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00034-017-0694-3

Keywords

Navigation