Abstract
Correntropy as an efficient information theoretic (ITL) criterion has been extensively applied in many non-Gaussian applications. In order to maximize correntropy, several optimization algorithms have been proposed. Fixed-point maximum correntropy (FP-MCC) and correntropy-based Levenberg–Marquardt (CLM) are the fastest of the proposed methods. As the convergence rate of these operational methods has not been studied before, in this paper we prove the ability of their quadratic convergence from a theoretic point of view and establish the influence of bandwidth on their convergence order. Then, theoretic results are validated through numerical experiments.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig1_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig2_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig3_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig4_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig5_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig6_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig7_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig8_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig9_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig10_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig11_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig12_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig13_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig14_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig15_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig16_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig17_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig18_HTML.gif)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs00034-017-0694-3/MediaObjects/34_2017_694_Fig19_HTML.gif)
Similar content being viewed by others
References
S. Banach, Sur les operations dans les ensembles abstraits et leur application aux equations integrales. Fund. Math. 3(1), 133–181 (1922)
L. Boccato, D. Fantinato, D. Silva, R. Ferrari, A. Neves, R. Attux, Analysis of ITL criteria in the context of FIR channel equalization. J. Commun. Inf. Syst. 31(1), 1–29 (2016). https://doi.org/10.14209/jcis.2016.1
R.L. Burden, J.D. Faires, Numerical Analysis. 2001 (Brooks-Cole, Pacific Grove, 2001)
L. Cao, H.M. Schwartz, Exponential convergence of the Kalman filter based parameter estimation algorithm. Int. J. Adapt. Control Signal Process. 17(10), 763–783 (2003). https://doi.org/10.1002/acs.774
B. Chen, J. Wang, H. Zhao, N. Zheng, J.C. Principe, Convergence of a fixed-point algorithm under maximum correntropy criterion. IEEE Signal Process. Lett. 22(10), 1723–1727 (2015)
S. Dixit, D. Nagaria, Design and analysis of cascaded LMS adaptive filters for noise cancellation. Circuits Syst. Signal Process. (2016). https://doi.org/10.1007/s00034-016-0332-5
D. Erdogmus, Information theoretic learning: Renyi’s entropy and its applications to adaptive systems training. Ph.D. Dissertation, University of Florida (2002)
D. Erdogmus, R. Agrawal, J.C. Principe, A mutual information extension to the matched filter. Signal Process. 85(5 SPEC. ISS.), 927–935 (2005). https://doi.org/10.1016/j.sigpro.2004.11.018
D. Erdogmus, J.C. Principe, An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems. IEEE Trans. Signal Process. 50(7), 1780–1786 (2002). https://doi.org/10.1109/TSP.2002.1011217
J. Fan, The modified Levenberg–Marquardt method for nonlinear equations with cubic convergence. Math. Comput. 81(277), 447–466 (2012). https://doi.org/10.1090/S0025-5718-2011-02496-8
Jy Fan, Yx Yuan, On the quadratic convergence of the Levenberg–Marquardt method without nonsingularity assumption. Computing 74(1), 23–39 (2005). https://doi.org/10.1007/s00607-004-0083-1
M.M. Fateh, S. Khorashadizadeh, Robust control of electrically driven robots by adaptive fuzzy estimation of uncertainty. Nonlinear Dyn. 69(3), 1465–1477 (2012)
A. Garde, L. Sornmo, R. Jane, B.F. Giraldo, Correntropy-based spectral characterization of respiratory patterns in patients with chronic heart failure. IEEE Trans. Biomed. Eng. 57(8), 1964–1972 (2010). https://doi.org/10.1109/TBME.2010.2044176
M. Hajiabadi, G.A. Hodtani, H. Khoshbin, Adaptive multitask network based on maximum correntropy learning algorithm. Int. J. Adapt. Control Signal Process. (2017). https://doi.org/10.1002/acs.2760
H. Hammouri, M. Kinnaert, E. El Yaagoubi, Observer-based approach to fault detection and isolation for nonlinear systems. IEEE Trans. Autom. Control 44(10), 1879–1884 (1999)
S. Han, A family of minimum Renyi’s error entropy algorithm for information processing. Ph.D. Dissertation, University of Florida (2007)
R. He, B.-G. Hu, W.-S. Zheng, X.-W. Kong, Robust principal component analysis based on maximum correntropy criterion. IEEE Trans. Image Process. 20(6), 1485–1494 (2011). https://doi.org/10.1109/TIP.2010.2103949
R. He, L. Wang, Z. Sun, Y. Zhang, B. Li, Information theoretic subspace clustering. IEEE Trans. Neural Netw. Learn. Syst. PP(99), 1–13 (2015). https://doi.org/10.1109/TNNLS.2015.2500600
R. He, W.S. Zheng, B.G. Hu, X.W. Kong, Two-stage nonnegative sparse representation for large-scale face recognition. IEEE Trans. Neural Netw. Learn. Syst. 24(1), 35–46 (2013). https://doi.org/10.1109/TNNLS.2012.2226471
J. Heinonen, Lectures on Lipschitz analysis. Lecture Notes, pp. 1–77 (2005)
A.R. Heravi, G.A. Hodtani, A new robust correntropy based Levenberg–Marquardt algorithm, in 2016 Iran Workshop on Communication and Information Theory (IWCIT) (IEEE, 2016), pp. 1–6. https://doi.org/10.1109/IWCIT.2016.7491623
F. Huang, J. Zhang, S. Zhang, Adaptive filtering under a variable kernel width maximum correntropy criterion, IEEE Trans. Circuits Syst. II Express Briefs 64(10), 1247–1251 (2017)
R. Izanloo, S.A. Fakoorian, H.S. Yazdi, D. Simon, Kalman filtering based on the maximum correntropy criterion in the presence of non-Gaussian noise, in 2016 Annual Conference on Information Science and Systems (CISS) (IEEE, 2016), pp. 500–505. https://doi.org/10.1109/CISS.2016.7460553
R. Kalaba, A. Tishler, J. Wang, Rate of convergence of the generalized newton algorithm using the fixed-point approach. J. Optim. Theory Appl. 43(4), 543–555 (1984)
E.W. Kamen, A recursive parameter estimator yielding exponential convergence under sufficient excitation. Circuits Syst. Signal Process. 8(2), 207–228 (1989). https://doi.org/10.1007/BF01599938
A. Khalili, A. Rastegarnia, M.K. Islam, T.Y. Rezaii, Steady-state tracking analysis of adaptive filter with maximum correntropy criterion. Circuits Syst. Signal Process. 36(4), 1725–1734 (2017)
S. Khorashadizadeh, M.M. Fateh, Adaptive fourier series-based control of electrically driven robot manipulators, in 2013 3rd International Conference on Control, Instrumentation, and Automation (ICCIA) (IEEE, 2013), pp. 213–218
K. Levenberg, A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2(2), 164–168 (1944)
H. Li, Y. Gao, P. Shi, H.K. Lam, Observer-based fault detection for nonlinear systems with sensor fault and limited communication capacity. IEEE Trans. Autom. Control 61(9), 2745–2751 (2016)
W. Liu, P.P. Pokharel, J.C. Principe, Correntropy: A Localized Similarity Measure, in The 2006 IEEE International Joint Conference on Neural Network Proceedings, vol 5 (IEEE, 2006), pp. 4919–4924. https://doi.org/10.1109/IJCNN.2006.247192
W. Liu, P. Pokharel, J. Principe, Error entropy, correntropy and m-estimation, in Proceedings of the 2006 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing, 2006 (IEEE, 2006), pp. 179–184
W. Liu, P.P. Pokharel, J.C. Principe, Correntropy: properties and applications in non-Gaussian signal processing. IEEE Trans. Signal Process. 55(11), 1–30 (2007). https://doi.org/10.1109/TSP.2007.896065
L. Lu, H. Zhao, Active impulsive noise control using maximum correntropy with adaptive kernel size. Mech. Syst. Signal Process. 87, 180–191 (2017)
S. Luan, T. Qiu, Y. Zhu, L. Yu, Cyclic correntropy and its spectrum in frequency estimation in the presence of impulsive noise. Signal Process. 120, 503–508 (2016). https://doi.org/10.1016/j.sigpro.2015.09.023
W. Ma, B. Chen, H. Zhao, G. Gui, J. Duan, J.C. Principe, Sparse least logarithmic absolute difference algorithm with correntropy-induced metric penalty. Circuits Syst. Signal Process. 35(3), 1077–1089 (2016). https://doi.org/10.1007/s00034-015-0098-1
A. Mahmood, Digital communications in additive white symmetric alpha-stable noise. Ph.D Thesis, National University of Singapore (2014)
D.W. Marquardt, An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1963)
M. Mohammadi, G.A. Hodtani, M. Yassi, A robust correntropy-based method for analyzing multisample aCGH data. Genomics 106(5), 257–264 (2015). https://doi.org/10.1016/j.ygeno.2015.07.008
E. Parzen, On estimation of a probability density function and mode. Ann. Math. Stat. 33(3), 1065–1076 (1962)
J.C. Principe, Information Theoretic Learning: Renyi’s Entropy and Kernel Perspectives (Springer, Berlin, 2010)
J.C. Principe, D. Xu, Q. Zhao, J.W. Fisher III, Learning from examples with information theoretic criteria. J. VLSI Signal Process. 26(1–2), 61–77 (2000). https://doi.org/10.1023/A:1008143417156
M. Rabinowitz, G.M. Gutt, G.F. Franklin, An adaptive Gauss–Newton algorithm for training multilayer nonlinear filters that have embedded memory. Circuits Syst. Signal Process. 18(4), 407–429 (1999). https://doi.org/10.1007/BF01200791
M. Schatzman, M. Schatzman, Numerical Analysis: A Mathematical Introduction (Oxford University Press, Oxford, 2002)
J.S. Sheu, T.K. Woo, J.H. Wen, A novel convergence accelerator for the LMS adaptive filter. Circuits Syst. Signal Process. 31(1), 283–300 (2012). https://doi.org/10.1007/s00034-011-9287-8
A. Singh, J.C. Príncipe, A closed form recursive solution for maximum correntropy training, in 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) (IEEE, 2010), pp. 2070–2073
R. Wang, B. Chen, N. Zheng, J.C. Principe, A variable step-size adaptive algorithm under maximum correntropy criterion, in 2015 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2015), pp. 1–5
Z. Wu, S. Peng, B. Chen, H. Zhao, Robust hammerstein adaptive filtering under maximum correntropy criterion. Entropy 17(10), 7149–7166 (2015). https://doi.org/10.3390/e17107149
J. Xu, Nonlinear signal processing based on reproducing kernel, Ph.D. Dissertation, University of Florida (2007)
N. Yamashita, M. Fukushima, On the rate of convergence of the Levenberg-Marquardt method, in Topics in Numerical Analysis.Analysis, ed. by G. Alefeld, X. Ehen, Computing Supplementa, vol. 15 (Springer, Berlin, 2001), pp. 239–249
H. Yan, F. Qian, H. Zhang, F. Yang, G. Guo, \({H}_\infty \) fault detection for networked mechanical spring-mass systems with incomplete information. IEEE Trans. Ind. Electron. 63(9), 5622–5631 (2016)
L. Yu, T. Qiu, A. Song, A time delay estimation algorithm based on the weighted correntropy spectral density. Circuits Syst. Signal Process. 36(3), 1115–1128 (2017)
Y. Yu, H. Zhao, B. Chen, Z. He, Two improved normalized subband adaptive filter algorithms with good robustness against impulsive interferences. Circuits Syst. Signal Process. 35(12), 4607–4619 (2016). https://doi.org/10.1007/s00034-016-0289-4
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Heravi, A.R., Hodtani, G.A. Comparison of the Convergence Rates of the New Correntropy-Based Levenberg–Marquardt (CLM) Method and the Fixed-Point Maximum Correntropy (FP-MCC) Algorithm. Circuits Syst Signal Process 37, 2884–2910 (2018). https://doi.org/10.1007/s00034-017-0694-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00034-017-0694-3