Skip to main content
SpringerLink
Log in

A new and fast correntropy-based method for system identification with exemplifications in low-SNR communications regime

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

One of the most significant issues in machine learning is system identification with many applications, e.g., channel estimation (CE) in digital communications. Introducing a new correntropy-based method, this paper deals with the comparison between mean square error (MSE) and information theoretic measures in non-Gaussian noise channel estimation, by analyzing the MSE, minimum error entropy (MEE) and correntropy algorithms in several channel models utilizing neural networks. The first contribution of this paper is introducing a new correntropy-based conjugate gradient (CCG) method and applying it in the CE problem, which this new algorithm converges faster than standard maximum correntropy criterion algorithm. Aiming at this contribution, the better convergence rate is discussed analytically and it is proved that the CCG could converge to the optimal solution quadratically. Next, the performance of an extended MSE algorithm is compared with information theoretic criteria; in addition, a comparison between MEE and correntropy-based algorithm is presented. The Monte Carlo results illustrate that correntropy and MEE outperform MSE algorithm in low-SNR communications especially in the presence of impulsive noise. Then, we apply the trained neural networks in the receiver as an equalizer to obtain the intended performance for different SNR values.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Salehi M, Proakis J (2008) Digital communications, 5th edn. MCGraw-Hill, New York

    Google Scholar 

  2. Mulgrew B, Cowan CF (2012) Adaptive filters and equalisers. Springer Science & Business Media, Berlin

    Google Scholar 

  3. Ekman T (2002) Prediction of mobile radio channels: modeling and design. Ph.d. dissertation, Uppsala University

  4. Colieri S, Ergen M, Puri A, Bahai A (2002) A study of channel estimation in OFDM systems. In: Proceedings IEEE 56th vehicular technology conference, volume 2, pp 894–898

  5. Banani SA, Vaughan RG (2011) Blind channel estimation for equalisation in dispersive fading channel. IET Commun 5(11):1577–1586

    Article  MathSciNet  MATH  Google Scholar 

  6. Coleri S, Ergen M, Puri A, Bahai A (2002) Channel estimation techniques based on pilot arrangement in OFDM systems. IEEE Trans Broad 48(3):223–229

    Article  Google Scholar 

  7. Amleh K, Li H, Wang R (2007) Blind channel estimation, equalisation and CRB for OFDM with unmodelled interference. IET Commun 1(3):489

    Article  MathSciNet  MATH  Google Scholar 

  8. Noh M, Lee Y, Park H (2006) Low complexity LMMSE channel estimation for OFDM. IEE Proc Commun 153(5):645

    Article  Google Scholar 

  9. Eykhofl P (1974) System identification. Parameter and state estimation. Wiley, New York

    Google Scholar 

  10. Ljung L (1998) System identification. In signal analysis and prediction. Springer, Berlin, pp 163–173

    Book  Google Scholar 

  11. Fausett L (1994) Fundamentals of neural networks: architectures, algorithms, and applications. Prentice-Hall Inc., Upper Saddle River

    MATH  Google Scholar 

  12. Patra JC, Pal RN, Baliarsingh R, Panda G (1999) Nonlinear channel equalization for QAM signal constellation using artificial neural networks. IEEE Trans Syst Man Cybern B Cybern 29(2):262–271

    Article  Google Scholar 

  13. Siu S, Gibson GJ, Cowan CFN (1989) Decision feedback equalization using neural network structures. In: First IEE international conference on artificial neural networks (Conf. Publ. No. 313), pp 125–128

  14. Qureshi IM, Naveed A (2004) Blind equalization and estimation of channel using artificial neural networks. In: 8th International multitopic conference, 2004. Proceedings of INMIC, pp 184–190

  15. Zhang L, Zhang X (2007) MIMO channel estimation and equalization using three-layer neural networks with feedback. Tsinghua Sci Technol 12(6):658–662

    Article  Google Scholar 

  16. Zhang JS, Zhao HQ, Zeng XP, Li TR (2011) Equalisation of non-linear time-varying channels using a pipelined decision feedback recurrent neural network filter in wireless communication systems. IET Commun 5(3):381–395

    Article  MathSciNet  Google Scholar 

  17. Jun S, Dong-Feng Y (2006) Neural network channel estimation based on least mean error algorithm in the OFDM systems. Springer, Berlin, pp 706–711

    Google Scholar 

  18. Necmi T, Nuri Seyman M (2010) Back propagation neural network approach for channel estimation in OFDM system. In 2010 IEEE international conference on wireless communications, networking and information security, pp 265–268. IEEE

  19. Mitra A, Sarma KK (2013) Multiple-input multiple-output channel modelling using multi-layer perceptron with finite impulse response and infinite impulse response synapses. IET Commun 7(14):1540–1549

    Article  Google Scholar 

  20. Weidemann H, Stear E (1970) Entropy analysis of estimating systems. IEEE Trans Inf Theory 16(3):264–270

    Article  MathSciNet  MATH  Google Scholar 

  21. Erdogmus D, Principe JC (2002) An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems. IEEE Trans Signal Process 50(7):1780–1786

    Article  Google Scholar 

  22. Deniz E (2002) Information theoretic learning: Renyi’s entropy and its applications to adaptive systems training. Ph.d. dissertation, University of Florida

  23. Han S, Rao S, Erdogmus D, Jeong K-H, Principe J (2007) A minimum-error entropy criterion with self-adjusting step-size (MEE-SAS). Signal Process 87(11):2733–2745

    Article  MATH  Google Scholar 

  24. Seungju H (2007) A family of minimum Renyi’S error entropy algorithm for information processing. Ph.d. dissertation, University of florida

  25. Erdogmus D, Principe JC, Hild II KE (2002) Beyond second-order statistics for learning: a pairwise interaction model for entropy estimation. Nat Comput 1(1):85–108

    Article  MathSciNet  MATH  Google Scholar 

  26. Santamaría I, Erdogmus D, Principe JC (2002) Entropy minimization for supervised digital communications channel equalization. IEEE Trans Signal Process 50(5):1184–1192

    Article  Google Scholar 

  27. Erdogmus D, Hild KE, Principe JC, Lazaro M, Santamaria I (2004) Adaptive blind deconvolution of linear channels using Renyi’s entropy with Parzen window estimation. IEEE Trans Signal Process 52(6):1489–1498

    Article  MathSciNet  MATH  Google Scholar 

  28. Singh A, Principe JC (2009) Using correntropy as a cost function in linear adaptive filters. Int Joint Conf Neural Netw 1:2950–2955

    Google Scholar 

  29. Verdu S (1996) The exponential distribution in information theory. Problemy peredachi informatsii 32(1):100–111

    MathSciNet  MATH  Google Scholar 

  30. Zimmermann M, Dostert K (2002) Analysis and modeling of impulsive noise in broad-band powerline communications. IEEE Trans Electromagn Compat 44(1):249–258

    Article  Google Scholar 

  31. Bessa RJ, Miranda V, Gama J (2009) Entropy and correntropy against minimum square error in offline and online three-day ahead wind power forecasting. IEEE Trans Power Syst 24(4):1657–1666

    Article  Google Scholar 

  32. Zhang Y, Meratnia N, Havinga P (2010) Outlier detection techniques for wireless sensor networks: a survey. IEEE Commun Surv Tutor 12(2):159–170

    Article  Google Scholar 

  33. Berger LT, Schwager A, Pagani P, Schneider DM (2015) Mimo power line communications. IEEE Commun Surv Tutor 17(1):106–124

    Article  Google Scholar 

  34. Nirenberg L (1975) Low SNR digital communication over certain additive non-gaussian channels. IEEE Trans Commun 23(3):332–341

    Article  MathSciNet  MATH  Google Scholar 

  35. Erdogmus D, Principe JC (2002) Generalized information potential criterion for adaptive system training. IEEE Trans Neural Netw 13(5):1035–1044

    Article  Google Scholar 

  36. Liu W, Pokharel PP, Principe JC (2007) Correntropy: properties and applications in non-Gaussian signal processing. IEEE Trans Signal Process 55(11):5286–5298

    Article  MathSciNet  MATH  Google Scholar 

  37. Ma W, Qu H, Gui G, Zhao J, Chen B (2015) Maximum correntropy criterion based sparse adaptive filtering algorithms for robust channel estimation under non-Gaussian environments. J Franklin Inst 352:2708–2727

    Article  MATH  Google Scholar 

  38. Songlin Z, Badong C, Jose CP (2011) Kernel adaptive filtering with maximum correntropy criterion. In: Proceedings of the international joint conference on neural networks, pp 2012–2017

  39. Shi L, Lin Y (2014) Convex combination of adaptive filters under the maximum correntropy criterion in impulsive interference. IEEE Signal Process Lett 21(11):1385–1388

    Article  Google Scholar 

  40. Izanloo R, Fakoorian SA, Yazdi HS, Simon D (2016) Kalman filtering based on the maximum correntropy criterion in the presence of non-Gaussian noise. In: 2016 Annual conference on information science and systems (CISS) IEEE, pp 500–505

  41. Wang R, Chen B, Zheng N, Principe JC (2015) A variable step-size adaptive algorithm under maximum correntropy criterion. In: 2015 International joint conference on neural networks (IJCNN), vol 6, pp 1–5. IEEE

  42. He R, Zheng W-S, Bao-Gang H, Kong X-W (2011) A regularized correntropy framework for robust pattern recognition. Neural Comput 23:2074–2100

    Article  MATH  Google Scholar 

  43. Mujahid N. Syed, Jose C. Principe, Panos M. Pardalos (2012) Correntropy in Data Classification. pages 81–117

  44. Ren H-R, Xing H-R (2012) Robust feature extraction for novelty detection based on regularized correntropy criterion. In: 2012 IEEE international conference on systems, man, and cybernetics (SMC), pp 975–980. IEEE

  45. Li L, Yang J, Xu Y, Qin Z, Zhang H (2014) Documents clustering based on max-correntropy nonnegative matrix factorization. In: 2014 International conference on machine learning and cybernetics, pp 850–855. IEEE

  46. Nayyeri M, Noghabi HS (2016) Cancer classification by correntropy-based sparse compact incremental learning machine. Gene Rep 3:31–38

    Article  Google Scholar 

  47. Chen L, Hua Q, Zhao J, Chen B, Principe JC (2016) Efficient and robust deep learning with Correntropy-induced loss function. Neural Comput Appl 27(4):1019–1031

    Article  Google Scholar 

  48. Khalili A, Rastegarnia A, Darvish B (2015) A robust adaptive carrier frequency offset estimation algorithm for OFDM. Am J Signal Process 5:26–31

    Google Scholar 

  49. Mohammadi M, Hodtani GA, Yassi M (2015) A robust Correntropy-based method for analyzing multisample aCGH data. Genomics 106(5):257–264

    Article  Google Scholar 

  50. Luan S, Qiu T, Zhu Y, Ling Y (2016) Cyclic correntropy and its spectrum in frequency estimation in the presence of impulsive noise. Signal Process 120:503–508

    Article  Google Scholar 

  51. Ling Y, Qiu T-S, Song A-M (2016) A time delay estimation algorithm based on the weighted correntropy spectral density. Circuits Syst Signal Process 36(3):1115–1128

    MATH  Google Scholar 

  52. Heravi AR, Hodtani GA (2017) Comparison of the convergence rates of the new Correntropy-based Levenberg–Marquardt (CLM) method and the Fixed-Point Maximum Correntropy (FP-MCC) algorithm. Circuits Syst Signal Process 1–27. https://doi.org/10.1007/s00034-017-0694-3

  53. He R, Zheng WS, Hu BG (2011) Maximum correntropy criterion for robust face recognition. IEEE Trans Pattern Anal Mach Intell 33(8):1561–1576

    Article  Google Scholar 

  54. He R, Zheng W-S, Bao-Gang H, Kong X-W (2013) Two-stage nonnegative sparse representation for large-scale face recognition. IEEE Trans Neural Netw Learn Syst 24(1):35–46

    Article  Google Scholar 

  55. Huang C, Zeng L (2015) Robust image segmentation using local robust statistics and correntropy-based K-means clustering. Opt Lasers Eng 66:187–203

    Article  Google Scholar 

  56. Álvarez-Meza AM, Molina-Giraldo S, Castellanos-Dominguez G (2016) Background modeling using object-based selective updating and correntropy adaptation. Image Vis Comput 45:22–36

    Article  Google Scholar 

  57. Van Der Malsburg C (1986) Frank Rosenblatt: Principles of neurodynamics: perceptrons and the theory of brain mechanisms. In Brain theory. Springer, Berlin, pp 245–248

    Google Scholar 

  58. Ayat S, Farahani HA, Aghamohamadi M, Alian M, Aghamohamadi S, Kazemi Z (2013) A comparison of artificial neural networks learning algorithms in predicting tendency for suicide. Neural Comput Appl 23(5):1381–1386

    Article  Google Scholar 

  59. Nocedal SJ, Wright J (2006) Numerical optimization. Springer series in operations research and financial engineering. Springer, New York

    Google Scholar 

  60. Brent RP (2013) Algorithms for minimization without derivatives. Courier Corporation, North Chelmsford

    MATH  Google Scholar 

  61. Shi ZJ, Shen J (2005) New inexact line search method for unconstrained optimization. J Optim Theory Appl 127(2):425–446

    Article  MathSciNet  MATH  Google Scholar 

  62. Fletcher R (1964) Function minimization by conjugate gradients. Comput J 7(2):149–154

    Article  MathSciNet  MATH  Google Scholar 

  63. Hestenes MR, Stiefel E (1952) Methods of conjugate gradients for solving linear systems. J Res Nat Bur Stand 49(6):411–436

    Article  MathSciNet  MATH  Google Scholar 

  64. Polak E, Ribiere G (1969) Note sur la convergence de directions conjugees. Rev Fr Inform Rech Oper 16:35–43

    MATH  Google Scholar 

  65. Polyak BT (1969) The conjugate gradient method in extremal problems. USSR Comput Math Math Phys 9(4):94–112

    Article  MATH  Google Scholar 

  66. Liu Y, Storey C (1991) Efficient generalized conjugate gradient algorithms, part 1: theory. J Optim Theory Appl 69(1):129–137

    Article  MathSciNet  MATH  Google Scholar 

  67. Dai YH, Yuan Y (1999) A nonlinear conjugate gradient method with a strong global convergence property. SIAM J Optim 10(1):177–182

    Article  MathSciNet  MATH  Google Scholar 

  68. Hager WW, Zhang H (2006) Algorithm 851: CG_DESCENT, a conjugate gradient method with guaranteed descent. ACM Trans Math Softw (TOMS) 32(1):113–137

    Article  MATH  Google Scholar 

  69. Hager WW, Zhang H (2006) A survey of nonlinear conjugate gradient methods. Pac J Optim 2(1):35–58

    MathSciNet  MATH  Google Scholar 

  70. Dai YH, Yuan Y (2001) An efficient hybrid conjugate gradient method for unconstrained optimizations. Ann Oper Res 103(1/4):33–47

    Article  MathSciNet  MATH  Google Scholar 

  71. Dai Y, Yuan Y (2001) A three-parameter family of nonlinear conjugate gradient methods. Math Comput 70(235):1155–1167

    Article  MathSciNet  MATH  Google Scholar 

  72. Shamir O, Zhang T (2013) Stochastic gradient descent for non-smooth optimization: convergence results and optimal averaging schemes. International conference on machine learning, pp 71–79

  73. van der Sluis A, van der Vorst H (1986) The rate of convergence of conjugate gradients. Numerische Mathematik 48(5):543–560

    Article  MathSciNet  MATH  Google Scholar 

  74. Strakos Z (1991) On the real convergence rate of the conjugate gradient method. Linear Algebra Appl 154:535–549

    Article  MathSciNet  MATH  Google Scholar 

  75. Cohen AI (1972) Rate of convergence of several conjugate gradient algorithms. SIAM J Numer Anal 9(2):248–259

    Article  MathSciNet  MATH  Google Scholar 

  76. Axelsson O, Karátson J (2002) On the rate of convergence of the conjugate gradient method for linear operators in hilbert space

  77. Mahmood A (2014) Digital communications in addtive white symmetric alpha-stable noise. Ph.d thesis, National University of Singapore

Download references

Author information

Authors and Affiliations

  1. Electrical Engineering Department, Ferdowsi University of Mashhad, Mashhad, Iran

    Ahmad Reza Heravi & Ghosheh Abed Hodtani

Authors
  1. Ahmad Reza Heravi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ghosheh Abed Hodtani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ghosheh Abed Hodtani.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Heravi, A.R., Hodtani, G.A. A new and fast correntropy-based method for system identification with exemplifications in low-SNR communications regime. Neural Comput & Applic 31, 4407–4422 (2019). https://doi.org/10.1007/s00521-017-3306-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-017-3306-z

Keywords

Search

Navigation