Skip to main content
Springer Nature Link
Log in

Analyzing the LMS Weight Error Covariance Matrix: An Exact Expectation Approach

  • Published:
Circuits, Systems, and Signal Processing Aims and scope Submit manuscript

Abstract

One of the most relevant tasks of high-dimensional estimation is the computation of the parameters covariance matrix. This matrix offers valuable insights into the uncertainties associated with the estimation process. In the context of adaptive filtering, the weight error covariance matrix is a key parameter that determines how the filter adapts to input statistics and noise levels. Additional statistical information about the weight error vector allows one to depict a more precise description of the stochastic coupling between the adaptive weights. This paper concentrates on an in-depth study of the least mean squares asymptotic weight error covariance matrix, employing the exact expectation analysis. Through this examination, a key conclusion emerges: The symmetries commonly engendered by traditional analyses are not present in the actual covariance matrix. Instead, these symmetries are artifacts of the widespread assumption of independence. In summary, the advanced analysis performed in this work reveals a significantly more nuanced learning behavior exhibited by the least mean squares algorithm, challenging the conventional understanding put forth by traditional approaches. The theoretical predictions are confirmed by extensive simulations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. A.Q.J. Althahab, A new robust adaptive algorithm based adaptive filtering for noise cancellation. Analog Integr. Circuit. Signal. Process 94(2), 217–231 (2018). https://doi.org/10.1007/s10470-017-1091-3

    Article  Google Scholar 

  2. S. Becker, J. Vielhaben, M. Ackermann et al., Audiomnist: exploring explainable artificial intelligence for audio analysis on a simple benchmark. J. Frankl. Inst. 1, 1 (2023). https://doi.org/10.1016/j.jfranklin.2023.11.038

    Article  Google Scholar 

  3. N. Bershad, L. Qu, on the probability density function of the lms adaptive filter weights, in ICASSP ’87. IEEE International Conference on Acoustics, Speech, and Signal Processing (1987), pp. 109–112. https://doi.org/10.1109/ICASSP.1987.1169750

  4. A.H. Bukhari, M.A.Z. Raja, M. Sulaiman et al., Fractional neuro-sequential arfima-lstm for financial market forecasting. IEEE Access 8, 71326–71338 (2020). https://doi.org/10.1109/ACCESS.2020.2985763

    Article  Google Scholar 

  5. H. Butterweck, An approach to lms adaptive filtering without use of the independence assumption, in 1996 8th European Signal Processing Conference (EUSIPCO 1996) (1996), pp. 1–4

  6. M.L.R. de Campos, G. Strang, QR decomposition an annotated bibliography, in QRD-RLS Adaptive Filtering. (Springer US, Boston, 2009), pp.1–22

    Google Scholar 

  7. J.F. Cardoso, Blind signal separation: statistical principles. Proc. IEEE 86(10), 2009–2025 (1998)

    Article  Google Scholar 

  8. H. Chen, X. Long, Y. Tang et al., Passive and h\(\infty \) control based on non-fragile observer for a class of uncertain nonlinear systems with input time-delay. J. Vib. Control (2023). https://doi.org/10.1177/10775463231193458

    Article  Google Scholar 

  9. Y.S. Choi, Subband adaptive filtering with-norm constraint for sparse system identification. Math. Probl. Eng. 2013, 1–7 (2013)

    MathSciNet  Google Scholar 

  10. P.S.R. Diniz, Adaptive Filtering (Springer International Publishing, Berlin, 2020). https://doi.org/10.1007/978-3-030-29057-3

    Book  Google Scholar 

  11. S. Douglas, T.Y. Meng, Exact expectation analysis of the LMS adaptive filter without the independence assumption, in 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1992. ICASSP-92 (IEEE, 1992), pp. 61–64

  12. S. Douglas, T.Y. Meng, Stochastic gradient adaptation under general error criteria. IEEE Trans. Signal Process. 42(6), 1335–1351 (1994). https://doi.org/10.1109/78.286951

    Article  Google Scholar 

  13. S.C. Douglas, Exact expectation analysis of the sign-data LMS algorithm for i.i.d. input data, in [1992] Conference Record of the Twenty-Sixth Asilomar Conference on Signals, Systems Computers, vol. 1 (1992), pp. 566–570

  14. S.C. Douglas, Exact expectation analysis of the LMS adaptive filter for correlated gaussian input data, in 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 3 (1993), pp. 519–522

  15. S.C. Douglas, Exact expectation analysis of the LMS adaptive filter for correlated Gaussian input data, in 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 3 (1993), pp. 519–522

  16. S.C. Douglas , T.H.Y. Meng, Exact expectation analysis of the LMS adaptive filter without the independence assumption, in [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing (1992), pp. 61–64

  17. S.C. Douglas, W. Pan, Exact expectation analysis of the LMS adaptive filter. IEEE Trans. Signal Process. 43(12), 2863–2871 (1995)

    Article  Google Scholar 

  18. S.C. Douglas, W. Pan, Exact expectation analysis of the LMS adaptive filter. IEEE Trans. Signal Process. 43(12), 2863–2871 (1995)

    Article  Google Scholar 

  19. E. Eweda, A new approach for analyzing the limiting behavior of the normalized LMS algorithm under weak assumptions. Signal Process. 89(11), 2143–2151 (2009)

    Article  Google Scholar 

  20. J. Foley, F. Boland, A note on the convergence analysis of lms adaptive filters with Gaussian data. IEEE Trans. Acoust. Speech Signal Process. 36(7), 1087–1089 (1988). https://doi.org/10.1109/29.1632

    Article  Google Scholar 

  21. S. Gazor, Prediction in lms-type adaptive algorithms for smoothly time varying environments. IEEE Trans. Signal Process. 47(6), 1735–1739 (1999). https://doi.org/10.1109/78.765152

    Article  Google Scholar 

  22. S. Guan, Z. Li, Normalised spline adaptive filtering algorithm for nonlinear system identification. Neural Process. Lett. 46(2), 595–607 (2017). https://doi.org/10.1007/s11063-017-9606-6

    Article  Google Scholar 

  23. Z. Habibi, H. Zayyani, Markovian adaptive filtering algorithm for block-sparse system identification. IEEE Trans. Circuits Syst. II Express Briefs 68(8), 3032–3036 (2021). https://doi.org/10.1109/TCSII.2021.3069879

    Article  Google Scholar 

  24. S. Haykin, B. Widrow, Least-Mean-Square Adaptive Filters. Wiley Online Library (2003)

  25. J. Huang, H. Chen, C. Shen, Event-triggered model-free adaptive control for wheeled mobile robot with time delay and external disturbance based on discrete-time extended state observer. J. Dyn. Syst. Meas. Control (2023). https://doi.org/10.1115/1.4063996

    Article  Google Scholar 

  26. S.Z. Islam, S.Z. Islam, R. Jidin et al, Performance study of adaptive filtering algorithms for noise cancellation of ecg signal, in 2009 7th International Conference on Information, Communications and Signal Processing (ICICS) (2009), pp. 1–5. https://doi.org/10.1109/ICICS.2009.5397744

  27. T. Kailath, Lectures on Wiener and Kalman Filtering (Springer, Vienna, 1981). https://doi.org/10.1007/978-3-7091-2804-6

    Book  Google Scholar 

  28. C.L. Keppenne, M. Ghil, Adaptive filtering and prediction of the southern oscillation index. J. Geophys. Res.: Atmos. 97(D18), 20449–20454 (1992)

    Article  Google Scholar 

  29. R. Kwong, E. Johnston, A variable step size lms algorithm. IEEE Trans. Signal Process. 40(7), 1633–1642 (1992). https://doi.org/10.1109/78.143435

    Article  Google Scholar 

  30. P. Lara, F. Igreja, L.D. Tarrataca et al., Exact expectation evaluation and design of variable step-size adaptive algorithms. IEEE Signal Process. Lett. 26(1), 74–78 (2018)

    Article  Google Scholar 

  31. P. Lara, K.S. Olinto, F.R. Petraglia et al., Exact analysis of the least-mean-square algorithm with coloured measurement noise. Electron. Lett. 54(24), 1401–1403 (2018)

    Article  Google Scholar 

  32. P. Lara, D.B. Haddad, L. Tarrataca, Advances on the analysis of the LMS algorithm with a colored measurement noise. SIViP 14, 529–536 (2019)

    Article  Google Scholar 

  33. P. Lara, L.D. Tarrataca, D.B. Haddad, Exact expectation analysis of the deficient-length LMS algorithm. Signal Process. 162, 54–64 (2019)

    Article  Google Scholar 

  34. P. Lara, F. Igreja, T.T.P. Silva et al., Exact expectation analysis of the LMS adaptive identification of non-linear systems. Electron. Lett. 56(1), 45–48 (2020)

    Article  Google Scholar 

  35. J. Li, P. Stoica, An adaptive filtering approach to spectral estimation and sar imaging. IEEE Trans. Signal Process. 44(6), 1469–1484 (1996)

    Article  Google Scholar 

  36. Y. Li, K.R. Liu, Static and dynamic convergence behavior of adaptive blind equalizers. IEEE Trans. Signal Process. 44(11), 2736–2745 (1996). https://doi.org/10.1109/78.542180

    Article  Google Scholar 

  37. D. Margaris, A. Kobusińska, D. Spiliotopoulos et al., An adaptive social network-aware collaborative filtering algorithm for improved rating prediction accuracy. IEEE Access 8, 68301–68310 (2020). https://doi.org/10.1109/ACCESS.2020.2981567

    Article  Google Scholar 

  38. V. Mathews, Z. Xie, A stochastic gradient adaptive filter with gradient adaptive step size. IEEE Trans. Signal Process. 41(6), 2075–2087 (1993). https://doi.org/10.1109/78.218137

    Article  Google Scholar 

  39. R. Morrison, R. Baptista, E. Basor, Diagonal nonlinear transformations preserve structure in covariance and precision matrices. J. Multivar. Anal. 190, 104983 (2022). https://doi.org/10.1016/j.jmva.2022.104983

    Article  MathSciNet  Google Scholar 

  40. B.B. Nair, V. Mohandas, N. Sakthivel et al, Application of hybrid adaptive filters for stock market prediction, in 2010 International Conference on Communication and Computational Intelligence (INCOCCI) (IEEE, 2010), pp. 443–447

  41. A.H. Sayed, Adaptive Filters. Wiley (2008). https://doi.org/10.1002/9780470374122

  42. H. Simon, Adaptive Filter Theory, vol. 2 (Prentice Hall, Hoboken, 2002), pp.478–481

    Google Scholar 

  43. V. Solo, LMS: past, present and future, in ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2019), pp. 7740–7744

  44. J. Tan, J. Zhang, An optimal adaptive filtering algorithm with a polynomial prediction model. Sci. China Inf. Sci. 54(1), 153–162 (2011). https://doi.org/10.1007/s11432-010-4141-3

    Article  MathSciNet  Google Scholar 

  45. N.V. Thakor, Y.S. Zhu, Applications of adaptive filtering to ecg analysis: noise cancellation and arrhythmia detection. IEEE Trans. Biomed. Eng. 38(8), 785–794 (1991)

    Article  Google Scholar 

  46. M.D.S. Vieitos, M.P. Tcheou, D.B. Haddad et al., Improved proportionate constrained normalized least mean square for adaptive beamforming. Circuits Syst. Signal Process. 42(12), 7651–7665 (2023). https://doi.org/10.1007/s00034-023-02459-3

    Article  Google Scholar 

  47. B. Widrow, J.M. McCool, M.G. Larimore et al., Stationary and nonstationary learning characteristics of the LMS adaptive filter. Proc. IEEE 64(8), 1151–1162 (1976). https://doi.org/10.1109/PROC.1976.10286

    Article  MathSciNet  Google Scholar 

  48. A. Zaknich, Principles of Adaptive Filters and Self-Learning Systems (Springer, London, 2005)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), in part by Coordenação de Aperfeiçoamento de Pessoal de Nível Superior Brasil (CAPES, Finance Code 001) and in part by Fundação Carlos Chagas Filho de Amparo á Pesquisa do Estado do Rio de Janeiro (FAPERJ).

Author information

Authors and Affiliations

  1. PhD in Instrumentation and Applied Optics (PPGIO), CEFET/RJ, Rua General Canabarro, Rio de Janeiro, RJ, 20271-204, Brazil

    Filipe Igreja, Ana L. F. de Barros & Diego B. Haddad

  2. Computer Engineering Department, CEFET/RJ, Rua do Imperador, 971, Petrópolis, RJ, 25620-003, Brazil

    Pedro Lara, Luís Tarrataca, Laura S. de Assis & Diego B. Haddad

  3. Electrical Engineering Program, Universidade Federal do Rio de Janeiro (UFRJ), Av. Athos da Silveira Ramos, 149, Centro de Tecnologia, Bloco H, Rio de Janeiro, RJ, 21941-972, Brazil

    Fernanda D. V. R. Oliveira

Authors
  1. Filipe Igreja
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Pedro Lara
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Luís Tarrataca
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Laura S. de Assis
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Fernanda D. V. R. Oliveira
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Ana L. F. de Barros
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Diego B. Haddad
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Filipe Igreja.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Igreja, F., Lara, P., Tarrataca, L. et al. Analyzing the LMS Weight Error Covariance Matrix: An Exact Expectation Approach. Circuits Syst Signal Process 43, 4390–4411 (2024). https://doi.org/10.1007/s00034-024-02656-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00034-024-02656-8

Keywords