Elsevier

Signal Processing

Volume 92, Issue 4, April 2012, Pages 1044-1068
Signal Processing

On-line adaptive principal component extraction algorithms using iteration approach

https://doi.org/10.1016/j.sigpro.2011.10.016Get rights and content

Abstract

Two new on-line algorithms for adaptive principal component analysis (APCA) are proposed and discussed in order to solve the problem of on-line industrial process monitoring in this paper. Both the algorithms have the capability of extracting principal component eigenvectors on-line in a fixed size sliding data window with high dimensional input data. The first algorithm is based on the steepest gradient descent approach, which updates the covariance matrix with deflation transformation and on-line iteration. Based on neural networks, the second algorithm constructs the input data sequence with an on-line iteration method and trains the neural network in every data frame. The convergence of the two algorithms is then analyzed and the simulations are given to illustrate the effectiveness of the two algorithms. At last, the applications of the two algorithms are discussed.

Highlights

► We propose two new online algorithms for APCA. ► Algorithms are applied to the online industrial process monitoring. ► Both of the algorithms can extract principal component on-line. ► Both of the algorithms can deal with high-dimensional input data.

Introduction

Principal component analysis (PCA) is a feature extraction technology which derives an optimal linear transformation for a given target space dimension. The optimality criterion is based on least mean square error of data reconstructed from the actual principal component. At present, PCA is widely used in pattern recognition, data fusion, security monitoring and other fields.

Unlike PCA, adaptive principal component analysis (APCA) uses a mathematical model for adapting principal components which vary with the correlation of the system states. For instance, a novel neural network [17] is proposed to solve the smallest singular component of general matrix, based on an extension of the Hebbian rule and a modification of cross-coupled Hebbian rule. Moreover, an adaptive kernel principal component analysis method [18] is proposed, which has the flexibility to accurately track the kernel principal components (KPC). Recently, APCA is widely used in process monitoring [1], signal processing [2], [3], fault diagnosis [4]and image processing [5], [6], [7].

However, conventional APCA algorithms are of limited use in many application fields where require real-time and high dimensional data processing (on-line monitoring of industrial processes). Thus, in the last decade, four classes of approaches have been proposed to address the needs. The first class is the recursive PCA algorithms [8], in which rank-one modification and Lanczos tridiagonalization are used. While this approach is computationally inexpensive, it would inevitably result in cumulative error due to the existence of neglected terms in the perturbation method (which is used in the algorithm). The second class of methods used to analyze principal component of stochastic data is based on incremental singular value decomposition (SVD) [9] which avoids a direct computation of covariance matrix. Nevertheless, an off-line computation of singular value decomposition is necessary after the simplification of the input matrix. (Usually, the matrix is decomposed by Lanczos algorithm [10] as the computational complexity of Lanczos algorithm is O(n3).) Furthermore, the orthogonality of the principal component eigenvectors which are derived from Lanczos algorithm is not very strong. Therefore, this approach is not suitable for large matrix computation in spite of the absence of cumulative error. The third class is to use the objective function [11]. By searching the optimal point of the objective function, principal component eigenvectors can be extracted and some gradient descent technique is used to accelerate the searching process. The biggest drawback lies on the assignment of searching step, which would inevitably increase the computational complexity. In contrast, the fourth class is based on neural network, e.g. an algorithm based on neural network for adaptive principal component extraction (APEX) [12], proposed by Kung and Diamantaras, is computationally inexpensive and suitable for extracting large matrix (which is constructed by input data). Nevertheless, the convergence of the algorithm based on the principle of gradient descent is greatly influenced by the length of the iteration step. In addition, their off-line algorithms cannot be applied to on-line industrial process monitoring.

Most of the above methods are only applicable to zero-mean stochastic process. New methods are needed to process high dimensional non-stationary data. In order to meet the requirement of industrial applications, this paper proposes two algorithms of on-line APCA based on steepest gradient descent and neural networks enabling rapid and accurate computation for extracting eigenvectors of large matrix. The basis of our solution lies in adaptively adjusting principal component eigenvectors based on gradient descent method and neural network training. The contribution made in this paper is twofold: (1) two algorithms are developed for recursively extracting eigenvectors to adapt to the changing characteristics of high dimensional non-stationary data and (2) the performances of the two proposed algorithms are compared by simulation to illustrate their usefulness in applications.

The rest of the paper is organized as follows. Section 2 presents a brief description of principal component extraction based on gradient descent and neural network. Section 3 describes the proposed on-line principal component extraction approach based on steepest gradient descent principle, including the iteration of covariance matrix and optimal searching technology for eigenvector extraction. Section 4 describes the proposed on-line principal component extraction based on neural network. Section 5 discusses the simulation results of the algorithms. The computational complexity is also analyzed in the last part of Section 5. And the paper is concluded in Section 6.

Section snippets

Principal component extraction by gradient descent algorithm

APEX algorithm based on gradient descent approach (simplified into GDAPEX) has exhibited high precision and superb adaptive ability [11]. The rate of converging to principal component eigenvectors can be accelerated using optimal searching technique. The objective function is given asJ(wki)=t=1kβktxtj=1i1wkjwkjTxtwkiwkiTxtwhere xt is the kth input vector in time sequence. wkj is the estimated eigenvector of covariance matrix, which will converge to the principal component eigenvector. β

An on-line algorithm for APCA based on steepest gradient descent algorithm

In this section, a solution of on-line adaptive PCA based on steepest gradient descent algorithm is developed for the actual applications. While frequently used in other studies, a forgotten factor is not used here, for the result of APCA may be influenced by the input data which do not include current system state information after the forgotten factor has been used for a long time. Instead, a limited memory sliding data window (simplified into LMSDW) is used and the principal component

On-line algorithm for adaptive PCA based on neural network

NAPEX has been introduced in previous Section 2. For the application of on-line process monitoring, a novel algorithm of on-line NAPEX will be given in the following part. At the end of this section, the convergence of this algorithm is analyzed.

The on-line NAPEX algorithm can be simply described in three steps. First, input on-line data frame into this algorithm, and construct the input data for neural network training; second, train the neural network; third, extract principal components from

Simulation result and analysis

Several simulations were conducted to examine five properties of the proposed algorithms: (1) the effectiveness of the proposed methods when dealing with high dimensional zero-mean stationary data; (2) the effectiveness of the proposed methods when dealing with non-stationary data; (3) the effectiveness of the proposed methods when dealing with varied-mean non-stationary data; (4)the effectiveness of the proposed methods when dealing with varied LMSDW sized data; (5) analytical and empirical

Conclusion

In terms of computation capability, both the on-line algorithms described above have the capability of extracting principal component eigenvector on-line under the condition of high dimensional input data in some application fields. The performances of the two algorithms are both affected by the covariance variation of the input data in time sequence. The fewer principal components exist, the better the algorithms perform. If the size of sliding data window is very small, the neural network

Acknowledgments

This work is supported in part by the National Nature Science Foundation of China under Grant no. 60974090, Doctoral Fund of Ministry of Education of China Grant no. 102063720090013 and Fundamental Research Funds for the Central Universities Grant no. GDJXS10170010, Mr. Ho Simon Wang, a freelance technical editor reachable at [email protected], has helped improve the manuscript.

References (19)

  • W.H. Li et al.

    Recursive PCA for adaptive process monitoring

    Journal of Process Control

    (2000)
  • D.Z. Feng et al.

    Cross-correlation neural network models for the smallest singular component of general matrix

    Signal Processing

    (1998)
  • M.T. Ding et al.

    Adaptive kernel principal component analysis

    Signal Processing

    (2010)
  • Z.Q. Ge et al.

    Semiconductor manufacturing process monitoring based on adaptive substatistical PCA

    IEEE Transactions on Semiconductor Manufacturing

    (2010)
  • A. Perera et al.

    On-line novelty detection by recursive dynamic principal component analysis and gas sensor arrays under drift conditions

    Sensors Journal, IEEE

    (2006)
  • X.L. Zhu et al.

    Adaptive nonlinear PCA algorithms for blind source separation without prewhitening

    IEEE Transactions on Circuits and Systems

    (2006)
  • B.Z. Lu, Y.H. Zhao, Z.H. Mao, Fault diagnosis method based on moving window PCA, in: Proceedings of the Control and...
  • L. Zhang et al.

    PCA-based spatially adaptive denoising of CFA Images for single-sensor digital cameras

    IEEE Transactions on Image Processing

    (2009)
  • C. Cocianu, L. State, P.V. Lamos, A new adaptive PCA scheme for noise removal in image processing, in: Proceedings of...
There are more references available in the full text version of this article.

Cited by (6)

  • Compression of results of geodetic displacement measurements using the PCA method and neural networks

    2020, Measurement: Journal of the International Measurement Confederation
    Citation Excerpt :

    There are many algorithms for determining the main components. The most popular of them are the Hebb algorithm, generalized by and presented in [56,57], and the APEX algorithm (Adaptive Principal Component EXtraction) [58,59]. The Hebba rule used in the PCA network has properties to reduce the level of noise that distorts data.

  • Inferential sensor-based adaptive principal components analysis of mould bath level for breakout defect detection and evaluation in continuous casting

    2015, Applied Soft Computing Journal
    Citation Excerpt :

    There are many related works in the field of soft sensor for quality prediction and evaluation applied in different systems and processes, the following are some of them: The prediction and evaluation of the key factors characterizing the product quality in various processes using soft sensing based multivariate statistical process control, fuzzy means and multivariate identification [1–10]. Soft sensing methods including support vector machine, data driven and data mining [11–19].

  • Uncertainty estimation of mechanical testing properties using sensitivity analysis and stochastic modelling

    2015, Measurement: Journal of the International Measurement Confederation
    Citation Excerpt :

    There are few related works in the field of the uncertainty evaluation in mechanical and metallurgical testing, from which we cite: The prediction and evaluation of the key factors characterising the product quality in various processes using soft sensing based Multivariate Statistical Process control, Fuzzy means and multivariate identification [1–10]. Soft Sensing methods including Support Vector Machine, data driven and data mining [11–15].

  • Inferential sensor - Based adaptive principal components analysis for mechanical properties prediction and evaluation

    2013, Measurement: Journal of the International Measurement Confederation
    Citation Excerpt :

    There are many related works in the field of soft sensor for quality prediction and evaluation, from which we cite: The prediction and evaluation of the key factors characterizing the product quality in various processes using soft sensing based Multivariate Statistical Process Control, Fuzzy means and multivariate identification [1–10]. Soft sensing methods including Support Vector Machine, data driven and data mining [11–15].

View full text