A joint-norm distance metric 2DPCA for robust dimensionality reduction
Introduction
Dimensionality reduction is widely used in image processing, pattern recognition and knowledge extraction [1], [2], [3]. It projects the original high-dimensional data to a low-dimensional subspace via linear or nonlinear mapping [4], [5], mainly including principal component analysis (PCA) [6], linear discriminant analysis (LDA) [7], independent component analysis (ICA) [8], etc. PCA and two-dimensional PCA (2DPCA) [9], as the most representative dimensionality reduction methods, find an optimal projection subspace to achieve the two objectives of maximum projection variance and minimum reconstruction error. Both methods utilize the squared F-norm (also called squared -norm) as the distance metric, which is sensitive to noise or outliers hidden in images [10]. To enhance the robustness of algorithms, three similar metric learning approaches [11] are commonly performed in their objective functions, i.e., replacing the squared F-norm with other norms, optimizing two objectives with a single norm by adjusting the objective function, and using two different norms to jointly measure the distances of the two objectives.
Various robust PCA and 2DPCA methods were proposed for optimizing a single objective of the maximum projection distance or the minimum reconstruction error by replacing the squared F-norm with the -norm, -norm and -norm (including the corresponding vector forms such as the -norm, -norm and -norm). To achieve robust PCA performance, -norm distance metric can effectively suppress the effect of outliers [12]. The greedy and non-greedy solution algorithms for PCA- were put forward [13], [14]. Subsequently, -norm-based 2DPCA with greed (2DPCA-) and non-greedy (2DPCA--NG) solutions were proposed, respectively [15], [16]. Then, the -norm metric was also introduced into sparse PCA and sparse 2DPCA. Robust sparse PCA (RSPCA) applied the -norm in the objective function and constraint function [17]. 2DPCA- with sparsity (2DPCA--S) was performed by combining 2DPCA- with RSPCA to implement robustness and sparse modeling simultaneously [18]. Besides, block PCA with the -norm greedy and non-greedy algorithms were also researched [19], [20]. Considering that the -norm is a special case of the -norm, a greedy algorithm based on the gradient ascent method and the Lagrangian multiplier method was presented for PCA- with the objective function measured by the -norm [21]. To further extend PCA to more general cases, a generalized PCA (GPCA) applied the -norm into the constraint function and employed the successive linearization technique to solve the optimization function [22]. Then, GPCA was extended to 2DPCA with the -norm constraint (G2DPCA-) [23], and the local optimal solutions were obtained in the Minorization-Maximization (MM) Framework [24]. In addition, the -norm means the p power of 2-norm, and it is actually a single norm metric. The -norm-based methods were presented with the single objective of minimizing the reconstruction error [25], [26], [27]. Although these -norm, -norm and -norm metrics enhance the robustness, they utilize one single norm to measure and optimize the single objective with little consideration of the two objective optimizations of the maximum projection distance and the minimum reconstruction error. As these methods lose the advantage of the squared F-norm, they mostly fail to minimize reconstruction errors [28] and inevitably lose the invariance in the presence of rotation transformation [29]. In particular, the -norm and -norm methods extend the diversity of distance metrics with different p values, and the parameter p may bring additional uncertainty in algorithms.
The above-mentioned methods only employ a single norm to measure the distance of an optimization objective. There exist several robust 2DPCA methods that utilize one single norm metric and indirectly achieve both objectives by adjusting their objective functions. The F-norm-based method [30] and the -norm-based method [31] were utilized to minimize the ratio of the reconstruction error to the projection distance. To maximize the ratio of the projection distance to the input data, the robust optimal mean cosine angle 2DPCA (ROMCA-2DPCA) [32] with the -norm metric and the greedy cosine 2DPCA (Cos-2DPCA) [33] with weighted -norm maximization were presented, respectively. Under the F-norm metric, Area-2DPCA was proposed for minimizing the sum of areas between the projected vector and the reconstruction error [34]. The objective function of -2-DPCA actually uses the -norm metric, which considers the dual objectives of minimizing reconstruction errors and maximizing projection distances, while also ensuring the rotational invariance property [35]. Then, an efficient non-greedy PCA method was provided to solve the general -norm maximization [36]. All of these methods balance two optimization objectives to some extent. For their objective functions in the form of ratios, the distances of the dual objectives are measured by the same norm. In fact, it is difficult to achieve both optimal objectives at the same time. One reason is that both objectives are constrained by the same projection matrix. The other is that the maximum projection distance is not equivalent to the minimum reconstruction error under the F-norm or -norm metric. Thus, it is still possible to continue improving the robustness by using different norms to measure the distances of different objectives.
Obviously, it is requisite to introduce joint-norm metrics in the objective function and to integrate the respective advantages of specific norms. In the field of low-rank decomposition of PCA, the robust PCA via optimal mean was proposed with the objective function consisting of the -norm loss function and the Schatten -norm regularization [37]. In addition, the -norm and -norm constrained graph Laplacian PCA was presented for robust tumor sample clustering [38]. Similar joint-norm work can be seen in other subspace learning fields, such as LDA and neighborhood preserving projection (NPP) [39], [40], [41], [42], [43]. In the objective functions of these methods, different norms are used for measuring the distances of different optimization objectives. However, there is no joint-norm related studies in terms of PCA similar distance metric learning.
To flexibly utilize the respective advantages of different norm metrics, this paper attempts to join two norms in the objective function of 2DPCA to measure two optimization objectives, which have been never done in the field of robust 2DPCA metric learning. First, we investigate existing objective functions based on similar distance metrics and propose a generalized 2DPCA method joining the 2-norm and the -norm (2DPCA-2-Lp) for robust dimensionality reduction. The -norm is utilized to measure the distances of projected vectors, while the 2-norm is used for measuring row vectors of input samples. Therefore, the dual objectives of directly maximizing projection distances and indirectly minimizing reconstruction errors are achieved. In addition, the similarity in measuring distances between vectors, the property of rotation invariance and the ability to extend to more general cases are comprehensively considered. Concretely, the contributions are summarized as follows.
- •
The proposed 2DPCA-2-Lp introduces the joint-norm into robust 2DPCA dimensionality reduction for the first time and extends the single-norm distance metric to the joint-norm one.
- •
We develop a novel closed-form greedy algorithm to solve the joint-norm objective function, which can be applied to more different joint-norm cases.
- •
The relationship between reconstruction errors and projection distances is analyzed under different distance metrics and constraints. The analysis provides a theoretical basis for achieving the dual objective optimization of 2DPCA methods based on the joint-norm metric.
The remainder of this paper is organized as follows. Several related 2DPCA methods are briefly reviewed in Section 2. The motivation and solution of the proposed 2DPCA-2-Lp are elaborated in Section 3. Experimental results are reported and analysed in Section 4. Finally, Section 5 concludes this paper and future work.
Section snippets
Related work
Suppose that there exists a dataset with n samples, and the i-th sample matrix is represented by , where means the j-th row of . All these samples satisfy . Corresponding to the first k maximum eigenvalues in a descending order, the projection matrix consists of k dominant principal components (also called projection vectors), where k means the number of principal components (NPCs), and means the d-th projection vector of
Motivation and objective function
For the 2DPCA methods mentioned in Section 2, their objective functions employ different single norms to measure distances for robust dimensionality reduction. The conventional 2DPCA method uses the Euclidean distance as a metric (also called as squared F-norm metric), which can directly realize two objectives of minimizing reconstruction errors and maximizing projection distances. However, this metric is based on the squared operation of the projected data, which may greatly amplify the effect
Datasets
In this section, a series of experiments were conducted on the four datasets, including Georgia Tech (GT) face dataset, Columbia University Image Library (COIL-100) image dataset, CMU Pose, Illumination, and Expression (PIE) dataset, and NEC toy animal object recognition dataset. To evaluate the robustness of the proposed algorithm, the noise blocks with different occlusions were added to some samples of each dataset. These noise blocks have the same aspect ratio as the images in the same
Conclusions and future work
In this paper, we present a joint-norm-based 2DPCA method, called 2DPCA-2-Lp, for robust dimensionality reduction. It joins the 2-norm and the -norm as distance metrics in the objective function, thus balancing the two objectives of minimizing reconstruction errors and maximizing projection distances. Unlike most existing 2DPCA methods, the proposed joint-norm model deals with the row vectors of matrices, thus reducing the effect of noise and improving the robustness. In addition, the
CRediT authorship contribution statement
Huanxing Zhang: Conceptualization, Methodology, Data Curation, Writing - Original Draft. Hongxu Bi: Data Curation, Writing - Review & Editing. Xiaofeng Wang: Project administration, Supervision, Writing - Review & Editing. Peng Zhang: Software, Validation, Investigation.
Experiment platform
All the experiments are programmed using Visual Studio 2010 and OpenCV 2.4.10, and performed on the computer configured with: AMD Ryzen 7 4800H 2.90 GHz and RAM-16GB.
CRediT authorship contribution statement
Huanxing Zhang: Conceptualization, Data curation, Methodology, Writing – original draft. Hongxu Bi: Data curation, Writing – review & editing. Xiaofeng Wang: Project administration, Supervision, Writing – review & editing. Peng Zhang: Investigation, Software, Validation.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This work is supported by the National Key Research and Development Program of China (No. 2018AAA0103004) and the Tianjin Science and Technology Planned Key Project (No. 20YFZCGX00550).
References (44)
- et al.
Intrinsic dimension estimation: advances and open problems
Inf. Sci.
(2016) - et al.
A novel lifelong learning model based on cross domain knowledge extraction and transfer to classify underwater images
Inf. Sci.
(2021) - et al.
Knowledge extraction and retention based continual learning by using convolutional autoencoder-based learning classifier system
Inf. Sci.
(2022) - et al.
An overview of incremental feature extraction methods based on linear subspaces
Knowl.-Based Syst.
(2018) - et al.
Independent component analysis: algorithms and applications
Neural Netw.
(2000) - et al.
Dual robust regression for pattern classification
Inf. Sci.
(2021) - et al.
Distance metric learning for ordinal classification based on triplet constraints
Knowl.-Based Syst.
(2018) - et al.
Robust dimensionality reduction via feature space to feature space distance metric learning
Neural Netw.
(2019) - et al.
Improve robustness of sparse PCA by L-1-norm maximization
Pattern Recognit.
(2012) - et al.
2DPCA with L1-norm for simultaneously robust and sparse modelling
Neural Netw.
(2013)