Multiplicative updates for non-negative projections

Abstract

We present here how to construct multiplicative update rules for non-negative projections based on Oja's iterative learning rule. Our method integrates the multiplicative normalization factor into the original additive update rule as an additional term which generally has a roughly opposite direction. As a consequence, the modified additive learning rule can easily be converted to its multiplicative version, which maintains the non-negativity after each iteration. The derivation of our approach provides a sound interpretation of learning non-negative projection matrices based on iterative multiplicative updates—a kind of Hebbian learning with normalization. A convergence analysis is scratched by interpretating the multiplicative updates as a special case of natural gradient learning. We also demonstrate two application examples of the proposed technique, a non-negative variant of the linear Hebbian networks and a non-negative Fisher discriminant analysis, including its kernel extension. The resulting example algorithms demonstrate interesting properties for data analysis tasks in experiments performed on facial images.

Introduction

Projecting high-dimensional input data into a lower-dimensional subspace is a fundamental research topic in signal processing and pattern recognition. Non-negative projection is desired in many real-world applications, for example, for images, spectra, etc., where the original data are non-negative. However, most classical subspace approaches such as principal component analysis (PCA) and Fisher discriminant analysis (FDA), which are solved by singular value decomposition (SVD), fail to produce the non-negativity property.

Recently, Lee and Seung [12], [13] introduced iterative multiplicative updates, which are based on the decomposition of the gradient of given objective function, for non-negative optimizations. They applied the technique to the non-negative matrix factorization (NMF) which seems to yield sparse representations. Several variants of NMF such as [5], [22], [24] have later been proposed, where the original NMF objective function is combined with various regularization terms. More recently, Yuan and Oja [23] presented a method called projective non-negative matrix factorization (P-NMF) without any additional terms, but directly derived from the objective function of PCA networks except that the projection was constrained to be non-negative. The simulation results of P-NMF indicate that it can learn highly localized and non-overlapped part-based basis vectors. However, none of the above works provides an explanation why the multiplicative updates can produce sparser and more localized base components.

The multiplicative update rules of the above algorithms are based on decomposition of the gradients of an objective function into positive and negative parts, one as the numerator and the other as the denominator. Nevertheless, such method would fail when the gradient is not naturally expressed in positive and negative parts. Sha et al. [21] proposed an alternative decomposition of the gradient and applied it to the minimization of a quadratic objective. This method albeit cannot handle the situation where the gradient contains only one positive (negative) term. Furthermore, how to combine orthogonality or quadratic unit norm constraints with this method is still unknown.

In this paper we present a more general technique to reformulate a variety of existing additive learning algorithms to their multiplicative versions in order to produce non-negative projections. The derivation is based on Oja's rule [17] which integrates the normalization factor into the additive update rule. Therefore, our method provides a natural way to form the numerator and denominator in the multiplicative update rule even if external knowledge of gradient decomposition is not available. Another major contribution of our approach is that its derivation also provides a sound interpretation of the non-negative learning based on iterative multiplicative updates—a kind of Hebbian learning with normalization.

We demonstrate applicability of the proposed method for two classical learning algorithms, PCA and FDA, as examples. In the unsupervised PCA learning, our multiplicative implementation of linear Hebbian networks outperforms the NMF in localized feature extraction, and its derivation provides an interpretation why P-NMF can learn non-overlapped and localized basis vectors. In the supervised FDA learning, our non-negative variant of the linear discriminant analysis (LDA) can serve as a feature selector and its kernel extension can reveal an underlying factor in the data and be used as a sample selector. The resulting algorithms of the above examples are verified by experiments on facial image analysis with favorable results.

The remaining of the paper is organized as follows. First we introduce the basic idea of multiplicative update rules in Section 2. The non-negative projection problem is then formulated in Section 3. In Section 4 we review Oja's rule and present the technique how to use it in forming the multiplicative update rules. The proposed method is applied in two examples in Section 5: one for unsupervised learning and the other for supervised. The experimental results of the resulting algorithms are presented in Section 6. Finally, conclusions are drawn in Section 7.