Sparse probability density function estimation using the minimum integrated square error

Abstract

We develop a new sparse kernel density estimator using a forward constrained regression framework, within which the nonnegative and summing-to-unity constraints of the mixing weights can easily be satisfied. Our main contribution is to derive a recursive algorithm to select significant kernels one at time based on the minimum integrated square error (MISE) criterion for both the selection of kernels and the estimation of mixing weights. The proposed approach is simple to implement and the associated computational cost is very low. Specifically, the complexity of our algorithm is in the order of the number of training data N, which is much lower than the order of N² offered by the best existing sparse kernel density estimators. Numerical examples are employed to demonstrate that the proposed approach is effective in constructing sparse kernel density estimators with comparable accuracy to those of the classical Parzen window estimate and other existing sparse kernel density estimators.

Introduction

The finite mixture model [1] is a general approach to the probability density function (PDF) estimation problem that is fundamental to many pattern recognition, data analysis and other engineering applications [2], [3], [4], [5], [6], [7]. The celebrated Parzen window (PW) estimate [8] can be regarded as a special case of the finite mixture model, in which the number of mixtures is equal to that of the training data samples and all the mixing weights are equal. However, the point density estimate using the PW estimator for a future data sample can be computationally expensive if the number of training data samples is very large. Much of the existing works in the fitting of a finite mixture model are based on fixing the number of mixtures and applying the expectation-maximisation (EM) algorithm [9] to provide the maximum likelihood (ML) estimate of the mixture model's parameters. This associated ML optimisation, in general, is a highly nonlinear optimisation process requiring extensive computation, but for the Gaussian mixture model, the EM algorithm can be derived in an explicit iterative form [10]. However, this EM algorithm based ML estimation is well known to be ill posed and has a slow convergence speed, and to tackle the associated numerical difficulties, it is often required to apply resampling techniques [11], [12], [13], [14]. In general, the correct number of mixture components is unknown, and simultaneously determining the required number of mixture components and estimating the associated parameters of the finite mixture model is a challenging problem. Hence it is highly desirable to develop new methods of fitting a finite mixture model with the capability to infer a minimum number of mixtures from the data automatically and efficiently.

There is a considerable interest into research on the sparse PDF estimation. The support vector machine (SVM) density estimation technique has been proposed [15], [16], in which the density estimation problem is formulated as a supervised learning mode whilst the mean absolute deviation between the empirical cumulative distribution function (CDF) calculated from the training data and the CDF based on the PDF estimator also calculated from the training data are minimised. The optimisation in the SVM method is to solve a constrained quadratic optimisation problem. This yields the sparsity inducing property, i.e. at the optimality, many kernels' weights are driven to zeros. Alternatively a novel regression-based PDF estimation method has been introduced [17], in which the empirical CDF is constructed, in the same manner as in the SVM density estimation approach, to be used as the desired response. The orthogonal forward regression (OFR) approach is an efficient supervised regression model construction method [18]. In order to automatically determine the model structure with the improved model generalisation, the OFR method has been combined with a leave-one-out test score and local regularisation [19], [20]. The regression-based idea of [17] and the approach in [19], [20] have been extended to yield a new OFR based sparse density estimation algorithm [21] which is capable of automatically constructing very sparse kernel density estimate, with comparable performance to that of the PW estimate. In [17], [21], the regressors are the CDFs of the kernels and the target response is the empirical CDF. The calculation of CDFs becomes inconvenient and difficult for many types of kernels whose corresponding CDFs are difficult to compute. A simple and viable alternative approach has been proposed to use kernels directly as regressors by adopting the PW estimate as the target response [22].

The desirable property of sparsity inducing also happens in the interesting approach of reduced set density estimator (RSDE) [23]. The RSDE is different from the SVM in that it is based on the minimisation of the integrated square error (ISE) between the estimator and the true density. The minimum integrated square error (MISE) is a classical goodness of fit criterion for probability density estimation [2], [23], [24]. The optimisation problem in RSDE is a constrained quadratic optimisation one, and two efficient optimisation algorithms of both multiplicative updating of the weighting coefficients and sequential minimisation optimisation were introduced for the RSDE that has a complexity of $\mathcal{O}(N^2)$ per iteration, where N is the number of data samples and $\mathcal{O}(M)$ denotes the order of M, compared to a standard quadratic optimisation solver at $\mathcal{O}(N^3)$. Note that the RSDE is mainly restricted to using the Gaussian kernel, but the sparse density estimators of [21], [22] do not have this restriction. The complexity of the sparse density estimators [21], [22] is also $\mathcal{O}(N^2)$ scaled by the number of regressors selected, which is generally very small. Our extensive experience has shown that all the sparse density estimators [15], [16], [21], [22], [23] discussed here are capable of automatically producing sparse PDF estimates with comparable performance to that of the PW estimate, but the density estimators of [21], [22], [23] produce much sparser estimates than the SVM based density estimator.

Against this background, this paper introduces a new algorithm for sparse kernel density estimation based on the MISE and the forward constrained regression (FCR) [25]. In our proposed new sparse kernel density estimator, referred to as the FCR-MISE algorithm, a kernel term is selected one at a time which has the minimum ISE value among all the candidate kernels formed from the data points. Within the FCR framework, the mixing weights are computed using a recursion linking the weight for the newly selected kernel and the set of the mixing weights of the previous stages [25]. Thus the parameter estimation problem is reduced to a one-dimensional one, which is shown to have a closed-form solution using the MISE criterion. The proposed density estimation algorithm is very efficient due to the recursive computation and the closed-form solution of only one parameter per step. Specifically, the complexity of our proposed new algorithm is $\mathcal{O}(N)$ scaled by the squared number of kernels selected. Numerical examples are employed to demonstrate that our new sparse kernel density estimator is capable of producing very sparse PDF estimates with comparable accuracy to those of the PW estimator and other existing sparse kernel density estimators.

The paper is organised as follows. Section 2 introduces the idea of sparse kernel density estimator construction via the FCR framework. Section 3 proposes the new algorithm of joint kernel selection and mixing weight estimation based on the MISE and FCR. Numerical experiments are utilised to illustrate the effectiveness of the proposed algorithm in Section 4 and our conclusions are given in Section 5.