A statistical analysis of the kernel-based MMSE estimator with application to image reconstruction

Highlights

• We develop a statistical analysis of the kernel-based minimum mean square error (KMMSE) estimator with emphasis in locally linear functions.

• Two error risk measures are proposed for this case.

• We show that KMMSE with a suitable bandwidth estimation procedure provides an approximation to sparse linear prediction with a bias correction term.

• For the case of recursive estimation (as it is commonly used in image reconstruction), the propagation error is also assessed through a vector Taylor series expansion.

• The MSE and propagation error measures previously proposed are applied to a novel filling ordering procedure.

Abstract

In this paper we carry out a statistical analysis of a multivariate minimum mean square error (MMSE) estimator developed from a nonparametric kernel-based probability density function. This kernel-based MMSE (KMMSE) estimation has been recently proposed by the authors and successfully applied to image and video reconstruction. The statistical analysis that we present here allows us to understand the utility and limitations of this estimator. Thus, we propose a couple of estimation error measures intended for locally linear signals and show how KMMSE can approximate a sparse estimator. Also, we study how the estimation error propagates when signals are reconstructed recursively, that is, when already-reconstructed samples are used for estimating new samples. As an application, we focus on the problem of the filling ordering (FO) associated to the reconstruction of heterogeneous image blocks. Thus, borrowing the concept of soft data from missing data theory, the error measures established in the first part of the paper can be transformed into reliability measures from which a novel FO procedure is developed. We show that the proposed FO outperforms other state-of-the-art procedures.

Introduction

Multimedia signal reconstruction is an active research area which plays an important role in a number of applications such as signal enhancement, inpainting or error concealment. The aim of signal reconstruction is to fill in a missing signal block with a set of samples so that a reasonable signal, according to a certain criterion and to the available samples surrounding the missing area, is obtained. There exist several approaches to signal reconstruction like interpolation, linear prediction or estimation [1], [2].

Recently, we have developed a novel reconstruction framework for image/video error concealment where the signal is reconstructed by sparse linear prediction [3], [4]. We also showed that the prediction coefficients can be approximated by exponential functions so that the resulting procedure can be interpreted as a multivariate non-parametric kernel-based reconstruction. In particular, the proposed method, called kernel-based minimum mean square error (KMMSE) estimation, can be viewed as a minimum mean square error (MMSE) estimation where the required probability density function (pdf) is obtained through kernel density estimation (KDE) [5], [6]. The use of non-parametric pdf estimates avoids the decision about a particular pdf model and naturally adapts the MMSE estimator to the local statistics. In addition, KMMSE can benefit from the formalism established for other well-known kernel-based estimation methods such as the Nadaraya-Watson (NW) or the local linear estimator [7]. The better performance yielded by KMMSE with respect a number of state-of-the-art image error concealment techniques is demonstrated in [6]. Moreover, it must be pointed out that KMMSE allows an efficient, scalable implementation as recently proposed in [8].

Multivariate kernel-based estimation suffers from the so-called curse of dimensionality [7]. This means that, in a multidimensional space, we can expect to frequently find regions where the available data is quite sparse, which leads to bad estimates. Thus, the NW estimator shows significant bias and covariance in these sparse regions. In principle, bias can be mitigated if a local linear estimator is used instead of NW (which is a local constant estimator) [7]. However, local linear estimation must be avoided in sparse regions of the signal space since it suffers from numerical instability as it is described in [9]. In a signal reconstruction application this is not a choice since an estimate must be provided for every missing sample. In this context, KMMSE becomes an attractive option since it is as numerically stable as NW but with a smaller bias. Furthermore, it can straightforwardly deal with vector estimates which means that, in certain applications (e.g., image reconstruction), it is able to further exploit signal correlations, outperforming other state-of-the-art reconstruction techniques [3], [6].

The first goal of this paper will be the use of the powerful mathematical framework associated to kernel-based estimators to carry out a statistical error analysis of the KMMSE estimator. We will pay special attention to the case of linear (or locally linear) signals, which play an important role in signal processing [10]. As a result of this error analysis, we will be able to derive a couple of error risk measures and to establish confidence intervals. Furthermore, the statistical analysis of the estimation error is the basis for the development of bandwidth estimation strategies. There is a wide literature on this topic but it is focused on regression. However, the goal of signal reconstruction is not to derive a certain function but to obtain a replacement for a specific missing sample. It is well known that signal reconstruction benefits from a sparsity criterion [3], [4], [11] more than from a minimum square error criterion, which may lead to signal oversmoothing or numerical instability. We will demonstrate that the bandwidth estimation procedure that we proposed for KMMSE in [6] provides an approximation to sparse estimation and it is therefore more suitable for reconstruction.

In the rest of the paper we focus on the application of the error analysis previously developed to the problem of reconstructing large blocks of images. While several techniques recover the whole missing block at once [12], [13], [14], it has been shown that a recursive recovery of smaller image blocks (patches, in the following), proceeding from outer to inner patches and reusing the already estimated samples, can yield better reconstruction quality [3], [15], [16], [17]. In order to apply this strategy, we must consider that a recursive recovery may propagate estimation errors from already estimated samples to the subsequent ones. We will evaluate this propagation error for the KMMSE estimator in Section 4.

As an application of the concepts introduced along this work, we will tackle the problem of spatial error concealment (SEC), that is, the reconstruction of image blocks lost during transmission, by means of a KMMSE recursive recovery. Recursive image reconstruction requires a filling ordering (FO) procedure in order to decide at every step which patch should be estimated next. Although a fixed ordering like a raster or concentric scan is possible, there are other choices for FO based on a more reasonable criteria. First, we can take into account that the larger the number of available surrounding pixels, the better the estimate will be [15], [16]. Since the reconstruction of a given patch will be based on these surrounding pixels, we should also take into account their reliability. For example, as we have already proposed in [3], we can distinguish between correctly received pixels and already estimated pixels through a fixed reliability factor, although this leads to a fixed ordering (for a given shape of the lost area). This fact is also considered in [16] through a confidence parameter. However, the procedure of this latter reference uses isophotes so that linear structures are prioritized, which may lead to considerable error propagation. This confidence factor is combined in [18] with another parameter based on a fractional derivative. This can provide strong edges which may be useful for inpainting, but not for SEC. In fact, it must be pointed out that most of the work developed on FO corresponds to image inpainting (where the resulting reconstruction must be subjectively evaluated), while it has been less studied for SEC, where our goal is to obtain a reconstructed image as similar as possible to the original one. In a recent work [19], and in the context of SEC, we have shown that, for those SEC methods that can yield some measure of the reconstruction error, an improved reconstruction can be obtained when the error associated to the already-reconstructed pixels is heuristically employed in the FO as a reliability measure. In this paper, we will show that the estimation and propagation error measures developed along the paper for KMMSE can be also used as reliability factors (obtained from the concept of soft data) to make an accurate FO decision.

This paper is organized as follows. Section 2 provides a brief review of the KMMSE estimator. Section 3 is devoted to presenting the statistical analysis of the KMMSE estimator, including an in-depth study for the special case of signals with a locally linear behavior. Error propagation in recursive estimation is dealt with in Section 4. The reliability-based filling ordering procedure for image reconstruction is proposed and tested in Section 5. Finally, Section 6 is devoted to conclusions.