Consistency and asymptotic normality of FastICA and bootstrap FastICA

Abstract

Independent component analysis (ICA) is possibly the most widespread approach to solve the blind source separation problem. Many different algorithms have been proposed, together with several highly successful applications. There is also an extensive body of work on the theoretical foundations and limits of the ICA methodology.

One practical concern about the use of ICA with real world data is the robustness of its estimates. Slight variations in the estimates, may stem from the inherent stochastic nature of the algorithms used or some deviations from the theoretical assumptions. To overcome this problem, different approaches have been proposed, most of which are based on the use of multiple runs of ICA algorithms with bootstrap.

Here we show the consistency and asymptotic normality of FastICA and bootstrap FastICA, based on empirical process theory, including Z-estimators and Hoeffding's inequality. These results give theoretical grounds for the robust use of FastICA, in a multiple run, bootstrap and randomly initialized manner. In this framework, it is also possible to assess the convergence of the algorithm through a normality test.

Introduction

In the recent years, blind source separation (BSS) has become a mainstream topic in signal and image processing, with independent component analysis (ICA) as possibly the most widespread approach to solve the aforementioned problem. The considerable amount of publications and dedicated conferences and workshops in these topics attest to their scientific spread. Furthermore, it is believed that the basic theoretical foundations of ICA, as well as various of its applications are rather well understood. A multitude of algorithms has been proposed and carefully analyzed (cf., [14], [6], [4]). Especially, for the FastICA algorithm (cf., [13], [15]), which is the main topic of this paper, theoretical limits have been presented earlier (cf., [16], [13], [19]).

One persistent concern for the practical use of ICA with real data is the robustness of the estimated sources. Repeated use of most ICA algorithms results in slight variations in the estimated components. This may have a variety of possible factors, including the possible inherently stochastic nature of the ICA implementation, or some departures from the ideal theoretical assumptions made [23]. To assess the robustness of the source estimation, several methods have been proposed, often based on a bootstrapped analysis of the estimated components [8], or using a bagging approach of FastICA (cf., [15]), a widely used algorithmic implementation of ICA. In [10], the multiple runs of FastICA are clustered by a self-organizing map, revealing common properties of the many estimated sources. In [23], using a different clustering strategy, the estimating variability itself is addressed, as a source of information on the robustness properties of the estimates.

We have shown experimentally that a multiple run of FastICA in a bagging manner [2] can lead to very interesting new insights into the analysis of independent components of functional magnetic resonance images (fMRI, [12]) (cf., [21], [10]). Nevertheless, in spite of such positive experimental outcomes, there has been no clear study of the theoretical validity of a bootstrapping approach to ICA algorithms.

In the present paper, we show the asymptotic normality of FastICA and bootstrapped FastICA, using the method of empirical process theory and Z-estimators in particular and their bootstrapped versions [20]. We first review the basic theoretical background behind FastICA, followed by a thorough theoretic study of the consistency and asymptotic normality of the algorithm. Experimental illustrations of these properties will be shown, using real fMRI data. The paper concludes with a brief discussion of the limitations and the implications of the current study. Note that, although we focus on the FastICA algorithm, we believe that several of the considerations made throughout the paper can be extended to other ICA methods. Yet, such extension is beyond the scope of the current paper.

In the following, we introduce the notation used throughout the paper. We assume ${\mathit{x}}_1,\dots ,{\mathit{x}}_n$ are independent and identically distributed by some joint distribution P, ${\mathit{x}}_i\in \mathcal{X}\subset {\mathbb{R}}^d$, satisfying the ICA model defined by $\mathit{x}=A\mathit{s}$. $\mathit{s}=(s_1,\dots ,s_d{)}^{\top }$ with s_i and s_j statistically independent for $i\ne j$, and $A\in {\mathbb{R}}^{d\times d}$ is full rank. Let us denote the covariance matrix of $\mathit{x}$ by ${\Sigma }_d=\mathrm{cov}\{\mathit{x}\}$. Then, we define the whitened random vector $\mathit{z}={\Sigma }_d^{-1/2}\mathit{x}$, which has identity covariance matrix. In general, the empirical estimate will be denoted by ${\widehat{\theta }}_n$ or ${\widehat{\mathit{w}}}_n$. Convergence in distribution is denoted by $⇝$, whereas convergence in probability is denoted by $\stackrel{P}{\to }$. The bootstrap sample and corresponding estimate carry superscript $⁎$ , e.g. ${\mathit{z}}_i^{⁎}$, ${\widehat{\theta }}_n^{⁎}$ for original samples ${\mathit{z}}_i$ and sample estimates ${\widehat{\theta }}_n$. Parameter with subscript “0”, ${\mathit{w}}_0$, denotes the true parameter. $\mathit{Ef}≔{\int }_{\mathcal{X}}f\mathrm{d}P$ and $E_{n}f≔(1/n){\sum }_{i=1}^{n}f(x_i)$, where $E_n=(1/n){\sum }_{i=1}^n{\delta }_{{\mathit{x}}_i}(·)$ is the empirical measure, which puts mass $1/n$ on samples. The r-norm is defined by $\parallel f{\parallel }_r≔(E|f{|}^r{)}^{1/r}$. L_r(P) is the space defined by the norm $\parallel ·{\parallel }_r$. The operator $\mathrm{diag}[\cdots ]$ returns a diagonal matrix with its argument on the diagonal elements. The Lipschitz norm is denoted by $|·{|}_L$. o_P stands for stochastic order and O is the numerical order. The operator norm is denoted by $\parallel ·{\parallel }_o$ and is defined by $\parallel A{\parallel }_o={\sup }_{\parallel t{\parallel }_2=1}\parallel \mathit{At}{\parallel }_2$.