Using conditional independence for parsimonious model-based Gaussian clustering

Abstract

In the framework of model-based cluster analysis, finite mixtures of Gaussian components represent an important class of statistical models widely employed for dealing with quantitative variables. Within this class, we propose novel models in which constraints on the component-specific variance matrices allow us to define Gaussian parsimonious clustering models. Specifically, the proposed models are obtained by assuming that the variables can be partitioned into groups resulting to be conditionally independent within components, thus producing component-specific variance matrices with a block diagonal structure. This approach allows us to extend the methods for model-based cluster analysis and to make them more flexible and versatile. In this paper, Gaussian mixture models are studied under the above mentioned assumption. Identifiability conditions are proved and the model parameters are estimated through the maximum likelihood method by using the Expectation-Maximization algorithm. The Bayesian information criterion is proposed for selecting the partition of the variables into conditionally independent groups. The consistency of the use of this criterion is proved under regularity conditions. In order to examine and compare models with different partitions of the set of variables a hierarchical algorithm is suggested. A wide class of parsimonious Gaussian models is also presented by parameterizing the component-variance matrices according to their spectral decomposition. The effectiveness and usefulness of the proposed methodology are illustrated with two examples based on real datasets.

Appendix A

1.1 A.1 Proof of Theorem 1

In order to prove Theorem 1 it is possible to exploit arguments similar to the ones used in Maugis et al. (2009b) to prove identifiability of Gaussian mixture models with irrelevant variables.

Given Eq. (3), both f(.|θ _M ) and $f(.|\boldsymbol{\theta}^{*}_{M^{*}})$ can be written as Gaussian mixture models. Namely:

$$ f(\mathbf{x}_i|\boldsymbol{\theta}_M) = \sum _{k=1}^K \pi_k \phi_{P} (\mathbf{\tilde{x}}_i|\boldsymbol{ \mu}_{k},\boldsymbol{\varSigma}_{k} ), $$

where $\mathbf{\tilde{x}}_{i}=(\mathbf{x}_{i}^{S_{1}\top}, \ldots, \mathbf{x}_{i}^{S_{g}\top}, \ldots, \mathbf{x}_{i}^{S_{G}\top})^{\top}$ is obtained by permuting vector x _i so that the P variables are listed according to the order established by (S ₁,…,S _G ), $\boldsymbol{\mu}_{k}=(\boldsymbol{\mu}_{k1}^{\top}, \ldots, \boldsymbol{\mu}_{kg}^{\top}, \ldots, \boldsymbol{\mu}_{kG}^{\top})^{\top}$, and $\boldsymbol{\varSigma}_{k}=\bigoplus_{g=1}^{G} \boldsymbol{\varSigma}_{kg}$. Analogously,

$$ f\bigl(\mathbf{x}_i|\boldsymbol{\theta}^*_{M^*} \bigr) = \sum_{k=1}^{K^*} \pi_k^* \phi_{P} \bigl(\mathbf{\tilde{x}}_i^*|\boldsymbol{ \mu}_{k}^*,\boldsymbol{\varSigma}_{k}^* \bigr). $$

Then, since the couples (μ _k ,Σ _k ) for k=1,…,K are distinct as well as $(\boldsymbol{\mu}_{k}^{*},\boldsymbol{\varSigma}_{k}^{*})$ for k=1,…,K ^∗, the identifiability of Gaussian mixture models gives that K=K ^∗ and, up to a permutation of mixture components and of the elements of x _i , $\pi_{k}=\pi_{k}^{*}$, $\boldsymbol{\mu}_{k}=\boldsymbol{\mu}_{k}^{*}$ and $\boldsymbol{\varSigma}_{k}=\boldsymbol{\varSigma}_{k}^{*}$ (see, for example, Yakowitz and Spragins 1968).

In order to complete the proof it is now proven by contradiction that, under the constraint (5), G=G ^∗ and each element of (S ₁,…,S _G ) coincides with one and only one element of $(S_{1}^{*}, \ldots, S_{G^{*}}^{*})$.

Consider S _g ∈(S ₁,…,S _G ). Since both (S ₁,…,S _G ) and $(S_{1}^{*}, \ldots, S_{G^{*}}^{*})$ are partitions of the variable index set $\mathcal{I}$, there exists at least one $S_{h}^{*} \in (S_{1}^{*}, \ldots, S_{G^{*}}^{*})$ such that $S_{g} \cap S_{h}^{*} \neq \emptyset$. Let $s=S_{g} \cap S_{h}^{*}$, $t=S_{g}^{c} \cap S_{h}^{*}$, and $\bar{s}=S_{g} \cap S_{h}^{*c}$.

Suppose that t≠∅. Since t∩S _g =∅, according to model M=(G,K,S ₁,…,S _G ) Σ _k,ts =0 _ts ∀k. Due to identifiability of Gaussian mixture models, this implies that $\boldsymbol{\varSigma}^{*}_{k,ts}=\boldsymbol{\varSigma}^{*}_{kh,ts}=\mathbf{0}_{ts}$ ∀k. However, this result contradicts the constraint (5). Thus, t=∅.

Analogously, suppose now that $\bar{s} \neq \emptyset$. Since $\bar{s} \cap S_{h}^{*} = \emptyset$, model $M^{*}=(G^{*}, K^{*}, S_{1}^{*}, \ldots, S_{G^{*}}^{*})$ implies that $\boldsymbol{\varSigma}^{*}_{k,s\bar{s}}=\mathbf{0}_{s\bar{s}}$ ∀k. Together with the identifiability of Gaussian mixture models, this also implies that $\boldsymbol{\varSigma}_{k,s\bar{s}}=\boldsymbol{\varSigma}_{kg,s\bar{s}}=\mathbf{0}_{s\bar{s}}$ ∀k. However, since this result contradicts the constraint (5), it follows from this that $\bar{s} = \emptyset$.

These two results imply that, under the constraint (5), $S_{g} \cap S_{h}^{*} \neq \emptyset$ ⇒ $S_{g} = S_{h}^{*}$. Thus, since S _g and $S_{h}^{*}$ belong to two partitions of the variable index set $\mathcal{I}$, there exists a one-to-one correspondence between the two partitions, and G=G ^∗.

1.2 A.2 Proof of Corollary 1

In order to prove Corollary 1 it is sufficient to note that under the constraint (5) the parameter space of models in $\mathcal{M}_{I}$ is a subset of $\varTheta_{(G, K, S_{1}, \ldots, S_{G})}$ and, hence, Theorem 1 holds also for this subclass of models.

1.3 A.3 Proof of Theorem 2

A proof of Theorem 2 can be obtained by exploiting some results from Maugis et al. (2007) and by suitably modifying the proof of the consistency of the BIC criterion in selecting relevant variables for clustering with Gaussian mixture models (Maugis et al. 2009b).

Since the true number of mixture components K ⁰ is assumed to be known, by adapting the notation previously introduced to this assumption let now denote M=(G,K ⁰,S ₁,…,S _G ), $M^{0} = (G^{0}, K^{0}, S^{0}_{1}, \ldots, S^{0}_{G^{0}})$, and

$$ \hat{M} = \bigl(\hat{G}, K^0, \hat{S}_1, \ldots, \hat{S}_{\hat{G}}\bigr) = \operatorname{argmax}\limits _{M \in \mathcal{M}'} \mathit{BIC}(M), $$

where $\mathcal{M}' \subset \mathcal{M}$ is the subclass of the models obtainable from Eq. (4) with K=K ⁰. Furthermore, consider Δ _BIC(M)=BIC(M ⁰)−BIC(M); after some straightforward algebra it is possible to write

$$ \Delta_{\mathit{BIC}(M)}=2n [\mathbb{D}_{nM^0}- \mathbb{D}_{nM} ]+\gamma_{M}\ln(n), $$

(8)

where $\gamma_{M}=\lambda_{M}-\lambda_{M^{0}}$, with λ _M and $\lambda_{M^{0}}$ denoting the number of free parameters of models M and M ⁰, respectively, and

$$\mathbb{D}_{nM}=\frac{1}{n} \sum_{i=1}^n \ln \biggl[\frac{f(\mathbf{x}_i | \hat{\boldsymbol{\theta}}_{M})}{h(\mathbf{x}_i)} \biggr], $$

$$\mathbb{D}_{nM^0}=\frac{1}{n} \sum_{i=1}^n \ln \biggl[\frac{f(\mathbf{x}_i | \hat{\boldsymbol{\theta}}_{M^0})}{h(\mathbf{x}_i)} \biggr]. $$

Since $P (\hat{M}=M^{0} ) = P (\Delta_{\mathit{BIC}(M)} \geq 0, \forall M \in \mathcal{M}' )$, in order to prove Theorem 2 we have to show that

$$ P (\Delta_{\mathit{BIC}(M)} < 0 ) \operatorname{ \longrightarrow}\limits _{n \rightarrow \infty} 0\quad \forall M \in \mathcal{M}'. $$

(9)

Note that, when M=M ⁰, we have $\Delta_{\mathit{BIC}(M^{0})}=0$, thus $P (\Delta_{\mathit{BIC}(M^{0})} < 0 )=0$ ∀n.

Let now consider M≠M ⁰ and, for making easier the reading of this proof, let $D_{M}=-KL[h,f(\cdot|\boldsymbol{\breve{\theta}}_{M})]$, and $T_{nM}= \mathbb{D}_{nM^{0}}-\mathbb{D}_{nM}+\frac{\gamma_{M}\ln(n)}{2n}$. Given Eq. (8), it is possible to write P(Δ _BIC(M)<0)= P(T _nM <0). This probability is also equal to

$$ P (T_{nM}- D_{M^0} + D_{M^0} - D_M + D_M< 0 ). $$

According to Lemma 5 in Maugis et al. (2007), the following inequality holds ∀ϵ>0:

According to Proposition 1 (see below), we also have $\mathbb{D}_{nM} \stackrel{P}{\rightarrow} D_{M}$ $\forall M \in \mathcal{M}'$. Thus, ∀ϵ>0

$$ P (\mathbb{D}_{nM}-D_M> \epsilon ) \leq P \bigl(| \mathbb{D}_{nM}-D_M|> \epsilon \bigr) \operatorname{ \longrightarrow}\limits _{n \rightarrow \infty} 0. $$

Furthermore, according to the assumption (H1), $D_{M^{0}}=0$, and −D _M >0 since M≠M ⁰. Then,

Taking $\epsilon = \frac{-D_{M}}{4}$, since $\frac{\gamma_{M}\ln(n)}{2n} \operatorname{\longrightarrow}\limits_{n \rightarrow \infty} 0$ we also obtain

These results imply (9), thus proving the theorem.

1.4 A.4 Proposition 1

Under assumptions (H1) and (H2) the following convergence holds $\forall M \in \mathcal{M}'$:

$$ \frac{1}{n} \sum_{i=1}^n \ln \biggl( \frac{h(\mathbf{x}_i)}{f(\mathbf{x}_i | \hat{\boldsymbol{\theta}}_{M})} \biggr) \stackrel{P}{\rightarrow} KL\bigl[h,f( \cdot|\boldsymbol{\breve{\theta}}_{M'})\bigr]. $$

(10)

Proof

According to (H2), $\varTheta'_{M}$ is a compact metric space, and ln[f(x|θ _M )] is a continuous function of θ _M ∀x∈ℝ^P. Furthermore, it is possible to show that there exists an envelope function $H \in \mathcal{H}_{M}=\{\ln[f(\cdot|\boldsymbol{\theta}_{M})]; \boldsymbol{\theta}_{M} \in \varTheta'_{M}\}$ which is h-integrable. This latter result can be proved as follows.

Since Σ _kg is positive definite, $\|\mathbf{x}^{S_{g}}-\boldsymbol{\mu}_{kg}\|^{2}_{\boldsymbol{\varSigma}_{kg}^{-1}}\geq 0$, where $\|\mathbf{x}^{S_{g}}-\boldsymbol{\mu}_{kg}\|^{2}_{\boldsymbol{\varSigma}_{kg}^{-1}}= (\mathbf{x}^{S_{g}}-\boldsymbol{\mu}_{kg} )^{\top}\boldsymbol{\varSigma}_{kg}^{-1} (\mathbf{x}^{S_{g}}-\boldsymbol{\mu}_{kg} )$. Furthermore, $|\boldsymbol{\varSigma}_{kg}|^{-\frac{1}{2}} \leq a^{-\frac{1}{2}}$ (see Maugis et al. 2007, Lemma 3). Writing f(x|θ _M )= $\sum_{k=1}^{K} \pi_{k} g(\mathbf{x}|\boldsymbol{\vartheta}_{k})$, where

$$ g(\mathbf{x}|\boldsymbol{\vartheta}_k)= \prod _{g=1}^G |2 \pi\boldsymbol{\varSigma}_{kg}|^{-\frac{1}{2}} \exp \biggl(-\frac{\|\mathbf{x}^{S_g}-\boldsymbol{\mu}_{kg}\|^2_{\boldsymbol{\varSigma}_{kg}^{-1}}}{2} \biggr), $$

with ϑ _k =(μ _k1,…,μ _kG ,Σ _k1,…,Σ _kG ), and recalling that $\sum_{k=1}^{K} \pi_{k} =1$, the following upper bound of ln[f(x|θ _M )] holds:

For making shorter the following equations, let $d^{2}_{kg}=\|\mathbf{x}^{S_{g}}-\boldsymbol{\mu}_{kg}\|^{2}_{\boldsymbol{\varSigma}_{kg}^{-1}}$. Using the concavity of the logarithm function we obtain:

Since $\boldsymbol{\mu}_{kg} \in \mathcal{B}(\eta,P_{g})$ and using Lemma 3 in Maugis et al. (2007) it is possible to write:

Furthermore, since $|\boldsymbol{\varSigma}_{kg}| \leq b^{P_{g}}$ (see Maugis et al. 2007, Lemma 3), the lower bound of ln[f(x|θ _M )] is given by:

Thus, since each function of the family $\mathcal{H}_{M}$ is bounded by

for all $\boldsymbol{\theta}_{M} \in \varTheta'_{M}$ and all x∈ℝ^P we have

$$ \bigl|\ln\bigl[f(\mathbf{x}|\boldsymbol{\theta}_M)\bigr] \bigr| \leq C_1(a,b,P,G,\eta)+C_2(a)\|\mathbf{x}\|^2, $$

defining the envelope function H, where C ₁(a,b,P,G,η) and C ₂(a) are two positive constants.

The h-integrability of this function can be proved by showing that ∫∥x∥² h(x)d x<∞:

where inequalities are obtained using Lemmas 3 and 4 in Maugis et al. (2007) and assumption (H2).

Hence, according to Proposition 2 in Maugis et al. (2007),

$$ \frac{1}{n} \sum_{i=1}^n \ln \bigl[f(\mathbf{x}_i | \hat{\boldsymbol{\theta}}_{M}) \bigr] \stackrel{P}{\rightarrow} \mathbb{E}_\mathbf{X}\bigl[\ln f( \mathbf{X}| \boldsymbol{\breve{\theta}}_{M})\bigr]. $$

(11)

Then, since $\ln(h) \in \mathcal{H}_{M^{0}}$, it implies that $\mathbb{E}_{\mathbf{X}}[|\ln h(\mathbf{X})|] \leq \mathbb{E}_{\mathbf{X}}[H(\mathbf{X})]<\infty$. Thus, according to the law of large numbers

$$ \frac{1}{n} \sum_{i=1}^n \ln \bigl[h(\mathbf{x}_i) \bigr] \stackrel{P}{\rightarrow} \mathbb{E}_\mathbf{X}\bigl[\ln h(\mathbf{X})\bigr]. $$

(12)

Convergences (11) and (12) imply (10), thus proving the proposition. □

1.5 A.5 Proposition 2

Let Σ ₁,…,Σ _g ,…,Σ _G be G real, symmetric and definite positive matrices, whose dimensions are P _g ×P _g for g=1,…,G, and let $\boldsymbol{\varSigma} = \bigoplus_{g=1}^{G} \boldsymbol{\varSigma}_{g}$. Furthermore, let $\boldsymbol{\varSigma}_{g}= \lambda_{g} \mathbf{D}_{g}\mathbf{A}_{g}\mathbf{D}_{g}^{\top}$, where $\lambda_{g}=|\boldsymbol{\varSigma}_{g}|^{1/P_{g}}$, D _g is the matrix of orthonormal eigenvectors of Σ _g , and A _g is the diagonal matrix containing the eigenvalues of Σ _g (normalized in such a way that |A _g |=1). Then, Σ=λ DAD ^⊤, where $\lambda= \prod_{g=1}^{G}\lambda_{g}^{\frac{P_{g}}{P}}$, $\mathbf{D}=\bigoplus_{g=1}^{G} \mathbf{D}_{g}$, and $\mathbf{A}= \bigoplus_{g=1}^{G} \frac{\lambda_{g}}{\prod_{g=1}^{G}\lambda_{g}^{\frac{P_{g}}{P}}}\mathbf{A}_{g}$.

Proof

Consider the spectral decomposition of Σ _g : $\boldsymbol{\varSigma}_{g}=\mathbf{D}_{g}\mathbf{L}_{g}\mathbf{D}_{g}^{\top}$, where L _g is the diagonal matrix containing the eigenvalues of Σ _g , for g=1,…,G. Then, L _g =λ _g A _g .

According to some properties of the direct sum operator, the following results hold:

• 1. $|\boldsymbol{\varSigma}|=\prod_{g=1}^{G}|\boldsymbol{\varSigma}_{g}|=\prod_{g=1}^{G}\lambda_{g}^{P_{g}}$ (see, for example, Lütkepohl 1996, p. 22);

• 2. Σ=DLD ^⊤, where $\mathbf{D}=\bigoplus_{g=1}^{G} \mathbf{D}_{g}$ and $\mathbf{L}=\bigoplus_{g=1}^{G} \mathbf{L}_{g}$ (see Lütkepohl 1996, p. 66).

Hence, $|\boldsymbol{\varSigma}|^{\frac{1}{P}}=\prod_{g=1}^{G}\lambda_{g}^{\frac{P_{g}}{P}}=\lambda$, $\frac{1}{|\boldsymbol{\varSigma}|^{\frac{1}{P}}}\mathbf{L}=$ $\frac{1}{|\boldsymbol{\varSigma}|^{\frac{1}{P}}}\bigoplus_{g=1}^{G} \mathbf{L}_{g}$ $=\bigoplus_{g=1}^{G} \frac{\lambda_{g}}{\prod_{g=1}^{G}\lambda_{g}^{\frac{P_{g}}{P}}}\mathbf{A}_{g}=\mathbf{A}$, thus proving the proposition. □