Semantics characterization for eye shapes based on directional triangle-area curve clustering

Abstract

In this paper, we present a novel approach to address the problem of eye shape characterization via curve representation. Firstly, a directional triangle-area curve representation method (DTAR) is presented for this aim. Equipped with DTAR, the shape similarities between two eyes can be measured by the similarities between two corresponding DTAR curves. Secondly, in order to exploit the underlying information of eye shapes, a curve clustering algorithm is utilized to automatically discover a set of eye shape prototypes. Consequently, a semantics extraction method for eye shapes is proposed in terms of seven reference eye shapes. Finally, in order to validate the consistency of the clustering results and the extracted semantics, extensive experiments on AR and BU4DFE databases are designed and conducted, and all the results demonstrate the effectiveness of the proposed DTAR curve representation and the semantics extraction method.

Appendices

Appendix

A. Clustering by density estimation

Assume that the d-dimensional vector y_i is modeled by the mixture density

$$ p(\mathrm{\mathbf{y}}_{i}| {\Theta})={\sum\limits^{K}_{k}}\alpha_{k}p_{k}(\mathrm{\mathbf{y}}_{i}| \theta_{k}), $$

(10)

in which α_k is the k-th mixture weight and p_k is the k-th component density with parameter vector 𝜃_k. The mixture weights α_k sum to one and are nonnegative.

The component densities p_k(⋅) model individual specific sub-regions of density, while the mixture p(⋅) summarizes all of these sub-regions according to the mixture weights α_k.

The likelihood of a data set Y = y₁,…,y_n is any function of Θ that is proportional to the probability of the data p(Y |Θ). The log-likelihood is the log of the likelihood and takes the form

$$ \mathcal{L}({\Theta} | Y)=\log{p(Y |{\Theta})}=\sum\limits_{i}\log{p(\mathrm{\mathbf{y}}_{i} | {\Theta})}=\sum\limits_{i} \log {\sum\limits_{k} \alpha_{k}p_{k}(\mathrm{\mathbf{y}}_{i}| \theta_{k})} $$

(11)

assuming the y_i are i.i.d. (independently and identically distributed). ML estimates of the parameter vector Θ correspond to values of Θ that maximize (11).

EM is an approximate root-finding procedure that is used to seek the root of the likelihood equation–it iteratively searches for a set of parameters $\hat {\Theta }$ that maximize the probability of the observed data. EM provides an efficient framework for parameter estimation in the mixture context.

A particular advantage of the probabilistic approach is that the component PDFs can be defined on non-vector data. For example, suppose that y_i is now a sequence of curve measurements of length n_i, observed at the n_i time points in x_i. We can define a cluster-specific conditional probabilistic model p_k(y_i|x_i,𝜃_k) that relates y_i to x_i.

In fact, the overall density of y_i (now given x_i) is a mixture of the component PDFs. In other words, the density takes the form

$$ p(\mathrm{\mathbf{y}}_{i}|\mathrm{\mathbf{x}}_{i},{\Theta})={\sum\limits^{K}_{k}} \alpha_{k}p_{k}(\mathrm{\mathbf{y}}_{i}| \mathrm{\mathbf{x}}_{i},\theta_{k}). $$

(12)

This conditional mixture density is now defined on curves in this paper. This curve density can be used in place of the mixture density in (10) to establish a model-based clustering procedure for curves. The generative model for the curve mixture is as follows:

• Assign the i-th curve to cluster k with probability α_k,

• Generate the mean curve for cluster k according to the component density model p_k,

• Define the i-th curve y_i to be equal to the mean curve plus some randomly generated noise (e.g., add a Gaussian error term).

B. Polynomial regression mixtures

In the following, a curve clustering methodology based on polynomial regression mix- ture models (PRM) is described. PRMs employ polynomial regression models with Gaussian error terms as the component PDFs. The inclusion of these regression models into the model-based curve clustering framework outlined above leads to an efficient EM learning algorithm for curve clustering.

(a) Model definition

Suppose we have a set Y of n curves as y₁,…,y_i,…,y_n. Each curve has (a possibly unique) length of ni with measurements observed at the points (or times) in x_i.A p-th order polynomial regression relationship between y_i and x_i is assumed with an additive Gaussian error term (a common assumption in the presence of multiple exogenous, unexplained effects).

The regression of y_i on x_i can be summarized with the following equation:

$$ \mathrm{\mathbf{y}}_{i}=\mathrm{\mathbf{X}}_{i}\beta+\epsilon_{i}, \epsilon_{i}\sim \mathcal{N}(0,\sigma^{2}\textit{\textit{I}}), $$

(13)

where the n_i × p regression matrix X_i is the Vandermonde matrix evaluated at x_i, and β is the p-vector of regression coefficients. The p-th order Vandermonde matrix evaluated at x_i is equal to

$$\mathrm{\mathbf{X}}_{i}=\left( \begin{array}{ccccc} 1 & x_{i1} & x^{2}_{i1} & {\cdots} & x^{p}_{i1} \\ {\vdots} & {\vdots} & {\vdots} & {\cdots} & {\vdots} \\ 1 & x_{in_{i}} & x^{2}_{in_{i}} & {\cdots} & x^{p}_{in_{i}}\\ \end{array} \right). $$

This regression equation, along with the error model, defines the conditional PDF of y_i given x_i as $\mathcal {N}(\mathrm {\mathbf {y}}_{i}|\mathrm {\mathbf {X}}_{i}\beta ,\sigma ^{2} \textit {\textit {I}})$. This PDF represents a probabilistic curve model that naturally allows for curves of variable length with unique measurement intervals and missing observations. Furthermore, the polynomial fit also takes advantage of smoothness information existed in the data.

This PDF can be incorporated into a mixture density by adding dependence of this PDF on k. In notation, this dependence is added in the form of subscripts on the parameters as $\{\beta _{k},{\sigma ^{2}_{k}}\}$. The incorporation of these cluster-dependent PDFs into the conditional mixture density in (12) results in the definition of the PRMs as

$$ p(\mathrm{\mathbf{y}}_{i}|\mathrm{\mathbf{x}}_{i},{\Theta})= {\sum\limits^{K}_{k}} \alpha_{k} p_{k}(\mathrm{\mathbf{y}}_{i}|\mathrm{\mathbf{x}}_{i},\theta_{k})={\sum\limits^{K}_{k}}\alpha_{k}\mathcal{N}(\mathrm{\mathbf{y}}_{i}|\mathrm{\mathbf{X}}_{i}\beta,{\sigma^{2}_{k}} \textit{\textit{I}}). $$

(14)

The log-likelihood takes the sum over all n curves of this conditional density with the following form

$$ \log p(Y|X,{\Theta})=\sum\limits_{i} \log {\sum\limits^{K}_{k}}\alpha_{k} p_{k}(\mathrm{\mathbf{y}}_{i}|\mathrm{\mathbf{x}}_{i},\theta_{k}) $$

(15)

This function is used to calculate the out-of-sample test log-likelihood scores by substituting in an unseen dataset Y^′ for Y. This model can now be used to derive the EM learning algorithm for curve clustering with PRMs.

(b) EM algorithm for PRMs

For the sake of notational simplicity, we also assume that every PDF is implicitly conditioned on a set of parameters (e.g., Θ or 𝜃_k), and thus we leave out the explicit dependence on the parameter vector in our notation.

We begin by letting z_i give the cluster membership for curve i, and we write the joint density of y_i and z_i as

$$ p(\mathrm{\mathbf{y}}_{i},z_{i}| \mathrm{\mathbf{x}}_{i})=\alpha_{z_{i}}p_{z_{i}}(\mathrm{\mathbf{y}}_{i}| \mathrm{\mathbf{x}}_{i})=\alpha_{z_{i}}\mathcal{N}(\mathrm{\mathbf{y}}_{i} | \mathrm{\mathbf{X}}_{i}\beta_{z_{i}},\sigma^{2}_{z_{i}}\textit{\textit{I}}). $$

(16)

The cluster memberships {z_i} are regarded as being hidden. The hidden-data density then becomes the posterior p(z_i|y_i,x_i). The complete-data log-likelihood function $\mathcal {L}_{c}$ can be calculated by taking the sum over all n curves of the log joint density in (16):

$$ \mathcal{L}_{c}=\sum\limits_{i}\log\alpha_{z_{i}}\mathcal{N}(\mathrm{\mathbf{y}}_{i} | \mathrm{\mathbf{X}}_{i}\beta_{z_{i}},\sigma^{2}_{z_{i}}\textit{\textit{I}}). $$

(17)

E-step

In the E-step, we calculate the posterior p(z_i|y_i,x_i) which gives the membership probability that the i-th curve was generated from cluster z_i.The membership probability takes the form

$$ w_{ik}=p(z_{i}=k | \mathrm{\mathbf{y}}_{i}, \mathrm{\mathbf{x}}_{i}) $$

(18)

The posterior expectation of $\mathcal {L}_{c}$ in (17) is then taken with respect to the posterior above to get the Q-function as follows:

$$ Q=\textit{\textit{E}}[\mathcal{L}_{c} |\mathrm{\mathbf{y}}_{i},\mathrm{\mathbf{x}}_{i} ]=\sum\limits_{i}\sum\limits_{k}w_{ik}\log\alpha_{k}\mathcal{N}(\mathrm{\mathbf{y}}_{i} | \mathrm{\mathbf{X}}_{i}\beta_{k},{\sigma^{2}_{k}}\textit{\textit{I}}). $$

(19)

M-step

In the M-step, we maximize Q with respect to the parameters $\{\beta _{k},{\sigma ^{2}_{k}},\alpha _{k}\}$. The solutions are straightforward and are given as

$$ \hat{\beta}_{k}=\left[\sum\limits_{i}w_{ik}\mathrm{\mathbf{X}}^{\prime}_{i}\mathrm{\mathbf{X}}_{i}\right]^{-1}\sum\limits_{i}w_{ik}\mathrm{\mathbf{X}}^{\prime}_{i}\mathrm{\mathbf{y}}_{i}, $$

(20)

$$ \hat{\sigma}^{2}_{k}=\frac{1}{{\sum}_{i}w_{ik}}\sum\limits_{i}w_{ik}\|\mathrm{\mathbf{y}}_{i}-\mathrm{\mathbf{X}}_{i}\beta_{k}\|^{2}, $$

(21)

and

$$ \hat{\alpha}_{k}=\frac{1}{n}\sum\limits_{i}w_{ik}. $$

(22)

Initialization is carried out by randomly sampling values for the membership probabilities and then beginning the iterations with the M-step. Convergence is detected when the ratio of the incremental improvement in log-likelihood to the initial incremental improvement during the second iteration drops below a threshold (e.g., 1 × E^− 6 in this paper).