Complex Non-rigid 3D Shape Recovery Using a Procrustean Normal Distribution Mixture Model

Abstract

Recovering the 3D shape of a non-rigid object is a challenging problem. Existing methods make the low-rank assumption and do not scale well with the increased degree of freedom found in complex non-rigid deformations or shape variations. Moreover, in general, the degree of freedom of deformation is assumed to be known in advance, which limits the applicability of non-rigid structure from motion algorithms in a practical situation. In this paper, we propose a method for handling complex shape variations based on the assumption that complex shape variations can be represented probabilistically by a mixture of primitive shape variations. The proposed model is a generative probabilistic model, called a Procrustean normal distribution mixture model, which can model complex shape variations without rank constraints. Experimental results show that the proposed method significantly outperforms existing methods.

Appendix: Calculation of \(p({\mathbf {X}}_i|{\mathbf {D}}_i, {\varvec{\Phi }}_i)\) in (16)

This appendix section describes the effect of ignoring the Dirac-delta term in (16).

1.1 Without the Dirac-Delta Term in (16)

We omit subscripts i, k, and ik if no confusion arises. Using Bayes’ rule, the posterior distribution of ${\mathbf {X}}$ can be written as

$$\begin{aligned}&p({\mathbf {X}}|{\mathbf {D}}, {\varvec{\Phi }}) \propto p({\mathbf {D}}| {\mathbf {X}}, \sigma )p({\mathbf {X}}|{\varvec{\Phi }}) \nonumber \\&\propto \exp \bigg ( -\frac{1}{2} {\mathbf {vec}}({\mathbf {X}})^T {\mathbf {H}}{\mathbf {vec}}({\mathbf {X}}) + \frac{1}{\sigma ^2} {\mathbf {vec}}({\mathbf {D}})^T {\mathbf {F}}^T {\mathbf {vec}}({\mathbf {X}}) \nonumber \\&\quad + s {\mathbf {vec}}(\overline{{\mathbf {X}}})^T {\mathbf {Q}}\Sigma _{R}^{-1} {\mathbf {Q}}^T \left( {\mathbf {I}}\otimes {\mathbf {R}}\right) {\mathbf {vec}}({\mathbf {X}}) \bigg ) \nonumber \\&\quad = \exp \left( -\frac{1}{2} {\mathbf {vec}}({\mathbf {X}})^T {\mathbf {H}}{\mathbf {vec}}({\mathbf {X}}) + \frac{1}{\sigma ^2} {\mathbf {vec}}({\mathbf {D}})^T {\mathbf {vec}}({\mathbf {X}})\right) , \nonumber \\ \end{aligned}$$

(34)

where ${\mathbf {H}}= s^2 ({\mathbf {I}}\otimes {\mathbf {R}}^T) {\varvec{\Sigma }}^+ ({\mathbf {I}}\otimes {\mathbf {R}}) + \frac{1}{\sigma ^2} {\mathbf {F}}$ and we use ${\mathbf {vec}}(\overline{{\mathbf {X}}})^T {\mathbf {Q}}= 0$, ${\mathbf {vec}}({\mathbf {D}}) = {\mathbf {F}}{\mathbf {vec}}({\mathbf {D}})$, and ${\mathbf {F}}^2 = {\mathbf {F}}$.

We can also write $p({\mathbf {X}}|{\mathbf {D}}, {\varvec{\Phi }})$ as:

$$\begin{aligned} \begin{aligned}&p({\mathbf {vec}}({\mathbf {X}})|{\mathbf {D}}, {\varvec{\Phi }}) = {\mathcal {N}}\left( {\mathbf {m}}, {\varvec{\Omega }}\right) \\&\quad \propto \exp \left( -\frac{1}{2} \left( {\mathbf {vec}}({\mathbf {X}}) - {\mathbf {m}}\right) ^T {\varvec{\Omega }}^{-1} \left( {\mathbf {vec}}({\mathbf {X}}) - {\mathbf {m}}\right) \right) \\&\quad \propto \exp \left( -\frac{1}{2} {\mathbf {vec}}({\mathbf {X}})^T {\varvec{\Omega }}^{-1} {\mathbf {vec}}({\mathbf {X}}) + {\mathbf {m}}^T {\varvec{\Omega }}^{-1} {\mathbf {vec}}({\mathbf {X}}) \right) . \end{aligned} \end{aligned}$$

(35)

Comparing (35) with (34), we have

$$\begin{aligned} \begin{aligned} {\varvec{\Omega }}^{-1}&= {\mathbf {H}}, \\ {\mathbf {m}}^T {\varvec{\Omega }}^{-1}&= \frac{1}{\sigma ^2} {\mathbf {vec}}({\mathbf {D}})^T, \end{aligned} \end{aligned}$$

(36)

Therefore, we can represent $p({\mathbf {X}}|{\mathbf {D}}, {\varvec{\Phi }})$ as the following Gaussian distribution:

$$\begin{aligned} \begin{aligned}&p({\mathbf {X}}|{\mathbf {D}}, {\varvec{\Phi }}) \\&\quad =p({\mathbf {vec}}({\mathbf {X}})|{\mathbf {D}}, {\varvec{\Phi }})\sim {\mathcal {N}}\left( {\mathbf {m}}, {\varvec{\Omega }}\right) , \\&\quad {\mathbf {m}}= \frac{1}{\sigma ^2} {\varvec{\Omega }}{\mathbf {vec}}({\mathbf {D}}) \\&\quad = \frac{1}{\sigma ^2} \left( s^2 ({\mathbf {I}}\otimes {\mathbf {R}}^T) {\varvec{\Sigma }}^+ ({\mathbf {I}}\otimes {\mathbf {R}}) + \frac{1}{\sigma ^2} {\mathbf {F}}\right) ^+ {\mathbf {vec}}({\mathbf {D}}), \\&\quad {\varvec{\Omega }}= {\left( s^2 ({\mathbf {I}}\otimes {\mathbf {R}}^T) {\varvec{\Sigma }}^+ ({\mathbf {I}}\otimes {\mathbf {R}}) + \frac{1}{\sigma ^2} {\mathbf {F}}\right) }^+. \end{aligned} \end{aligned}$$

(37)

1.2 With the Dirac-Delta Term in (16)

Let ${\mathbf {v}}$ be a random vector drawn from ${\mathcal {N}}({\mathbf {0}}, \Sigma _R)$. Then, ${\mathbf {vec}}({\mathbf {X}})$ can be represented as follows:

$$\begin{aligned} {\mathbf {vec}}({\mathbf {X}}) = \frac{1}{s} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) \left( {\mathbf {Q}}{\mathbf {v}}+{\mathbf {vec}}(\overline{{\mathbf {X}}})\right) . \end{aligned}$$

(38)

By substituting (38) to ${\mathbf {vec}}({\mathbf {X}})$ of (35) in Section 1 and rearranging them with respect to ${\mathbf {v}}$, it can be written as

$$\begin{aligned} \begin{aligned}&p({\mathbf {v}}|{\mathbf {D}},{\varvec{\Phi }}) \\&\quad \propto \exp \bigg (-\frac{1}{2} {\mathbf {v}}^T \big ( {\varvec{\Sigma }}^{-1}_R\frac{1}{s^2\sigma ^2} {\mathbf {Q}}^T \left( {\mathbf {I}}\otimes {\mathbf {R}}\right) {\mathbf {F}}\left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) \big ){\mathbf {v}}\\&\quad +\frac{1}{s \sigma ^2} \left( s {\mathbf {vec}}({\mathbf {D}})^T \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {Q}}- \overline{{\mathbf {X}}}^T \left( {\mathbf {I}}\otimes {\mathbf {R}}\right) {\mathbf {Q}}\right) {\mathbf {v}}\bigg ). \end{aligned} \end{aligned}$$

(39)

Since (39) has only a quadric term and a linear term of ${\mathbf {v}}$, and a constant term, it can be represented as

$$\begin{aligned} \begin{aligned}&p({\mathbf {v}}|{\mathbf {D}},{\varvec{\Phi }}) = {\mathcal {N}}({\widehat{{\mathbf {m}}}}, \widehat{{\varvec{\Omega }}}) \\&\quad \propto \exp \left( -\frac{1}{2} ({\mathbf {v}}- {\widehat{{\mathbf {m}}}})^T \widehat{{\varvec{\Omega }}}^{-1} ({\mathbf {v}}- {\widehat{{\mathbf {m}}}}) \right) \\&\quad \propto \exp \left( -\frac{1}{2}{\mathbf {v}}^T \widehat{{\varvec{\Omega }}}^{-1} {\mathbf {v}}+ {\widehat{{\mathbf {m}}}}^T \widehat{{\varvec{\Omega }}}^{-1} {\mathbf {v}}\right) . \end{aligned} \end{aligned}$$

(40)

Comparing (39) with (40), we can write

$$\begin{aligned} \begin{aligned}&p({\mathbf {v}}|{\mathbf {D}},{\varvec{\Phi }}) = {\mathcal {N}}({\widehat{{\mathbf {m}}}}, \widehat{{\varvec{\Omega }}}), \\&{\widehat{{\mathbf {m}}}}= \frac{1}{s \sigma ^2} \widehat{{\varvec{\Omega }}} {\mathbf {Q}}^T \big ( {\mathbf {vec}}({\mathbf {D}}) - \frac{1}{s} {\mathbf {F}}\left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {vec}}(\overline{{\mathbf {X}}}) \big ), \\&\widehat{{\varvec{\Omega }}} = s^2 {\left( s^2 {\varvec{\Sigma }}^{-1}_R+ {\mathbf {Q}}^T \left( {\mathbf {I}}\otimes {\mathbf {R}}\right) \frac{{\mathbf {F}}}{\sigma ^2} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {Q}}\right) }^+, \end{aligned} \end{aligned}$$

(41)

where $\big ( {\mathbf {vec}}({\mathbf {D}}) - \frac{1}{s} {\mathbf {F}}\left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {vec}}(\overline{{\mathbf {X}}}) \big )$ corresponds to non-rigid variations and we can see only non-rigid variations affect ${\widehat{{\mathbf {m}}}}$ since ${\mathbf {Q}}$ is orthogonal to rigid variations by the definition of PND.

${\mathbf {X}}$ can be consider a linear transformed and translated version of ${\mathbf {v}}$ as shown in (38). By a linear property of a Gaussian distribution, we can also represent $p({\mathbf {X}}|{\mathbf {D}}, {\varvec{\Phi }})$ as a Gaussian distribution as

$$\begin{aligned} p({\mathbf {X}}|{\mathbf {D}}, {\varvec{\Phi }})= & {} p({\mathbf {vec}}({\mathbf {X}})|{\mathbf {D}}, {\varvec{\Phi }}) \sim {\mathcal {N}}({\mathbf {m}}, {\varvec{\Omega }}), \quad \text {where}\nonumber \\ {\mathbf {m}}= & {} {\mathbf {vec}}({E }[{\mathbf {X}}]) = \frac{1}{s} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) ({\mathbf {Q}}{\widehat{{\mathbf {m}}}}+ {\mathbf {vec}}(\overline{{\mathbf {X}}})) \nonumber \\= & {} \frac{1}{s^2 \sigma ^2} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {Q}}\widehat{{\varvec{\Omega }}} {\mathbf {Q}}^T \left( {\mathbf {I}}\otimes {\mathbf {R}}\right) \nonumber \\&\times \left( {\mathbf {vec}}({\mathbf {D}}) - \frac{1}{s} {\mathbf {F}}\left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {vec}}(\overline{{\mathbf {X}}}) \right) \nonumber \\&+ \frac{1}{s} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {vec}}(\overline{{\mathbf {X}}}), \nonumber \\ {\varvec{\Omega }}= & {} {E }[{\mathbf {vec}}({\mathbf {X}}- {\mathbf {m}}){\mathbf {vec}}({\mathbf {X}}- {\mathbf {m}})^T] \nonumber \\= & {} \frac{1}{s^2} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {Q}}\widehat{{\varvec{\Omega }}} {\mathbf {Q}}^T \left( {\mathbf {I}}\otimes {\mathbf {R}}\right) \nonumber \\= & {} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {Q}}\bigg (s^2 {\varvec{\Sigma }}^{-1}_R+ {\mathbf {Q}}^T \left( {\mathbf {I}}\otimes {\mathbf {R}}\right) \nonumber \\&\times \frac{{\mathbf {F}}}{\sigma ^2} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {Q}}\bigg ) {\mathbf {Q}}^T \left( {\mathbf {I}}\otimes {\mathbf {R}}\right) , \end{aligned}$$

(42)

where $({\mathbf {vec}}({\mathbf {D}}) - \frac{1}{s} {\mathbf {F}}\left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {vec}}(\overline{{\mathbf {X}}}))$ corresponds to non-rigid variations and $(\frac{1}{s} \left( {\mathbf {I}}\otimes {\mathbf {R}}^T\right) {\mathbf {vec}}(\overline{{\mathbf {X}}}))$ corresponds to rigid ones.

Since ${\mathbf {Q}}$ is a orthogonal to a subspace, ${\mathbf {Q}}_N(\overline{{\mathbf {X}}})$, on rigid motions, it means that we only consider an aligned prior mean shape $\frac{1}{s}{\mathbf {R}}^T \overline{{\mathbf {X}}}$ and non-rigid variations orthogonal to $\overline{{\mathbf {X}}}$ and other rigid variations are removed by ${\mathbf {Q}}$. However, we empirically found ignoring the Dirac-delta term makes the distribution $p({\mathbf {X}}|{\mathbf {D}}, {\varvec{\Phi }})$ close to the observation ${\mathbf {D}}$ and gives better reconstruction results, since s and ${\mathbf {R}}$ have inexact values in the early stage of the iteration process.