Camera Motion Correction with PGA

Abstract

In this paper we study the geometry and kinematics of stabilizing a moving camera in order to track a stationary scene both in the 2D and 3D setting. This is being done initially in a rather straightforward manner, using the tools of analytic and differential geometry, after which we discuss the advantages of the so-called ‘projective geometric algebra’ (PGA) approach in this context. In the planar case one can easily get equivalent results with complex numbers, but in 3D it is a convenient substitute for Plücker line geometry and the theory of screws. While a lot can be done using quaternions and differential geometry, PGA is quite handy when there are different rotation or screw axes involved. Its basic constructions and properties are briefly summarized in the appendix.

Appendix: PGA in a Nutshell

Here we provide a brief idea how projective geometric algebra (PGA) works in this setting. The classical Clifford’s geometric algebra is often introduced as a factorization of the tensor algebra or an extension to the Grassmann exterior algebra. However, it is easier to understand starting with a vector space $V\cong {\mathbb R}^n$ with additional structure, called geometric multiplication, which for vectors is just the sum of the familiar‘dot’ (interior) and‘wedge’ (exterior) products, i.e.,

$$\begin{aligned} {\boldsymbol{\xi }} {\boldsymbol{\eta }} = {\boldsymbol{\xi }} \cdot {\boldsymbol{\eta }} + {\boldsymbol{\xi }} \wedge {\boldsymbol{\eta }},\qquad {\boldsymbol{\xi }}, {\boldsymbol{\eta }}\in V. \end{aligned}$$

(21)

This yields a scalar and a bivector component, referred to as grade zero and grade two elements, respectively. Using grade projectors we may write $\langle {\boldsymbol{\xi }} {\boldsymbol{\eta }}\rangle _0 = {\boldsymbol{\xi }}\cdot {\boldsymbol{\eta }}$ and $\langle {\boldsymbol{\xi }} {\boldsymbol{\eta }}\rangle _2 = {\boldsymbol{\xi }}\wedge {\boldsymbol{\eta }}$. The maximal possible grade equals n and is one-dimensional, spanned by the so-called pseudo-scalar I. It yields the Hodge operator $\star$ switching from k to $n-k$ grades by sending each element to its orthogonal complement. This convenient duality is extensively used in all of geometry, and PGA in particular. Allowing hyperbolic or parabolic signatures we model different isometry groups, e.g. Lorenz transformations via the space-time algebra $\textrm{Cliff}({\mathbb R^{3,1}})$ or 3D Euclidean motions via PGA: $\textrm{Cliff}({\mathbb R^{3,0,1}})$. They are contained in the subalgebra of even grade elements, justified by the ${\mathbb Z}_2$-grading of $\textrm{Cliff}(V)$, and generated by bivectors. One simple example is the algebra of complex numbers $\mathbb {C}\cong \textrm{Cliff}_\circ ({\mathbb R^{2}})$

$$\begin{aligned} \textbf{e}_1^2 = \textbf{e}_2^2 = 1,\qquad \textbf{e}_1\cdot \textbf{e}_2 =0,\qquad \textbf{e}_1\wedge \textbf{e}_2 = \textbf{e}_1 \textbf{e}_2 = I \end{aligned}$$

spanned by mutually commuting scalars and bivectors, where we also have

$$\begin{aligned} I^2 = (\textbf{e}_1 \textbf{e}_2)^2 = -\textbf{e}_1^2 \textbf{e}_2^2 = -1\;\Rightarrow \;\textrm{Cliff}_\circ ({\mathbb R^{2}})\cong {\mathbb R}[i]\cong \mathbb {C}. \end{aligned}$$

Restriction to elements of unit norm yields the orthogonal group in the plane ${\mathrm O}(2)\cong {\mathrm U}(1)$. Similarly, the even part of $\textrm{Cliff}({\mathbb R^{3}})$ gives rise to the quaternion algebra $\mathbb {H}$ spanned by the scalars in $\mathbb {R}$ and the three basic bi-vectors

$$\begin{aligned} \textbf{i} = \textbf{e}_2 \textbf{e}_3,\qquad \textbf{j} = \textbf{e}_3 \textbf{e}_1,\qquad \textbf{k} = \textbf{e}_1 \textbf{e}_2 \end{aligned}$$

carved on that famous Dublin bridge by Hamilton, while $\textrm{Cliff}({\mathbb R^{3}})\cong \textrm{Cliff}_\circ ({\mathbb R^{3,1}})$ includes also reflections, interpreted as boosts from the relativistic perspective. In order to extend to the Galilean group of rigid transformations in $\mathbb {E}^n$, however, one needs to add one more dimension spanned by $\textbf{e}_\circ$ with $\textbf{e}_\circ ^2 = 0$. Then, it is convenient to use the $\star$-duality and represent points as n-vectors, lines as $(n-1)-$vectors etc., with actual vectors corresponding to their orthogonal hyper-planes.

In the 3D setting for instance, points are given by their homogeneous coordinates $\{p_i\}$ in the basis of three-vectors as

$$\begin{aligned} P = p_\circ I_3 - \textbf{e}_\circ ( p_1 \textbf{i} + p_2 \textbf{j} + p_3 \textbf{k}),\quad I_3 = \textbf{e}_1 \textbf{e}_2\textbf{e}_3 \end{aligned}$$

and the Euclidean projection is in the plane $p_\circ = 1$, i.e., $\textbf{r}_p$ has coordinates $p_i/p_\circ$, while $p_\circ = 0$ corresponds to a ‘point at infinity’ and cannot be normalized in this way. Similarly, lines are presented by their Plücker coordinates

$$\begin{aligned} \ell = u_1 \textbf{i} + u_2 \textbf{j} + u_3 \textbf{k} - \textbf{e}_\circ (m_1\textbf{e}_1 + m_2 \textbf{e}_2 + m_3\textbf{e}_3) \end{aligned}$$

with $\textbf{u}$ and $\textbf{m}$ denoting the so-called displacement and moment vector, respectively. Finally, planes are given by vectors in $\textrm{Cliff}({\mathbb R^{3,0,1}})$, namely as:

$$\begin{aligned} {\alpha } = n_1\textbf{e}_1 + n_2 \textbf{e}_2 + n_3\textbf{e}_3 - \delta \textbf{e}_\circ \end{aligned}$$

and the incidence relation $P\in \alpha$ is now written as $P\cdot \alpha = 0$ instead of the usual $\textbf{n}\cdot \textbf{r}_p = \delta$. If we denote bivectors with $\hat{\textbf{v}} = \textbf{v}I_3$, the so-called ‘flats’ are given as

$$\begin{aligned} P = p_\circ I_3 - \textbf{e}_\circ \hat{\textbf{p}},\qquad \ell = \hat{\textbf{u}} - \textbf{e}_\circ \textbf{m},\qquad \alpha = \textbf{n}-\delta \textbf{e}_\circ \end{aligned}$$

(22)

and we can use the wedge product as a ‘meet’ operator to find intersections, e.g.

$$\begin{aligned} \ell \wedge \alpha = (\hat{\textbf{u}} - \textbf{e}_\circ \textbf{m})\wedge (\textbf{n}-\delta \textbf{e}_\circ ) = \textbf{u}\cdot \textbf{n}I_3 - \textbf{e}_\circ (\delta \hat{\textbf{u}} - \textbf{n}\wedge \textbf{m}) \end{aligned}$$

which seems more handy that resolving linear algebraic systems. Note that if $\textbf{u}\perp \textbf{n}$, the two flats are parallel and we end up with a point at infinity ($p_\circ = 0$). Likewise, two parallel lines meet at infinity unless their moments are equal. We interpret $\textbf{e}_\circ$ as the normal vector to the plane at infinity (and to all $\mathbb {E}^3$) but should resist the temptation to replace it with the hypercomplex unit $\varepsilon$ used in dual numbers $\mathbb {D} \cong \mathbb {R}[\varepsilon ]$ as it does not commute with all elements of our algebra.

The interior product on the other hand, is used for projections, e.g. $(\ell \cdot \alpha )\alpha ^{-1}$ yields the $\alpha$-‘shadow’ of $\ell$. Using the Hodge duality, we define the ‘join’ operator

$$\begin{aligned} A \vee B = \star ^{-1}(\star A \wedge \star B),\quad \star A = A I_3^{-1} = - AI_3 \end{aligned}$$

(23)

via the famous De Morgan’s law and due to its symmetry, we also have

$$A \wedge B = \star ^{-1}(\star A \vee \star B).$$

In particular, one may use $\vee$ to represent the line $\ell = P\vee Q$ through points P and Q, or the plane $\alpha = \ell \vee P$ determined by the line $\ell$ and a point $P\notin \ell$, i.e. $\ell \wedge R = 0$, etc.

Finally, let us see how these geometric objects produce transformations. We begin with plane reflections as they generate all isometries in 3D due to the famous Catrtan-Dieudonne theorem, and in PGA they act as

$$\begin{aligned} \alpha :\; P \;\rightarrow \; \tilde{P} = \mathcal {M}_\alpha (P) = -\alpha P \alpha ^{-1}. \end{aligned}$$

(24)

They are interpreted as Lie elements generation motion via the exponential map

$$\begin{aligned} \mathcal {T}_\ell :\; P \;\rightarrow \; \tilde{P} = e^{-\ell } P e^\ell \end{aligned}$$

(25)

which corresponds to either a rotation ($\ell ^2<0$), translation ($\ell ^2 = 0$) or a screw displacement if $\ell$ is not a blade³. The magnitude of $\ell$ is proportional to the rotation angle, as we know from the examples of quaternions and complex numbers. For a more elaborate introduction to PGA we refer the reader to [9] and [10].