Construction of a flexible and scalable 4D light field camera array using Raspberry Pi clusters

Abstract

The light field is a 4D function that represents the radiance of light traveling in every direction through every point in 3D space. In this paper, we demonstrate how effectively 4D light fields can be sampled from the real world using a custom-built 4D light field camera system constructed based on cheaply and commonly available Raspberry Pi computers. The camera system consisting of a camera array and a dedicated calibration software is flexible and scalable in structure because the system can be reconfigured easily to meet the user’s needs. We show the effectiveness of the camera system by interactively visualizing captured 4D light fields using our GPU-accelerated light field renderer supporting a virtual camera model that synthesizes various photographic effects including zooming and panning, refocusing, focus breathing, and depth-of-field control. We believe that the do-it-yourself 4D light field camera will be explored effectively in traditional application fields, such as computer graphics and vision, and in other areas like science education.

Appendix: Estimation of the rigid body transformation for finding the geometry parameters of the camera array

Let $M(\alpha , \beta ;\theta ) = T(\alpha , \beta )R(\theta )$ be the rigid body transformation we are to seek. The transformed position $\bar{\mathbf{P}}_{ij}$ of $\mathbf{P}_{ij}$, the center of the (i, j)-th grid cell, is then

$$\begin{aligned} \begin{pmatrix} \bar{\mathbf{P}}_{ij}\\ 1 \end{pmatrix} = T(\alpha , \beta )R(\theta ) \begin{pmatrix} \mathbf{P}_{ij}\\ 1 \end{pmatrix} = \begin{bmatrix} x_{ij} \cos \theta + y_{ij} \sin \theta + \alpha \\ - \,x_{ij} \sin \theta + y_{ij} \cos \theta + \beta \\ 1 \end{bmatrix} \end{aligned}$$

where

$$\begin{aligned} \mathbf{P}_{ij} \equiv \begin{pmatrix} x_{ij}\\ y_{ij} \end{pmatrix} = \begin{pmatrix} -140\\ -140 \end{pmatrix} + \begin{pmatrix} 28i+14\\ 28j+14 \end{pmatrix} = \begin{pmatrix} 28i-126\\ 28j-126 \end{pmatrix}. \end{aligned}$$

The squared sum of the distances between $\bar{\mathbf{P}}_{ij}$ and $\mathbf{P}^*_{ij}= (x^*_{ij}, y^*_{ij})^t$, the projected position of the (i, j)-th camera in the 2D coordinate system, is now expressed as

$$\begin{aligned} F(\alpha , \beta ;\theta )= & {} \sum _{j=0}^{9} \sum _{i=0}^{9} \Vert \bar{\mathbf{P}}_{ij} - \mathbf{P}_{ij}^{*} \Vert _{2}^{2}\\= & {} \sum _{j=0}^{9} \sum _{i=0}^{9}\{ \,( x_{ij} \cos \theta + y_{ij} \sin \theta + \alpha - x_{ij}^{*} )^{2} \nonumber \\&+ \,( - x_{ij} \sin \theta + y_{ij} \cos \theta + \beta - y_{ij}^{*} )^{2} \,\}. \end{aligned}$$

Then,

$$\begin{aligned} \frac{\partial F}{\partial \alpha }= & {} \sum _{j=0}^{9} \sum _{i=0}^{9} {2(x_{ij} \cos \theta + y_{ij} \sin \theta + \alpha - x_{ij}^{*}) }\\= & {} 2\biggl (\cos \theta \sum _{j=0}^{9} \sum _{i=0}^{9} x_{ij} + \sin \theta \sum _{j=0}^{9} \sum _{i=0}^{9} y_{ij} \nonumber \\&+ \,\sum _{j=0}^{9} \sum _{i=0}^{9} \alpha - \sum _{j=0}^{9} \sum _{i=0}^{9} x_{ij}^{*} \biggl ) \\= & {} 2\left( \sum _{j=0}^{9} \sum _{i=0}^{9} \alpha - \sum _{j=0}^{9} \sum _{i=0}^{9} x_{ij}^{*}\right) . \end{aligned}$$

From $F(\alpha , \beta ;\theta ) = 0$ and the similarity between the two variables $\alpha$ and $\beta$, we find the total error is minimized at $(\alpha ^*, \beta ^*)$ where

$$\begin{aligned} \alpha ^{*} = \frac{ \sum _{j=0}^{9} \sum _{i=0}^{9} x_{ij}^{*} }{100}, \quad \beta ^{*} = \frac{ \sum _{j=0}^{9} \sum _{i=0}^{9} y_{ij}^{*} }{100}. \end{aligned}$$

Next, for the remaining variable,

$$\begin{aligned} \frac{\partial F}{\partial \theta }= & {} \sum _{j=0}^{9} \sum _{i=0}^{9} \{ \,2(x_{ij} \cos \theta + y_{ij} \sin \theta + \alpha - x_{ij}^{*}) \\&\cdot (-\,x_{ij} \sin \theta + y_{ij} \cos \theta ) \\&+ \,2(-\,x_{ij} \sin \theta + y_{ij} \cos \theta + \beta - y_{ij}^{*}) \\&\cdot (-\,x_{ij} \cos \theta - y_{ij} \sin \theta ) \,\} \\= & {} 2 \left\{ \,\sin \theta \sum _{j=0}^{9} \sum _{i=0}^{9} (x_{ij}^{*}+y_{ji}^{*}) x_{ij} \right. \nonumber \\&\left. - \,\cos \theta \sum _{j=0}^{9} \sum _{i=0}^{9} (x_{ji}^{*}+y_{ij}^{*}) y_{ij} \,\right\} . \end{aligned}$$

To estimate $\theta ^*$ such that $\frac{\partial F}{\partial \theta } = 0$, the Newton–Raphson method [6] is applied to the function $f(\theta ) = S \sin \theta - C\cos \theta$ with $S = \sum _{j=0}^{9} \sum _{i=0}^{9} (x_{ij}^{*}+y_{ji}^{*}) x_{ij}$ and $C = \sum _{j=0}^{9} \sum _{i=0}^{9} (x_{ji}^{*}+y_{ij}^{*}) y_{ij}$. Note that since the rotational displacement is usually small, the following Newton–Rapson iteration converges quickly to the solution $\theta ^*$ for the initial value $\theta _0 = 0$:

$$\begin{aligned} \theta _{k+1} = \theta _{k}-\frac{f(\theta _{k})}{f'(\theta _{k})} = \theta _{k}-\frac{S \sin \theta _{k} - C \cos \theta _{k}}{S \cos \theta _{k} + C \sin \theta _{k}}. \end{aligned}$$