Fast depth from defocus from focal stacks

Abstract

We present a new depth from defocus method based on the assumption that a per pixel blur estimate (related with the circle of confusion), while ambiguous for a single image, behaves in a consistent way when applied over a focal stack of two or more images. This allows us to fit a simple analytical description of the circle of confusion to the different per pixel measures to obtain approximate depth values up to a scale. Our results are comparable to previous work while offering a faster and flexible pipeline.

Appendix A: Least squares function analysis

In this appendix, we show how to cast the depth estimation problem as an optimization problem. Consider the optimization problem for a single signal with $n$ blur estimates, and each $c_i$ is captured with a focal position $S_1^i$. Let

$$\begin{aligned} g_i(x) = \left( c_i - A \frac{|x-S_1^i|}{x} \frac{f}{S_1 - f}\right) ^2 \end{aligned}$$

(13)

The function $g_i(x)$ has a critical point at $S_1^i$ because the derivative at $S_1^i$ of $g_i(x)$ does not exist due to the term $|x-S_1^i|$ in the function. Furthermore, if the blur estimate $c_i$ is less than the circle of confusion size

$$\begin{aligned} c=\frac{f^2}{N(S_1^i-f)} \end{aligned}$$

(14)

for a depth $x$ at infinity, then the function will have two local minimizers, as shown in Fig. 10, at the points $g(x)=0$ where

$$\begin{aligned} x=\frac{S_1^iAf}{Af-c_i(S_1^i-f)} \end{aligned}$$

(15)

and

$$\begin{aligned} x=\frac{S_1^iAf}{Af+c_i(S_1^i-f)}. \end{aligned}$$

(16)

Fig. 10

Plot of $g_i(x)$ showing a local maximizer at the point $S_1^i = 0.75$ m and two local minimizers on either size of the maximizer

However, if $c_i=0$ then the function will have one minimizer at $x=S_1^i$, and similarly if $x$ is larger than the circle of confusion size for a depth at infinity, then $g_i(x)$ will have only one minimizer somewhere within the interval $(0,S_1^i)$.

For the purposes of optimization, we assume that

$$\begin{aligned} 0<c_i<\frac{f^2}{N(S_1^i-f)}. \end{aligned}$$

(17)

This assumption introduces the restriction that the depth of a signal in the focal stack cannot be too close to the lens.

A further restriction for the depth $x$ is that $S_1^1<x<S_1^n$ where $0<S_1^1<S_1^2<\cdots <S_1^n$. This restriction limits the depth of any point in the focal stack to be between the closest focal position of the lens and the farthest focal position.

With these assumptions, we can now look at the least squares optimization equation

$$\begin{aligned} z(x)=\sum _{i=1}^ng_i(x). \end{aligned}$$

(18)

Because each $g_i'(x)$ is undefined at $x=S_1^i$ for all $i=1,\ldots ,n$, the function $z(x)$ has critical points at $S_1^1,\ldots ,S_1^n$. Furthermore, $z(x)$ is continuous everywhere else for $x>0$ because the functions $g_i(x)$ are continuous where $x>0$ and $x \ne S_1^i$. Because $g_i(x)$ has a local maximizer at $S_1^i$, this point may be a local maximizer for $z(x)$. This gives us $n-1$ intervals on which $z(x)$ is continuous for $S_1^1<x<S_1^n$, and these intervals are $(S_1^1,S_1^2),(S_1^2,S_1^3),\ldots ,(S_1^{n-1},S_1^n)$. These open intervals may or may not contain a local minimizer, and if an interval does contain a local minimizer, it might be the global minimizer of $z(x)$ on the interval $(S_1^1,S_1^n)$.

Under certain conditions, $z(x)$ is convex within the interval $(S_1^1,S_1^{i+1})$ for all $i=1,\ldots ,n-1$. Note that $g_j(x)$ is convex within the open interval for all $j=1,\ldots ,n$. To see this, assume that Eq. 11 holds and that the focus position of the lens is always greater than the focal length $f$ of the lens so that $r>0$. We also assume that

$$\begin{aligned} S_1^n \le \frac{3rS_1^j}{2c_j}. \end{aligned}$$

(19)

If $x<S_1^j$ then the absolute value term $|x-S_1^j|$ in $g_i(x)$ becomes $-x+S_1^j$. From this, we know that

$$\begin{aligned} rS_1^j \ge 2c_jx \end{aligned}$$

(20)

from Relation 11 and because $x$ and $S_1^j$ are positive. Rearranging the relation, we get

$$\begin{aligned} -2c_jS_1^j + rS_1^j \ge 0. \end{aligned}$$

(21)

Since $x<S_1^j$, $2rx<2rS_1^j$ and $2rS_1^j-2rx>0$. Therefore,

$$\begin{aligned} -2c_jx + 3rS_1^j - 2rx&= -2c_jx + rS_1^j + (2rS_1^j - 2rx)\nonumber \\&\ge 2rS_1^j - 2rx\nonumber \\&> 0 \end{aligned}$$

(22)

Furthermore, since $x>0$, $r>0$, and $S_1^j>0$, we know that

$$\begin{aligned} \frac{2rS_1^j}{x^4} > 0. \end{aligned}$$

(23)

Therefore, we know that

$$\begin{aligned} g_j''(x) = \frac{2rS_1^j(-2c_jx+3rS_1^j-2rx)}{x^4} > 0 \end{aligned}$$

(24)

for $0<x<S_1^j$.

If $x>S_1^j$, then

$$\begin{aligned} x < S_1^n \le \frac{3rS_1^j}{2c_j} \end{aligned}$$

(25)

from Eq. 19 and that $x<S_1^n$. Since $c_j>0$, we can multiply the relation by $2c_j$ to get

$$\begin{aligned} 3rS_1^j>2c_jx. \end{aligned}$$

(26)

From relation (11), we can say that

$$\begin{aligned} 2r-2c_j \ge 4c_j-2c_j = 2c_j. \end{aligned}$$

(27)

Therefore,

$$\begin{aligned} 3rS_1^j > x(2r-2c_j) \ge x(2c_j). \end{aligned}$$

(28)

Distributing $x$ in the above relation, we get

$$\begin{aligned} 3rS_1^j > 2rx-2c_jx \end{aligned}$$

(29)

Rearranging the terms, we get

$$\begin{aligned} 2c_jx+3rS_1^j - 2rx > 0. \end{aligned}$$

(30)

Multiplying by the left-hand side of (23), we get

$$\begin{aligned} g_j''(x) = \frac{2rS_1^j(2c_jx+3rS_1^j-2rx)}{x^4} > 0 \end{aligned}$$

(31)

for $S_1^j < x < S_1^n$.

As shown above, the second derivative of $g_j(x)$ is always positive on the interval $(S_1^1,S_1^n)$ except at the point $S_1^j$ for all $j=1,\ldots ,n$. Since $z(x)$ is the summation of all $g_j(x)$, $z(x)$ is also convex on the interval except at the points $S_1^1,S_1^2,\ldots ,S_1^n$. Therefore, $z(x)$ is convex in the intervals $(S_1^i, S_1^{i+1})$ for all $i=1,2,\ldots ,n-1$. As a consequence, if $S_1^i$, and $S_1^{i+1}$ are local maximizers, then there is some local minimizer within the open interval $(S_1^1,S_1^n)$. From this, a global minimizer can be identified which gives the best depth estimate for the given signal on the interval $(S_1^1,S_1^n)$. Figure 11 shows an example of $z(x)$ with the local maximizers and minimizers.

Fig. 11

Plot of $z(x)$ shown in show dark blue with $g_1(x)$, $g_2(x)$, and $g_3(x)$ shown in red, light blue, and green, respectively. This shows $z(x)$ with local maximizers at $S_1^1=0.75$, $S_1^2=1$, and $S_1^3=1.5$ and local minimizers in the intervals $(S_1^1, S_1^2)$ and $(S_1^2,S_1^3)$