Object Detection using Geometrical Context Feedback

Abstract

We propose a new coherent framework for joint object detection, 3D layout estimation, and object supporting region segmentation from a single image. Our approach is based on the mutual interactions among three novel modules: (i) object detector; (ii) scene 3D layout estimator; (iii) object supporting region segmenter. The interactions between such modules capture the contextual geometrical relationship between objects, the physical space including these objects, and the observer. An important property of our algorithm is that the object detector module is capable of adaptively changing its confidence in establishing whether a certain region of interest contains an object (or not) as new evidence is gathered about the scene layout. This enables an iterative estimation procedure where the detector becomes more and more accurate as additional evidence about a specific scene becomes available. Extensive quantitative and qualitative experiments are conducted on the table-top dataset (Sun et al. in ECCV, 2010b) and two publicly available datasets (Hoiem et al. in CVPR, 2006; Sudderth et al. in IJCV, 2008), and demonstrate competitive object detection, 3D layout estimation, and segmentation results.

Appendices

Appendix A: Derivation of Object Detect Score

In detail, we define V(O,x|D) as the sum of individual probabilities over all observed images patches at location l _j and for all possible depth $d_{j}^{p}\in D$, i.e.,

where the summation over j aggregates the evidence from individual patch location, and the summation over depth $d_{j}^{p}$ marginalizes out the uncertainty of depth corresponding to each image patch location. Since C _j is calculated deterministically from l _j and $d_{j}^{p}$, and assuming O only depending on C _j , we obtain:

We further assign image patches with different depths to different index j. As a result, we can take only the summation over patch index j and obtain (1).

Appendix B: Proof of Three Objects Requirement

Equation (14) admits one or at most two non-trivial solutions of {f,n ₁,n ₂,n ₃} if at least three non-aligned observations (u _i ,v _i ) (i.e. non-collinear in the image) are available. If the observations are collinear, then (14) has infinite number of solutions.

$$ \left[ \begin{array}{c@{\quad}c@{\quad}c} u_1 & v_1 & f\\ u_2 & v_2 & f\\ u_3 & v_3 & f\\ &\vdots&\\ u_N & v_N & f\\ \end{array} \right] \left( \begin{array}{c} n_1\\ n_2\\ n_3 \end{array} \right)= \left( \begin{array}{c} -\cos\phi_1 \sqrt{u_1^2+v_1^2+f^2}\\ -\cos\phi_2 \sqrt{u_2^2+v_2^2+f^2}\\ -\cos\phi_3 \sqrt{u_3^2+v_3^2+f^2}\\ \vdots\\ -\cos\phi_N \sqrt{u_N^2+v_N^2+f^2}\\ \end{array} \right) $$

(14)

Proof

Suppose at least three objects are not collinear in a image, then the rank of the left matrix in the left-hand side of (14) is 3. Therefore (14) provides 3 independent constraints. Recall the unknowns in (14) are n ₁,n ₂,n ₃,f. With these constraints, each of n ₁,n ₂,n ₃ can be expressed as a function of f, i.e. n _i =n _i (f). Because ∥n∥=1, we obtain an equation about f:

$$\sum_{i=1,\ldots,3}{n_i^2(f)}=1 $$

In the above equation, f appears in the order of f ² and f ⁴. Therefore, there are at most two real positive solutions of f. Given f, {n ₁,n ₂,n ₃} can be computed as n _i =n _i (f).

On the other hand, if all objects are collinear in the image, then infinite number of solutions of (14) exist. If all objects are collinear, the rank of the left matrix in the left-hand side of (14) is 2. Without loss of generality, assume (u ₁,v ₁)≠0. In such a case, after using Gaussian elimination, (14) will be in the following form:

$$ \left[ \begin{array}{c@{\quad}c@{\quad}c} \alpha& \beta& f \\ \gamma& \epsilon& 0 \\ 0&0&0\\ &\vdots& \end{array} \right] \left( \begin{array}{c} n_1\\ n_2\\ n_3 \end{array} \right)= \begin{array}{c} \zeta\\ \eta\\ 0\\ \vdots \end{array} $$

(15)

If $\widehat{f}, \widehat{n}_{1}, \widehat{n}_{2}, \widehat{n}_{3}$ is solution, then $\widehat{f}, \widehat{n}_{1}+km_{1}, \widehat{n}_{2}+km_{2}, \widehat{n}_{3}+km_{3}$ is also a solution of (15), where (m ₁,m ₂,m ₃) is the non-trivial solution the following equation:

$$\left[ \begin{array}{c@{\quad}c@{\quad}c} \alpha& \beta& f \\ \gamma& \epsilon& 0 \\ \end{array} \right] \left( \begin{array}{c} m_1\\ m_2\\ m_3 \end{array} \right)= 0 $$

Hence, (14) admits infinite solutions. □