Abstract
There are two main strategies for solving correspondence problems in computer vision: sparse local feature based approaches and dense global energy based methods. While sparse feature based methods are often used for estimating the fundamental matrix by matching a small set of sophistically optimised interest points, dense energy based methods mark the state of the art in optical flow computation. The goal of our paper is to show that this separation into different application domains is unnecessary and can be bridged in a natural way. As a first contribution we present a new application of dense optical flow for estimating the fundamental matrix. Comparing our results with those obtained by feature based techniques we identify cases in which dense methods have advantages over sparse approaches. Motivated by these promising results we propose, as a second contribution, a new variational model that recovers the fundamental matrix and the optical flow simultaneously as the minimisers of a single energy functional. In experiments we show that our coupled approach is able to further improve the estimates of both the fundamental matrix and the optical flow. Our results prove that dense variational methods can be a serious alternative even in classical application domains of sparse feature based approaches.
Similar content being viewed by others
References
Alvarez, L., Esclarín, J., Lefébure, M., & Sánchez, J. (1999). A PDE model for computing the optical flow. In Proc. XVI congreso de ecuaciones diferenciales y aplicaciones (pp. 1349–1356). Las Palmas de Gran Canaria, Spain.
Alvarez, L., Deriche, R., Papadopoulo, T., & Sánchez, J. (2002a). Symmetrical dense optical flow estimation with occlusions detection. In A. Heyden, G. Sparr, M. Nielsen, & P. Johansen (Eds.), Lecture notes in computer science: Vol. 2350. Computer vision—ECCV 2002 (pp. 721–736). Berlin: Springer.
Alvarez, L., Deriche, R., Sánchez, J., & Weickert, J. (2002b). Dense disparity map estimation respecting image derivatives: a PDE and scale-space based approach. Journal of Visual Communication and Image Representation, 13(1/2), 3–21.
Amiaz, T., Lubetzky, E., & Kiryati, N. (2007). Coarse to over-fine optical flow estimation. Pattern Recognition, 40(9), 2496–2503.
Baker, S., Roth, S., Scharstein, D., Black, M. J., Lewis, J. P., & Szeliski, R. (2007). A database and evaluation methodology for optical flow. In Proc. 2007 IEEE international conference on computer vision. Rio de Janeiro: IEEE Computer Society Press.
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2009). A database and evaluation methodology for optical flow (Tech. Rep. MSR-TR-2009-179). Redmond: Microsoft Research, WA.
Barron, J. L., Fleet, D. J., & Beauchemin, S. S. (1994). Performance of optical flow techniques. International Journal of Computer Vision, 12(1), 43–77.
Bay, H., Ess, A., Tuytelaars, T., & Van Gool, L. J. (2008). Speeded-up robust features (SURF). Computer Vision and Image Understanding, 110(3), 346–359.
Ben-Ari, R., & Sochen, N. (2007). Variational stereo vision with sharp discontinuities and occlusion handling. In Proc. eleventh international conference on computer vision. Rio de Janeiro: IEEE Computer Society Press.
Bitton, D., Rosman, G., Nir, T., Bruckstein, A. M., Feuer, A., & Kimmel, R. (2009). Over-parameterized optical flow using a stereoscopic constraint (Tech. Rep. CIS-2009-18). Computer Science Department, Technion–Israel Institute of Technology, Israel.
Black, M. J., & Anandan, P. (1991). Robust dynamic motion estimation over time. In Proc. 1991 IEEE conference on computer vision and pattern recognition (pp. 292–302). Maui: IEEE Computer Society Press.
Brooks, M. J., Chojnacki, W., & Baumela, L. (1997). Determining the ego-motion of an uncalibrated camera from instantaneous optical flow. Journal of the Optical Society of America A, 14, 2670–2677.
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optic flow estimation based on a theory for warping. In T. Pajdla & J. Matas (Eds.), Lecture notes in computer science: Vol. 3024. Computer Vision—ECCV 2004 (pp. 25–36). Berlin: Springer.
Brox, T., Bregler, C., & Malik, J. (2009). Large displacement optical flow. In Proc. 2009 IEEE computer society conference on computer vision and pattern recognition (pp. 41–48). Miami: IEEE Computer Society Press.
Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision, 61(3), 211–231.
Bruhn, A., Weickert, J., Kohlberger, T., & Schnörr, C. (2006). A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. International Journal of Computer Vision, 70(3), 257–277.
Canny, J. (1986). A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8, 679–698.
Chan, T. F., & Vese, L. A. (2001). Active contours without edges. IEEE Transactions on Image Processing, 10(2), 266–277.
Chum, O., Matas, J., & Obdrzalek, S. (2004). Enhancing RANSAC by generalized model optimization. In K. S. Hong & Z. Zhang (Eds.), Lecture notes in computer science: Vol. 2. Proc. sixth Asian conference on computer vision (pp. 812–817).
Chum, O., Werner, T., & Matas, J. (2005). Two-view geometry estimation unaffected by a dominant plane. In Proc. 2005 IEEE computer society conference on computer vision and pattern recognition (pp. 772–779). San Diego: IEEE Computer Society Press.
Faugeras, O. (1992). What can be seen in three dimensions with an uncalibrated stereo rig. In G. Sandini (Ed.), Lecture notes in computer science: Vol. 588. Computer vision—ECCV 1992 (pp. 563–578). Berlin: Springer.
Faugeras, O., Luong, Q. T., & Papadopoulo, T. (2001). The geometry of multiple images. Cambridge: MIT Press.
Fischler, M., & Bolles, R. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24, 381–385.
Fitzgibbon, A. W. (2001). Simultaneous linear estimation of multiple view geometry and lens distortion. In Proc. 2001 IEEE computer society conference on computer vision and pattern recognition (pp. 125–132). Kauai: IEEE Computer Society Press.
Förstner, W., & Gülch, E. (1987). A fast operator for detection and precise location of distinct points, corners and centres of circular features. In Proc. ISPRS intercommission conference on fast processing of photogrammetric data (pp. 281–305). Interlaken, Switzerland.
Frahm, J. M., & Pollefeys, M. (2006). RANSAC for (quasi-) degenerate data (QDEGSAC). In Proc. 2006 IEEE computer society conference on computer vision and pattern recognition (pp. 453–460). New York: IEEE Computer Society Press.
Golub, G. H., & Van Loan, C. M. (1989). Matrix computations. Baltimore: The John Hopkins University Press.
Gwosdek, P., Zimmer, H., Grewenig, S., Bruhn, A., & Weickert, J. (2010). A highly efficient GPU implementation for variational optic flow based on the Euler-Lagrange framework. In Proc. 2010 ECCV workshop on computer vision with GPUs. Heraklion, Greece.
Hanna, K. J. (1991). Direct multi-resolution estimation of ego-motion and structure from motion. In Proc. workshop on visual motion (pp. 156–162). Washington: IEEE Computer Society Press.
Harris, C. G., & Stephens, M. (1988). A combined corner and edge detector. In Proc. fourth alvey vision conference (pp. 147–152). Manchester, England.
Hartley, R. (1997). In defense of the eight-point algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(6), 580–593.
Hartley, R., & Zisserman, A. (2000). Multiple view geometry in computer vision. Cambridge: Cambridge University Press.
Hartley, R., Gupta, R., & Chang, T. (1992). Stereo from uncalibrated cameras. In Proc. 1992 IEEE international conference on image processing (pp. 761–764). Champaign: IEEE Computer Society Press.
Horn, B., & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203.
Huber, P. J. (1981). Robust statistics. New York: Wiley.
Kanatani, K., Shimizu, Y., Ohta, N., Brooks, M. J., Chojnacki, W., & van den Hengel, A. (2000). Fundamental matrix from optical flow: optimal computation and reliability evaluation. Journal of Electronic Imaging, 9, 194–202.
Kim, Y. H., Martinez, A. M., & Kak, A. C. (2005). Robust motion estimation under varying illumination. Image and Vision Computing, 23(4), 365–375.
Klaus, A., Sormann, M., & Karner, K. (2006). Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure. In Proc. 18th international conference on pattern recognition, Part III (Vol. 3, pp. 15–18). Hong Kong, China.
Kolmogorov, V., & Zabih, R. (2002). Multi-camera scene reconstruction via graph cuts. In A. Heyden, G. Sparr, M. Nielsen, & P. Johansen (Eds.), Lecture notes in computer science: Vol. 2352. Computer vision—ECCV 2002, Part III (pp. 82–96). Berlin: Springer.
Lei, C., Selzer, J., & Yang, Y. H. (2006). Region-tree based stereo using dynamic programming optimization. In Proc. 2006 IEEE computer society conference on computer vision and pattern recognition (pp. 2378–2385). Washington: IEEE Computer Society Press.
Levenberg, K. (1944). A method for the solution of certain non-linear problems in least squares. Quarterly of Applied Mathematics, 2, 164–168.
Longuet-Higgins, H. C. (1981). A computer algorithm for reconstructing a scene from two projections. Nature, 293, 133–135.
Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.
Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In Proc. seventh international joint conference on artificial intelligence (pp. 674–679). Vancouver, Canada.
Luong, Q. T., & Faugeras, O. D. (1996). The fundamental matrix: theory, algorithms, and stability analysis. International Journal of Computer Vision, 17(1), 43–75.
Mainberger, M., Bruhn, A., & Weickert, J. (2008). Is dense optical flow useful to compute the fundamental matrix. In A. Campilho & M. Kamel (Eds.), Lecture notes in computer science: Vol. 5112. Image analysis and recognition (pp. 630–639). Póvoa de Varzim, Portugal. Berlin: Springer.
Marquardt, D. (1963). An algorithm for least-squares estimation of nonlinear parameters. SIAM Journal on Applied Mathematics, 11, 431–441.
Marr, D., & Hildreth, E. (1980). Theory of edge detection. Proceedings of the Royal Society of London, Series B, 207, 187–217.
Mémin, E., & Pérez, P. (1998). Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Transactions on Image Processing, 7(5), 703–719.
Mémin, E., & Pérez, P. (2002). Hierarchical estimation and segmentation of dense motion fields. International Journal of Computer Vision, 46(2), 129–155.
Mikolajczyk, K., & Schmid, C. (2005). A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(10), 1615–1630.
Mileva, Y., Bruhn, A., & Weickert, J. (2007). Illumination-robust variational optical flow with photometric invariants. In F. Hamprecht, C. Schnörr, & B. Jähne (Eds.), Lecture notes in computer science: Vol. 4713. Pattern recognition (pp. 152–162). Berlin: Springer.
Nagel, H. H., & Enkelmann, W. (1986). An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8, 565–593.
Nir, T., Bruckstein, A. M., & Kimmel, R. (2008). Over-parameterized variational optical flow. International Journal of Computer Vision, 76(2), 205–216.
Ohta, N., & Kanatani, K. (1995). Optimal structure from motion algorithm for optical flow. IEICE Transactions on Information and Systems, E78-D(12), 1559–1566.
Proesmans, M., Van Gool, L., Pauwels, E., & Oosterlinck, A. (1994). Determination of optical flow and its discontinuities using non-linear diffusion. In J. O. Eklundh (Ed.), Lecture notes in computer science: Vol. 801. Computer vision—ECCV ’94 (pp. 295–304). Berlin: Springer.
Raguram, R., Frahm, J. M., & Pollefeys, M. (2008). A comparative analysis of RANSAC techniques leading to adaptive real-time random sample consensus. In D. Forsyth, P. Torr, & A. Zisserman (Eds.), Lecture notes in computer science: Vol. 5303. Computer vision—ECCV 2008, Part II (pp. 500–513). Berlin: Springer.
Roth, S., & Black, M. (2005). On the spatial statistices of optical flow. In Proc. tenth international conference on computer vision (Vol. 1, pp. 42–49). Beijing: IEEE Computer Society Press.
Rousseeuw, P. J., & Leroy, A. M. (1987). Robust regression and outlier detection. New York: Wiley.
Saragih, J., & Goecke, R. (2007). Monocular and stereo methods for AAM learning from video. In Proc. 2007 IEEE computer society conference on computer vision and pattern recognition. Minneapolis: IEEE Computer Society Press.
Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1–3), 7–42.
Schlesinger, D., Flach, B., & Shekhovtsov, A. (2004). A higher order MRF-model for stereo-reconstruction. In C. Rasmussen, H. Bülthoff, M. Giese, & B. Schölkopf (Eds.), Lecture notes in computer science: Vol. 3175. Pattern recognition (pp. 440–446). Berlin: Springer.
Schnörr, C. (1994). Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. twelfth international conference on pattern recognition (Vol. A, pp. 661–663). Jerusalem: IEEE Computer Society Press.
Seitz, S., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. 2006 IEEE conference on computer vision and pattern recognition (pp. 519–528). New York: IEEE Computer Society Press.
Sheikh, Y., Hakeem, A., & Shah, M. (2007). On the direct estimation of the fundamental matrix. In Proc. 2007 IEEE computer society conference on computer vision and pattern recognition. Minneapolis: IEEE Computer Society Press.
Shi, J., & Tomasi, C. (1994). Good features to track. In Proc. 1994 IEEE computer society conference on computer vision and pattern recognition (pp. 593–600). Seattle: IEEE Computer Society Press.
Slesareva, N., Bruhn, A., & Weickert, J. (2005). Optic flow goes stereo: a variational approach for estimating discontinuity-preserving dense disparity maps. In W. Kropatsch, R. Sablatnig, & A. Hanbury (Eds.), Lecture notes in computer science: Vol. 3663. Pattern recognition (pp. 33–40). Berlin: Springer.
Steinbrücker, F., Pock, T., & Cremers, D. (2009a). Advanced data terms for variational optic flow estimation. In M. Magnor, B. Rosenhahn, & H. Theisel (Eds.), Proceedings of the vision, modeling, and visualization workshop (VMV), DNB (pp. 155–164).
Steinbrücker, F., Pock, T., & Cremers, D. (2009b). Large displacement optical flow computation without warping. In Proc. twelfth international conference on computer vision. Kyoto: IEEE Computer Society Press.
Stewart, C. V. (1999). Robust parameter estimation in computer vision. SIAM Review, 41(3), 513–537.
Strecha, C., Tuytelaars, T., & Van Gool, L. (2003). Dense matching of multiple wide-baseline views. In Proc. ninth international conference on computer vision (Vol. 2, pp. 1194–1201). Washington: IEEE Computer Society Press.
Strecha, C., Fransens, R., & Van Gool, L. (2004). A probabilistic approach to large displacement optical flow and occlusion detection. In D. Comaniciu, K. Kanatani, R. Mester, & D. Suter (Eds.), Lecture notes in computer science: Vol. 3247. Statistical methods in video processing (pp. 71–82). Berlin: Springer.
Strecha, C., von Hansen, W., Van Gool, L. J., Fua, P., & Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In Proc. 2008 IEEE computer society conference on computer vision and pattern recognition. Anchorage: IEEE Computer Society Press.
Sun, D., Roth, S., Lewis, J. P., & Black, M. J. (2008). Learning optical flow. In D. Forsyth, P. Torr, & A. Zisserman (Eds.), Lecture notes in computer science: Vol. 5304. Computer vision—ECCV 2008, Part III (pp. 83–97). Berlin: Springer.
Tomasi, C., & Kanade, T. (1991). Detection and tracking of point features (Tech. Rep. CMU-CS-91-132). School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.
Torr, P. H. S., & Murray, D. W. (1997). The development and comparison of robust methods for estimating the fundamental matrix. International Journal of Computer Vision, 24(3), 271–300.
Tsai, R., & Huang, T. (1984). Uniqueness and estimation of thee-dimensional motion parameters of rigid objects with curved surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(1), 13–26.
Valgaerts, L., Bruhn, A., & Weickert, J. (2008). A variational model for the joint recovery of the fundamental matrix and the optical flow. In G. Rigoll (Ed.), Lecture Notes in Computer Science: Vol. 3663. Pattern recognition (pp. 314–324). Berlin: Springer.
Valgaerts, L., Bruhn, A., Zimmer, H., Weickert, J., Stoll, C., & Theobalt, C. (2010). Joint estimation of motion, structure and geometry from stereo sequences. In K. Daniilidis, P. Maragos, & N. Paragios (Eds.), Lecture Notes in Computer Science: Vol. 6314. Computer vision—ECCV 2010 (pp. 568–581). Berlin: Springer.
Viéville, T., & Faugeras, O. (1995). Motion analysis with a camera with unknown, and possibly varying intrinsic parameters. In Proc. fifth international conference on computer vision (pp. 750–756). Cambridge: IEEE Computer Society Press.
Wang, H., & Brady, M. (1994). A practical solution to corner detection. In Proc. 1994 IEEE international conference on image processing (Vol. 1, pp. 919–923). Austin: IEEE Computer Society Press.
Wedel, A., Pock, T., Braun, J., Franke, U., & Cremers, D. (2008). Duality TV-L1 flow with fundamental matrix prior. In Proc. image and vision computing New Zealand. Auckland: IEEE Computer Society Press.
Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In Proc. twelfth international conference on computer vision. Kyoto: IEEE Computer Society Press.
Weickert, J., & Schnörr, C. (2001). A theoretical framework for convex regularizers in PDE-based computation of image motion. International Journal of Computer Vision, 45(3), 245–264.
Weng, J., Huang, T., & Ahuja, N. (1989). Motion and structure from two perspective views: algorithms, error analysis and error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(5), 451–476.
Weng, J., Ahuja, N., & Huang, T. (1993). Optimal motion and structure estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(9), 864–884.
Werlberger, M., Pock, T., & Bischof, H. (2010). Motion estimation with non-local total variation regularization. In Proc. 2010 IEEE computer society conference on computer vision and pattern recognition (pp. 2464–2471). San Francisco: IEEE Computer Society Press.
Zach, C., Pock, T., & Bischof, H. (2007a). A duality based approach for realtime TV-L 1 optical flow. In F. Hamprecht, C. Schnörr, & B. Jahne (Eds.), Lecture Notes in Computer Science: Vol. 4713. Pattern recognition (pp. 214–223). Berlin: Springer.
Zach, C., Pock, T., & Bischof, H. (2007b). A globally optimal algorithm for robust TV-L 1 range image integration. In Proc. ninth international conference on computer vision. Rio de Janeiro: IEEE Computer Society Press.
Zeisl, B., Georgel, P. F., Schweiger, F., Steinbach, E., & Navab, N. (2009). Estimation of location uncertainty for scale invariant feature points. In Proc. 2009 British machine vision conference. London, England.
Zhang, Z. (1998). Determining the epipolar geometry and its uncertainty: a review. International Journal of Computer Vision, 27(2), 161–195.
Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H. P. (2009). Complementary optic flow. In D. Cremers, Y. Boykov, A. Blake, & F. R. Schmidt (Eds.), Lecture notes in computer science: Vol. 5681. Energy minimization methods in computer vision and pattern recognition—EMMCVPR (pp. 207–220). Berlin: Springer.
Zimmer, H., Bruhn, A., & Weickert, J. (2011). Optic flow in harmony. International Journal of Computer Vision, 93(3), 638–388.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Valgaerts, L., Bruhn, A., Mainberger, M. et al. Dense versus Sparse Approaches for Estimating the Fundamental Matrix. Int J Comput Vis 96, 212–234 (2012). https://doi.org/10.1007/s11263-011-0466-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-011-0466-7