Skip to main content
SpringerLink
Log in

On the Spatial Statistics of Optical Flow

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We present an analysis of the spatial and temporal statistics of “natural” optical flow fields and a novel flow algorithm that exploits their spatial statistics. Training flow fields are constructed using range images of natural scenes and 3D camera motions recovered from hand-held and car-mounted video sequences. A detailed analysis of optical flow statistics in natural scenes is presented and machine learning methods are developed to learn a Markov random field model of optical flow. The prior probability of a flow field is formulated as a Field-of-Experts model that captures the spatial statistics in overlapping patches and is trained using contrastive divergence. This new optical flow prior is compared with previous robust priors and is incorporated into a recent, accurate algorithm for dense optical flow computation. Experiments with natural and synthetic sequences illustrate how the learned optical flow prior quantitatively improves flow accuracy and how it captures the rich spatial structure found in natural scene motion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • 2d3 Ltd. 2002. boujou. http://www.2d3.com

  • Alvarez, L., Weickert, J., and Sánchez, J. 2000. Reliable estimation of dense optical flow fields with large displacements. Int. J. Comput. Vision, 39(1):41–56.

    Article  MATH  Google Scholar 

  • Barbu, A. and Yuille, A. 2004. Motion estimation by Swendsen-Wang cuts. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 1, pp. 754–761.

  • Barron, J.L., Fleet, D.J., and Beauchemin, S.S. 1994. Performance of optical flow techniques. Int. J. Comput. Vision, 12(1):43–77.

    Article  Google Scholar 

  • Ben-Ari, R. and Sochen, N. 2006. A general framework and new alignment criterion for dense optical flow. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 1, pp. 529–536.

  • Betsch, B.Y., Einhäuser, W., Körding, K.P., and König, P. 2004. The world from a cat’s perspective—Statistics of natural videos. Biological Cybernetics, 90(1):41–50.

    Article  MATH  Google Scholar 

  • Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), pp. 296–302.

  • Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comput. Vis. Image Und., 63(1):75–104.

    Article  Google Scholar 

  • Bruhn, A. 2006. Personal Communication.

  • Bruhn, A., Weickert, J., and Schnörr, C. 2005. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. Int. J. Comput. Vision, 61(3):211–231.

    Article  Google Scholar 

  • Calow, D., Krüger, N., Wörgötter, F., and Lappe, M. 2004. Statistics of optic flow for self-motion through natural scenes. In Dynamic Perception, Ilg, U., Bülthoff, H. and Mallot, H. (eds.), pp. 133–138.

  • Cremers, D. and Soatto, S. 2005. Motion competition: A variational approach to piecewise parametric motion segmentation. Int. J. Comput. Vision, 62(3):249–265.

    Article  Google Scholar 

  • Davis, T.A. 2004.A column pre-ordering strategy for the unsymmetric-pattern multifrontal method. ACM Transactions on Mathematical Software, 30(2):165–195.

    Article  MATH  Google Scholar 

  • Fablet, R. and Bouthemy, P. 2001. Non parametric motion recognition using temporal multiscale Gibbs models. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 1, pp. 501–508.

  • Fermüller, C., Shulman, D., and Aloimonos, Y. 2001. The statistics of optical flow. Comput. Vis. Image Und., 82(1):1–32.

    Article  MATH  Google Scholar 

  • Fleet, D.J., Black, M.J., Yacoob, Y., and Jepson, A.D. 2000. Design and use of linear models for image motion analysis. Int. J. Comput. Vision, 36(3):171–193.

    Article  Google Scholar 

  • Fleet, D.J., Black, M.J., and Nestares, O. 2002. Bayesian inference of visual motion boundaries. In Exploring Artificial Intelligence in the New Millennium, G. Lakemeyer and B. Nebel (eds.), Morgan Kaufmann Publisher, pp. 139–174.

  • Grenander, U. and Srivastava, A. 2001. Probability models for clutter in natural images. IEEE Trans. Pattern Anal. Mach. Intell., 23(4):424–429.

    Article  Google Scholar 

  • Heitz, F. and Bouthemy, P. 1993. Multimodal estimation of discontinuous optical flow using Markov random fields. IEEE Trans. Pattern Anal. Mach. Intell., 15(12):1217–1232.

    Article  Google Scholar 

  • Hinton, G.E. 1999. Products of experts. In Int. Conf. on Art. Neur. Netw. (ICANN), vol. 1, pp. 1–6.

  • Hinton, G.E. 2002. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771–1800.

    Article  MATH  Google Scholar 

  • Horn, B.K.P. and Schunck, B.G. 1981. Determining optical flow. Artificial Intelligence, 17(1–3):185–203.

    Article  Google Scholar 

  • Huang, J., Lee, A.B., and Mumford, D. 2000. Statistics of range images. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 1, pp. 1324ff.

  • Huang, J. 2000. Statistics of Natural Images and Models. PhD thesis, Brown University.

  • Irani, M. 1999. Multi-frame optical flow estimation using subspace constraints. In IEEE Int. Conf. on Comp. Vis. (ICCV), vol. 1, pp. 626–633.

  • Kailath, T. 1967. The divergence and Bhattacharyya distance measures in signal selection.IEEE Transactions on Communication Technology, COM-15(1):52–60.

    Article  Google Scholar 

  • Konrad, J. and Dubois, E. 1988. Multigrid Bayesian estimation of image motion fields using stochastic relaxation. In IEEE Int. Conf. on Comp. Vis. (ICCV), pp. 354–362.

  • Krajsek, K. and Mester, R. 2006. On the equivalence of variational and statistical differential motion estimation. In Southwest Symposium on Image Analysis and Interpretation, Denver, Colorado, pp. 11–15.

  • Lee, A.B. and Huang, J. 2000. Brown range image database. http://www.dam.brown.edu/ptg/brid/index.html

  • Lee, A.B., Mumford, D., and Huang, J. 2001. Occlusion models for natural images: A statistical study of a scale-invariant dead leaves model. Int. J. Comput. Vision, 41(1–2):35–59.

    Article  MATH  Google Scholar 

  • Lewen, G.D., Bialek, W., and de Ruyter van Steveninck, R.R. 2001. Neural coding of naturalistic motion stimuli. Network: Comp. Neural, 12(3):317–329.

    Article  Google Scholar 

  • Lu, H. and Yuille, A.L. 2006. Ideal observers for detecting motion: Correspondence noise. In Adv. in Neur. Inf. Proc. Sys. (NIPS), vol. 18, pp. 827–834.

  • Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Int. J. Conf. on Art. Intel. (IJCAI), pp. 674–679.

  • Marroquin, J., Mitter, S., and Poggio, T. 1987. Probabilistic solutions of ill-posed problems in computational vision. J. Am. Stat. Assoc., 82(397):76–89.

    Article  MATH  Google Scholar 

  • Martin, D., Fowlkes, C., Tal, D., and Malik, J. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In IEEE Int. Conf. on Comp. Vis. (ICCV), vol. 2, pp. 416–423.

  • Mémin, É. and Pérez, P. 2002. Hierarchical estimation and segmentation of dense motion fields. Int. J. Comput. Vision, 46(2):129–155.

    Article  MATH  Google Scholar 

  • Murray, D.W. and Buxton, B.F. 1987. Scene segmentation from visual motion using global optimization. IEEE Trans. Pattern Anal. Mach. Intell., 9(2):220–228.

    Article  Google Scholar 

  • Olshausen, B.A. and Field, D.J. 1996. Natural image statistics and efficient coding. Network: Comp. Neural, 7(2):333–339.

    Article  Google Scholar 

  • Papenberg, N., Bruhn, A., Brox, T., Didas, S., and Weickert, J. 2006. Highly accurate optic flow computation with theoretically justified warping. Int. J. Comput. Vision, 67(2):141–158.

    Article  Google Scholar 

  • Proesmans, M., Van Gool, L.J., Pauwels, E.J., and Oosterlinck, A. 1994. Determination of optical flow and its discontinuities using non-linear diffusion. In Eur. Conf. on Comp. Vis. (ECCV), J.-O. Eklundh (ed.), vol. 801 of Lect. Notes in Comp. Sci., pp. 295–304.

  • Ross, M.G. and Kaelbling, L.P. 2005. Learning static object segmentation from motion segmentation. In Nat. Conf. on Art. Int. (AAAI), Menlo Park, California, AAAI Press, pp. 956–961.

  • Roth, S. and Black, M.J. 2005a. Fields of experts: A framework for learning image priors. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), vol. 2, pp. 860–867.

  • Roth, S. and Black, M.J. 2005b. On the spatial statistics of optical flow. In IEEE Int. Conf. on Comp. Vis. (ICCV), vol. 1, pp. 42–49.

  • Ruderman, D.L. 1994. The statistics of natural images. Network: Comp. Neural, 5(4):517–548.

    Article  MATH  Google Scholar 

  • Scharr, H. 2004. Optimal filters for extended optical flow. In First International Workshop on Complex Motion, vol. 3417 of Lect. Notes in Comp. Sci., Springer.

  • Scharr, H. and Spies, H. 2005. Accurate optical flow in noisy image sequences using flow adapted anisotropic diffusion. Signal Processing: Image Communication, 20(6):537–553.

    Article  Google Scholar 

  • Simoncelli, E.P., Adelson, E.H., and Heeger, D.J. 1991. Probability distributions of optical flow. In IEEE Conf. on Comp. Vis. and Pat. Recog. (CVPR), pp. 310–315.

  • Srivastava, A., Lee, A.B., Simoncelli, E.P., and Zhu, S.-C. 2003. On advances in statistical modeling of natural images. J. Math. Imaging Vision, 18(1):17–33.

    Article  MATH  MathSciNet  Google Scholar 

  • Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., and Rother, C. 2006. A comparative study of energy minimization methods for Markov random fields. In Eur. Conf. on Comp. Vis. (ECCV), A. Leonardis, H. Bischof, and A. Prinz (eds.), vol. 3952 of Lect. Notes in Comp. Sci., pp. 16–29.

  • Teh, Y.W., Welling, M., Osindero, S., and Hinton, G.E. 2003. Energy-based models for sparse overcomplete representations. J. Mach. Learn. Res., 4(Dec.):1235–1260.

    Article  MathSciNet  Google Scholar 

  • Torralba, A. 2003. Contextual priming for object detection. Int. J. Comput. Vision, 53(2):169–191.

    Article  Google Scholar 

  • Torralba, A. and Oliva, A. 2003. Statistics of natural image categories. Network: Comp. Neural, 14(2):391–412.

    Article  Google Scholar 

  • van Harteren, J.H. and Ruderman, D.L. 1998. Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. J. Roy. Stat. Soc. B, 265(1412):2315–2320.

    Google Scholar 

  • Weickert, J. and Schnörr, C. 2001. Variational optic flow computation with a spatio-temporal smoothness constraint. J. Math. Imaging Vision, 14(3):245–255.

    Article  MATH  Google Scholar 

  • Weiss, Y. and Adelson, E.H. 1998. Slow and smooth: A Bayesian theory for the combination of local motion signals in human vision. Technical Report AI Memo 1624, MIT AI Lab, Cambridge, Massachusetts.

  • Zhu, S.C. and Mumford, D. 1997. Prior learning and Gibbs reaction-diffusion. IEEE Trans. Pattern Anal. Mach. Intell., 19(11):1236–1250.

    Article  Google Scholar 

  • Zhu, S.C., Wu, Y., and Mumford, D. 1998. Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling. Int. J. Comput. Vision, 27(2):107–126.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Brown University, Providence, RI, USA

    Stefan Roth & Michael J. Black

Authors
  1. Stefan Roth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael J. Black
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Roth.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Roth, S., Black, M.J. On the Spatial Statistics of Optical Flow. Int J Comput Vision 74, 33–50 (2007). https://doi.org/10.1007/s11263-006-0016-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-006-0016-x

Keywords

Search

Navigation