Skip to main content
SpringerLink
Log in

Variational Optic Flow Computation with a Spatio-Temporal Smoothness Constraint

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Nonquadratic variational regularization is a well-known and powerful approach for the discontinuity-preserving computation of optic flow. In the present paper, we consider an extension of flow-driven spatial smoothness terms to spatio-temporal regularizers. Our method leads to a rotationally invariant and time symmetric convex optimization problem. It has a unique minimum that can be found in a stable way by standard algorithms such as gradient descent. Since the convexity guarantees global convergence, the result does not depend on the flow initialization. Two iterative algorithms are presented that are not difficult to implement. Qualitative and quantitative results for synthetic and real-world scenes show that our spatio-temporal approach (i) improves optic flow fields significantly, (ii) smoothes out background noise efficiently, and (iii) preserves true motion boundaries. The computational costs are only 50% higher than for a pure spatial approach applied to all subsequent image pairs of the sequence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. L. Alvarez, J. Esclarín, M. Lefébure, and J. Sánchez, “A PDE model for computing the optical flow,” in Proc. XVI Congreso de Ecuaciones Diferenciales y Aplicaciones (C.E.D.Y.A. XVI, Las Palmas de Gran Canaria, Sept. 21–24, 1999), pp. 1349–1356, 1999.

  2. L. Alvarez, J. Weickert, and J. Sánchez, “Reliable estimation of dense optical flow fields with large displacements,” Int. J. Comput. Vision, Vol. 39, pp. 41–56, 2000.

    Google Scholar 

  3. J. Arnspang, “Notes on local determination of smooth optic flow and the translational property of first order optic flow,” Report DIKU-88/1, Dept. of Computer Science, University of Copenhagen, Denmark, 1988.

    Google Scholar 

  4. G. Aubert, R. Deriche, and P. Kornprobst, “Computing optical flow via variational techniques,” SIAM J. Appl. Math., Vol. 60, pp. 156–182, 1999.

    Google Scholar 

  5. J.L. Barron, D.J. Fleet, and S.S. Beauchemin, “Performance of optical flow techniques,” Int. J. Comput. Vision, Vol. 12, pp. 43–77, 1994.

    Google Scholar 

  6. M.J. Black, “Recursive non-linear estimation of discontinuous flow fields,” in Computer Vision—ECCV '94, Volume I, Lecture Notes in Computer Science, Vol. 800, J.-O. Eklundh (Ed.), Springer: Berlin, 1994, pp. 138–145.

    Google Scholar 

  7. M.J. Black and P. Anandan, “Robust dynamic motion estimation over time,” in Proc. IEEE Comp. Soc. Conf. on Computer Vision and Pattern Recognition (CVPR '91, Maui, June 3–6, 1991), IEEE Computer Society Press: Los Alamitos, 1991, pp. 292–302.

    Google Scholar 

  8. A. Blake and A. Zisserman, Visual Reconstruction, MIT Press: Cambridge, Mass., 1987.

    Google Scholar 

  9. P. Charbonnier, “Reconstruction d'image: Régularisation avec prise en compte des discontinuités,” Ph.D. Thesis, Laboratoire Informatique, Signaux et Systémes de Sophia Antipolis, Université Nice—Sophia Antipolis, France, 1994.

    Google Scholar 

  10. P. Charbonnier, L. Blanc-Féraud, G. Aubert, and M. Barlaud, “Deterministic edge-preserving regularization in computed imaging,” IEEE Trans. Image Proc., Vol. 6, pp. 298–311, 1997.

    Google Scholar 

  11. I. Cohen, “Nonlinear variational method for optical flowcomputation,” in Proc. Eighth Scandinavian Conf. on Image Analysis (SCIA '93, Tromsø, May 25–28, 1993), Vol. 1, 1993, pp. 523–530.

    Google Scholar 

  12. N. Cornelius and T. Kanade, “Adapting optical flow to measure object motion in reflectance and X-ray image sequences,” in Proc. ACM SIGRAPH/SIGART Interdisciplinary Workshop on Motion: Representation and Perception (Toronto, April 1983), 1983, pp. 50–58.

  13. G. Demoment, “Image reconstruction and restoration: Overview of common estimation structures and problems,” IEEE Trans. Acoustics, Speech, and Signal Processing, Vol. 37, pp. 2024–2036, 1989.

    Google Scholar 

  14. R. Deriche, P. Kornprobst, and G. Aubert, “Optical-flow estimation while preserving its discontinuities: A variational approach,” in Proc. Second Asian Conf. Computer Vision, (ACCV'95, Singapore, December 5–8, 1995), Vol. 2, 1995, pp. 290–295.

    Google Scholar 

  15. D.J. Fleet and A.D. Jepson, “Computation of component image velocity from local phase information,” Int. J. Comput. Vision, Vol. 5, pp. 77–104, 1990.

    Google Scholar 

  16. B. Galvin, B. McCane, K. Novins, D. Mason, and S. Mills, “Recovering motion fields: An analysis of eight optical flow algorithms,” in Proc. 1998 British Machine Vision Conference (BMVC '98, Southampton, September 14–17, 1998).

  17. D. Geman and G. Reynolds, “Constrained restoration and the recovery of discontinuities,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 14, pp. 367–383, 1992.

    Google Scholar 

  18. G. Gerig, O. Kübler, R. Kikinis, and F.A. Jolesz, “Nonlinear anisotropic filtering of MRI data,” IEEE Trans. Medical Imaging, Vol. 11, pp. 221–232, 1992.

    Google Scholar 

  19. B.M. ter Haar Romeny (Ed.), Geometry-Driven Diffusion in Computer Vision, Kluwer: Dordrecht, 1994.

    Google Scholar 

  20. B. Horn and B. Schunck, “Determining optical flow,” Artif. Intell., Vol. 17, pp. 185–203, 1981.

    Google Scholar 

  21. A. Kumar, A.R. Tannenbaum, and G.J. Balas, “Optic flow: A curve evolution approach,” IEEE Trans. Image Proc., Vol. 5, pp. 598–610, 1996.

    Google Scholar 

  22. P.J.M. van Laarhoven and E.H.L. Aarts, Simulated Annealing: Theory and Applications, Reidel: Dordrecht, 1988.

    Google Scholar 

  23. B. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. Seventh Int. Joint Conf. on Artificial Intelligence (IJCAI '81, Vancouver, August 1981), 1981, pp. 674–679.

  24. A. Mitiche and P. Bouthemy, “Computation and analysis of image motion: A synopsis of current problems and methods,” Int. J. Comput. Vision, Vol. 19, pp. 29–55, 1996.

    Google Scholar 

  25. C.R. Moloney and E. Dubois, “Estimation of motion fields from image sequences with illumination variation,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP-93, Toronto, May 1993), Vol. 4, 1993, pp. 2425–2428.

    Google Scholar 

  26. K.W. Morton and D.F. Mayers, Numerical Solution of Partial Differential Equations, Cambridge University Press: Cambridge, 1994.

    Google Scholar 

  27. D.W. Murray and B.F. Buxton, “Scene segmentation from visual motion using global optimization,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 9, pp. 220–228, 1987.

    Google Scholar 

  28. H.H. Nagel, “Constraints for the estimation of displacement vector fields from image sequences,” in Proc. Eighth Int. Joint Conf. on Artificial Intelligence (IJCAI '83, Karlsruhe, August 8–12, 1983), 1983, pp. 945–951.

  29. H.H. Nagel, “Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow,” in Computer Vision—ECCV '90, Lecture Notes in Computer Science, Vol. 427, O. Faugeras (Ed.), Springer: Berlin 1990, pp. 139–148.

    Google Scholar 

  30. S. Negahdaripour and C.M. Yu, “A generalized brightness change model for computing optical flow,” in Proc. Fourth International Conference on Computer Vision (ICCV '93, Berlin, May 1993), 1993, pp. 2–11.

  31. P. Perona and J. Malik, “Scale space and edge detection using anisotropic diffusion,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 12, pp. 629–639, 1990.

    Google Scholar 

  32. M. Proesmans, L. Van Gool, E. Pauwels, and A. Oosterlinck, “Determination of optical flow and its discontinuities using non-linear diffusion,” Computer Vision—ECCV '94, Volume II, Lecture Notes in Computer Science, Vol. 801, J.-O. Eklundh (Ed.), Springer: Berlin, pp. 295–304, 1994.

    Google Scholar 

  33. L.I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D, Vol. 60, pp. 259–268, 1992.

    Google Scholar 

  34. C. Schnörr, “Determining optical flow for irregular domains by minimizing quadratic functionals of a certain class,” Int. J. Comput. Vision, Vol. 6, pp. 25–38, 1991.

    Google Scholar 

  35. C. Schnörr, “Unique reconstruction of piecewise smooth images by minimizing strictly convex nonquadratic functionals,” J. Math. Imag. Vision, Vol. 4, pp. 189–198, 1994.

    Google Scholar 

  36. C. Schnörr, “Segmentation of visual motion by minimizing convex non-quadratic functionals,” in Proc. 12th Int. Conf. Pattern Recognition (ICPR 12, Jerusalem, Oct. 9–13, 1994), Vol. A, IEEE Computer Society Press: Los Alamitos, 1994, pp. 661–663.

    Google Scholar 

  37. M.A. Snyder, “On the mathematical foundations of smoothness constraints for the determination of optical flow and for surface reconstruction,” IEEE Trans.Pattern Anal.Mach. Intell., Vol. 13, pp. 1105–1114, 1991.

    Google Scholar 

  38. R.L. Stevenson, B.E. Schmitz, and E.J. Delp, “Discontinuity preserving regularization of inverse visual problems,” IEEE Trans. Systems, Man and Cybernetics, Vol. 24, pp. 455–469, 1994.

    Google Scholar 

  39. C. Stiller and J. Konrad, “Estimating motion in image sequences,” IEEE Signal Proc. Magazine, Vol. 16, pp. 70–91, 1999.

    Google Scholar 

  40. A. Thompson and D.M. Titterington, “A study of choosing the smoothing parameters in image reconstruction by regularization,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 13, pp. 326–339, 1991.

    Google Scholar 

  41. S. Uras, F. Girosi, A. Verri, and V. Torre, “A computational approach to motion perception,” Biol. Cybern., Vol. 60, pp. 79–87, 1988.

    Google Scholar 

  42. J. Weber and J. Malik, “Robust computation of optical flow in a multi-scale differential framework,” Int. J. Comput. Vision, Vol. 14, pp. 67–81, 1995.

    Google Scholar 

  43. J.Weickert, Anisotropic Diffusion in Image Processing, Teubner-Verlag: Stuttgart, 1998.

    Google Scholar 

  44. J. Weickert, “On discontinuity-preserving optic flow,” in Proc. Computer Vision and Mobile Robotics Workshop (CVMR '98, Santorini, Sept. 17–18, 1998), S. Orphanoudakis, P. Trahanias, J. Crowley, and N. Katevas (Eds.), 1998, pp. 115–122.

  45. J. Weickert, B.M. ter Haar Romeny, and M.A. Viergever, “Efficient and reliable schemes for nonlinear diffusion filtering,” IEEE Trans. Image Proc., Vol. 7, pp. 398–410, 1998.

    Google Scholar 

  46. J. Weickert, J. Heers, C. Schnörr, K.J. Zuiderveld, O. Scherzer, and H.S. Stiehl, “Fast parallel algorithms for a broad class of nonlinear variational diffusion approaches,” in Real-Time Imaging, Vol. 7, pp. 31–45, 2001.

    Google Scholar 

  47. J. Weickert and C. Schnörr, Räumlich-zeitliche Berechnung des optischen Flusses mit nichtlinearen flußtiabhängigen Glattheitstermen, Mustererkennung 1999, W. Förstner, J.M. Buhmann, A. Faber, and P. Faber (Eds.), Springer: Berlin, 1999, pp. 317–324.

    Google Scholar 

  48. J. Weickert and C. Schnörr, “A theoretical framework for convex regularizers in PDE-based computation of image motion,” Int. J. Comput. Vision. in press.

Download references

Author information

Authors and Affiliations

  1. Computer Vision, Graphics, and Pattern Recognition Group, Department of Mathematics and Computer Science, University of Mannheim, 68131, Mannheim, Germany

    Joachim Weickert & Christoph Schnörr

Authors
  1. Joachim Weickert
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christoph Schnörr
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Weickert, J., Schnörr, C. Variational Optic Flow Computation with a Spatio-Temporal Smoothness Constraint. Journal of Mathematical Imaging and Vision 14, 245–255 (2001). https://doi.org/10.1023/A:1011286029287

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1011286029287

Search

Navigation