Skip to main content
Springer Nature Link
Log in

Variational optic flow on the Sony PlayStation 3

Accurate dense flow fields for real-time applications

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

While modern variational methods for optic flow computation offer dense flow fields and highly accurate results, their computational complexity has prevented their use in many real-time applications. With cheap modern parallel hardware such as the Cell Processor of the Sony PlayStation 3, new possibilities arise. For a linear and a nonlinear variant of the popular combined local-global method, we present specific algorithms on this architecture that are tailored towards real-time performance. They are based on bidirectional full multigrid methods with a full approximation scheme in the nonlinear setting. Their parallel design on the Cell hardware uses a temporal instead of a spatial decomposition, and processes operations in a vector-based manner. Memory latencies are reduced by a locality-preserving cache management and optimised access patterns. For images of size 316 × 252 pixels, we obtain dense flow fields for up to 210 frames per second.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Amiaz, T., Kiryati, N.: Piecewise-smooth dense optical flow via level sets. Int. J. Comput. Vis. 68(2), 111–124 (2006)

    Article  Google Scholar 

  2. Arribas, P.C., Maciá, F.M.H.: FPGA implementation of the Horn & Shunk optical flow algorithm for motion detection in real time images. In: Proceedings of XIII Design of Circuits and Integrated Systems Conference, Madrid, Spain, pp. 616–621 (1998)

  3. Bartlett, J.: Programming high-performance applications on the Cell BE processor, Part 1: An introduction to Linux on the Playstation 3. Tech. Rep. pa-linuxps3-1, IBM developerWorks. http://www.ibm.com/developerworks/linux/library/pa-linuxps3-1/ (2007). Retrieved 09-03-30

  4. Benthin, C., Wald, I., Scherbaum, M., Friedrich, H.: Ray tracing on the CELL processor. In: Proceedings of 2006 IEEE Symposium on Interactive Ray Tracing. IEEE Computer Society Press, Salt Lake City, UT, pp. 25–23 (2006)

  5. Bornemann, F., Deuflhard, P.: The cascadic multigrid method for elliptic problems. Numer. Math. 75, 135–152 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  6. Brandt, A.: Multi-level adaptive solutions to boundary-value problems. Math. Comput. 31(138), 333–390 (1977)

    MATH  MathSciNet  Google Scholar 

  7. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optic flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) Computer Vision—ECCV 2004. Lecture Notes in Computer Science, vol. 3024, pp. 25–36. Springer, Berlin (2004)

    Google Scholar 

  8. Bruhn, A., Weickert, J., Feddern, C., Kohlberger, T., Schnörr, C.: Variational optical flow computation in real-time. IEEE Trans. Image Process. 14(5), 608–615 (2005)

    Article  MathSciNet  Google Scholar 

  9. Bruhn, A., Weickert, J., Kohlberger, T., Schnörr, C.: Discontinuity-preserving computation of variational optic flow in real-time. In: Kimmel, R., Sochen, N., Weickert, J. (eds.) Scale-space and PDE Methods in Computer Vision. Lecture Notes in Computer Science, vol. 3459, pp. 279–290. Springer, Berlin (2005)

    Chapter  Google Scholar 

  10. Bruhn, A., Weickert, J., Schnörr, C.: Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. Int. J. Comput. Vis. 61(3), 211–231 (2005)

    Article  Google Scholar 

  11. Bruhn, A., Weickert, J., Kohlberger, T., Schnörr, C.: A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. Int. J. Comput. Vis. 70(3), 257–277 (2006)

    Article  Google Scholar 

  12. Buttari, A., Luszczek, P., Dongarra, J., Bosilca, G.: A rough guide to scientific computing on the Playstation 3. Tech. Rep. UT-CS-07-595. Innovative Computing Laboratory, University of Tennessee, Knoxville, TN (2007)

  13. Camus, T.: Real-time quantized optical flow. Real-Time Imag. 3, 71–86 (1997)

    Article  Google Scholar 

  14. Charbonnier, P., Blanc-Féraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of 1994 IEEE International Conference on Image Processing, vol. 2. IEEE Computer Society Press, Austin, TX, pp. 168–172 (1994)

  15. El Kalmoun, M., Köstler, H., Rüde, U.: 3D optical flow computation using a parallel variational multigrid scheme with application to cardiac C-arm CT motion. Image Vis. Comput. 25(9), 1482–1494 (2007)

    Article  Google Scholar 

  16. Elgersma, M.R., Yuen, D.A., Pratt, S.G.: Adaptive multigrid solution of Stokes’ equation on Cell processor. In: Proceedings of American Geophysical Union Fall Meeting 2006. American Geophysical Union, San Francisco, CA, pp. IN53B-0821 (2006)

  17. Flynn, M.J.: Some computer organizations and their effectiveness. IEEE Trans. Comput. C 21(9), 118–135 (1972)

    Article  MathSciNet  Google Scholar 

  18. Frohn-Schauf, C., Henn, S., Witsch, K.: Nonlinear multigrid methods for total variation denoising. Comput. Vis. Sci. 7(3–4), 199–206 (2004)

    MATH  MathSciNet  Google Scholar 

  19. Gelfand, I.M., Fomin, S.V.: Calculus of Variations. Dover, New York (2000)

    MATH  Google Scholar 

  20. Ghosal, S., Vaněk, P.Č.: Scalable algorithm for discontinuous optical flow estimation. IEEE Trans. Pattern Anal. Mach. Intell. 18(2), 181–194 (1996)

    Google Scholar 

  21. Glazer, F.: Multilevel relaxation in low-level computer vision. In: Rosenfeld, A. (ed.) Multiresolution Image Processing and Analysis. Springer, Berlin, pp. 312–330 (1984)

    Google Scholar 

  22. Grossauer, H., Thoman, P.: GPU-based multigrid: real-time performance in high resolution nonlinear image processing. In: Gasteratos, A., Vincze, M., Tsotsos, J.K. (eds.) Computer Vision Systems. Lecture Notes in Computer Science, vol. 5008, pp. 141–150. Springer, Berlin (2008)

    Google Scholar 

  23. Gwosdek, P., Bruhn, A., Weickert, J.: High performance parallel optical flow algorithms on the Sony Playstation 3. In: Deussen, O., Keim, D., Saupe, D. (eds.) VMV ’08: Proceedings of Vision, Modeling, and Visualization. AKA, Konstanz, Germany, pp. 253–262 (2008)

    Google Scholar 

  24. Hackbusch, W.: Multigrid Methods and Applications. Springer, New York (1985)

    Google Scholar 

  25. Hirai, T., Sasakawa, K., Kuroda, S.: Hardware of optical flow calculation and its application to tracking of moving object. Tech. Rep. 93-CV-81. Special Interest Group on Computer Vision, Information Processing Society of Japan, Tokyo, Japan (in Japanese) (1993)

  26. Horn, B., Schunck, B.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)

    Article  Google Scholar 

  27. Kahle, J.A., Day, M.N., Hofstee, H.P., Johns, C.R., Maeurer, T.R., Shippy, D.: Introduction to the Cell multiprocessor. IBM J. Res. Dev. 49(4), 589–604 (2005)

    Article  Google Scholar 

  28. Kollnig, H., Nagel, H.H., Otte, M.: Association of motion verbs with vehicle movements extracted from dense optical flow fields. In: Eklundh, J.O. (ed.) Computer Vision—ECCV ’94. Lecture Notes in Computer Science, vol. 801, pp. 338–347. Springer, Berlin (1994)

    Google Scholar 

  29. Köstler, H., Stürmer, M., Freundl, C., Rüde, U.: PDE based video compression in real time. Tech. Rep. 07-11. Institut für Informatik. Univ. Erlangen-Nürnberg, Germany (2007)

    Google Scholar 

  30. Lucas, B., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of Seventh International Joint Conference on Artificial Intelligence. Vancouver, Canada, pp. 674–679 (1981)

  31. Mizukami, Y., Tadamura, K.: Optical flow computation on Compute Unified Device Architecture. In: Proceedings of 14th International Conference on Image Analysis and Processing. IEEE Computer Society Press, Modena, Italy, pp. 179–184 (2007)

  32. Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8, 565–593 (1986)

    Article  Google Scholar 

  33. Nesi, P.: A chip for motion estimation. Electron. Imag. 4(2), 12 (1994)

    MathSciNet  Google Scholar 

  34. Nir, T., Bruckstein, A.M., Kimmel, R.: Over-parameterized variational optical flow. Int. J. Comput. Vis. 76(2), 205–216 (2008)

    Article  Google Scholar 

  35. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)

    Article  MATH  Google Scholar 

  36. Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. SIAM, Philadelphia (2003)

    MATH  Google Scholar 

  37. Srinivasan, V., Santhanam, A.K., Srinivasan, M.: Cell Broadband Engine processor DMA engines, Part 2: From an SPE point of view. Tech. Rep. pa-celldmas2. IBM developerWorks. http://www.ibm.com/developerworks/library/pa-celldmas2/ (2006). Retrieved 09-03-30

  38. Trottenberg, U., Oosterlee, C., Schüller, A.: Multigrid. Academic Press, San Diego (2001)

    MATH  Google Scholar 

  39. Wedel, A., Pock, T., Braun, J., Franke, U., Cremers, D.: Duality TV-L 1 flow with fundamental matrix prior. Image Vision and Computing. IEEE Computer Society Press, Auckland, New Zealand (2008)

  40. Wei, Z., Lee, D.J., Nelson, B.E.: FPGA-based real-time optical flow algorithm design and implementation. J. Multimed. 2(5), 38–45 (2007)

    Google Scholar 

  41. Williams, S., Shalf, J., Oliker, L., Kamil, S., Husbands, P., Yelick, K.: The potential of the Cell processor for scientific computing. In: CF ’06: Proceedings of ACM Conference on Computing Frontiers, Ischia, Italy, pp. 9–20 (2006)

  42. Williams, S., Shalf, J., Oliker, L., Kamil, S., Husbands, P., Yelick, K.: Scientific computing kernels on the Cell processor. Int. J. Parallel Program. 35(3), 263–298 (2007)

    Article  Google Scholar 

  43. Young, D.M.: Iterative Solution of Large Linear Systems. Dover, New York (2003)

    MATH  Google Scholar 

  44. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L 1 optical flow. In: Hamprecht F.A., Schnörr, C., Jähne, B. (eds.) Pattern Recognition. Lecture Notes in Computer Science, vol. 4713, pp. 214–223. Springer, Berlin (2007)

  45. Zini, G., Sarti, A., Lamberti, C.: Application of continuum theory and multi-grid methods to motion evaluation from 3D echocardiography. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 44(2), 297–308 (1997)

    Article  Google Scholar 

Download references

Acknowledgments

The authors thank the Cluster of Excellence Multimodal Computing and Interaction for partly funding this work.

Author information

Authors and Affiliations

  1. Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, Bldg. E1.1, 66041, Saarbrücken, Germany

    Pascal Gwosdek, Andrés Bruhn & Joachim Weickert

Authors
  1. Pascal Gwosdek
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Andrés Bruhn
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Joachim Weickert
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Pascal Gwosdek.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gwosdek, P., Bruhn, A. & Weickert, J. Variational optic flow on the Sony PlayStation 3. J Real-Time Image Proc 5, 163–177 (2010). https://doi.org/10.1007/s11554-009-0132-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-009-0132-2

Keywords