Skip to main content
SpringerLink
Log in

A two-level real-time vision machine combining coarse- and fine-grained parallelism

  • Special Issue
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In this paper, we describe a real-time vision machine having a stereo camera as input generating visual information on two different levels of abstraction. The system provides visual low-level and mid-level information in terms of dense stereo and optical flow, egomotion, indicating areas with independently moving objects as well as a condensed geometric description of the scene. The system operates at more than 20 Hz using a hybrid architecture consisting of one dual-GPU card and one quad-core CPU. The different processing stages of visual information have rather different characteristics that in some cases make fine-grained parallelization on a GPU less applicable. However, for most of the stages that are not efficiently implementable on a GPU, a coarse parallelization on multiple CPU-cores is applicable. We show that with such hybrid parallelism, we can achieve a speed up of approximately a factor 90 and a reduction of latency of a factor 26 compared to processing on a single CPU-core. Since the vision machine provides generic visual information it can be used in many contexts. Currently it is used in a driver assistance context as well as in two robotic applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://qt.nokia.com/.

  2. Compiled with ‘-O3 -march=native -m64 -fPIC’ optimization and the g++ compiler version 4.3.3. Using an optimized library for FFT computations when possible.

References

  1. Aarno, D., Sommerfeld, J., Kragic, D., Pugeault, N., Kalkan, S., Wörgötter, F., Kraft, D., Krüger, N.: Early reactive grasping with second order 3d feature relations. In: The IEEE International Conference on Advanced Robotics, Jeju Island, Korea (2007)

  2. Adelson, E.H., Anderson, C.H., Bergen, J.R., Burt, P.J., Ogden, J.M.: Pyramid methods in image processing. RCA Eng. 29(6), 33–41 (1984)

    Google Scholar 

  3. Aloimonos, Y., Shulman, D.: Integration of Visual Modules—An Extension of the Marr Paradigm. Academic Press, London (1989)

    Google Scholar 

  4. Amdahl, G.M.: Validity of the single processor approach to achieving large scale computing capabilities, pp. 79–81 (2000)

  5. Başeski, E., Kraft, D., Krüger, N.: Road interpretation for driver assistance based on an early cognitive vision system. VISAPP (2009)

  6. Baker, Z.K., Gokhale, M.B., Tripp, J.L.: Matched filter computation on FPGA, cell and GPU. Field-Programmable Custom Computing Machines. Annual IEEE Symposium, pp. 207–218 (2007). doi:10.1109/FCCM.2007.52

  7. Bouguet, J.Y.: Camera Calibration Toolbox for Matlab (2008). http://www.vision.caltech.edu/bouguetj/calib_doc/

  8. Bugge. H.: An evaluation of Intel’s core i7 architecture using a comparative approach. Comput. Sci. Res. Dev. 23(3–4), 203–209 (2009)

    Article  Google Scholar 

  9. Che, S., Li, J., Sheaffer, J., Skadron, K., Lach, J.: Accelerating compute-Intensive applications with GPUs and FPGAs. In: Application Specific Processors, 2008. SASP 2008. Symposium, pp 101–107 (2008). doi:10.1109/SASP.2008.4570793

  10. Daugman, J.: Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by 2D visual cortical filters. J. Opt. Soc. Am. 2(7), 1160–1169 (1985)

    Article  Google Scholar 

  11. Detry, R., Pugeault, N., Piater, J.: A probabilistic framework for 3D visual object representation. IEEE Trans. Pattern Anal. Mach. Intell. 31(10), 1790–1803 (2009)

    Article  Google Scholar 

  12. ECOVISION: Artificial visual systems based on early-cognitive cortical processing (EU-Project) (2001–2003). http://www.pspcdibeunigeit/ecovision/projecthtml

  13. Felsberg, M., Sommer, G.: The monogenic signal. IEEE Trans. Signal Process. 49(12), 3136–3144 (2001)

    Article  MathSciNet  Google Scholar 

  14. Felsberg, M., Kalkan, S., Krüger, N.: Continuous dimensionality characterization of image structures. Image Vis. Comput. 27, 628–636 (2009)

    Article  Google Scholar 

  15. Fleet, D., Jepson, A.: Computation of component image velocity from local phase information. Int. J. Comput. Vis. 5, 77–104 (1990)

    Article  Google Scholar 

  16. Gautama, T., Van Hulle, M.: A phase-based approach to the estimation of the optical flow field using spatial filtering. IEEE Trans. Neural Netw. 13(5), 1127–1136 (2002)

    Article  Google Scholar 

  17. Granlund, G.: In search of a general picture processing operator. Comput. Graph. Image Process. 8, 155–173 (1978)

    Article  Google Scholar 

  18. Granlund, G.: The complexity of vision. Signal Process. 74 (1999)

  19. Granlund, G.H., Knutsson, H.: Signal Processing for Computer Vision. Kluwer, Dordrecht (1995)

    Google Scholar 

  20. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

  21. Hubel, D., Wiesel, T.: Anatomical demonstration of columns in the monkey striate cortex. Nature 221, 747–750 (1969)

    Article  Google Scholar 

  22. Intel®: CoreTM i7 Desktop Processor, Product Brief (2009)

  23. Jensen, L.B.W., Kjær-Nielsen, A., Alonso, J.D., Ros, E., Krüger, N.: A hybrid fpga/coarse parallel processing architecture for multi-modal visual feature descriptors. In: RECONFIG ’08: Proceedings of the 2008 International Conference on Reconfigurable Computing and FPGAs, pp. 241–246 (2008)

  24. Jessen, J.B.: Real time sparse and dense stereo in an early cognitive vision system using cuda. Master’s thesis. The Cognitive Vision Group, Maersk Institute, University of Southern Denmark (2009). http://www.mip.sdu.dk/covig/publications/JessenMaster.pdf

  25. Kjær-Nielsen, A., Jensen, L.B.W., Sørensen, A.S., Krüger, N.: A real-time embedded system for stereo vision preprocessing using an fpga. In: RECONFIG ’08: Proceedings of the 2008 International Conference on Reconfigurable Computing and FPGAs, pp. 37–42 (2008)

  26. Kraft, D., Pugeault, N., Başeski, E., Popović, M., Kragic, D., Kalkan, S., Wörgötter, F., Krüger, N.: Birth of the Object: Detection of Objectness and Extraction of Object Shape through Object Action Complexes. Special Issue on “Cognitive Humanoid Robots” of the International Journal of Humanoid Robotics, vol. 5, pp. 247–265 (2009)

  27. Krüger, N., Felsberg, M.: An explicit and compact coding of geometric and structural information applied to stereo matching. Pattern Recognit. Lett. 25(8), 849–863 (2004)

    Article  Google Scholar 

  28. Krüger, N., Lappe, M., Wörgötter, F.: (2004) Biologically motivated multi-modal processing of visual primitives. The interdisciplinary. J. Artifi. Intell. Simul. Behav. 1(5), 417–428

    Google Scholar 

  29. Lades, M., Vorbrüggen, J., Buhmann, J., Lange, J., von der Malsburg, C., Würtz, R., Konen, W.: Distortion invariant object recognition in the dynamik link architecture. IEEE Trans. Comput. 42(3), 300–311 (1993)

    Article  Google Scholar 

  30. Longuet-Higgins, H.C., Prazdny, K.: The interpretation of a moving retinal image. Proc. R. Soc. Lond. Ser. B Biol. Sci. 208, 385–397 (1980)

    Article  Google Scholar 

  31. Malvar, H., He L., Cutler, R.: High-quality linear interpolation for demosaicing of Bayer-patterned color images. In: Acoustics, Speech, and Signal Processing, 2004. Proceedings. (ICASSP’04). IEEE International Conference on, vol 3, pp iii–485–8 (2004). doi:10.1109/ICASSP.2004.1326587

  32. Marr, D.: Vision. Freeman, San Francisco (1982)

  33. Mosteller, F., Tukey, J.: Data Analysis and Regression: A Second Course in Statistics. Addison-Wesley Reading (1977)

    Google Scholar 

  34. Pauwels, K.: Computational modeling of visual attention: neuronal response modulation in the thalamocortical complex and saliency-based detection of independent motion. PhD thesis, K.U.Leuven (2008)

  35. Pauwels, K., Van Hulle, M.: Realtime phase-based optical flow on the GPU. In: IEEE Conference on Computer Vision and Pattern Recognition, Workshop on Computer Vision on the GPU (2008)

  36. Pauwels, K., Krüger, N., Lappe, M., Wörgötter, F., Van Hulle, M.: A cortical architecture on parallel hardware for motion processing in real-time. J. Vis. (2010, submitted)

  37. Pollen, D., Ronner, S.: Phase-relationships between adjacent simple cells in the visual cortex. Science 212(4501), 1409–1411 (1981)

    Article  Google Scholar 

  38. Pugeault, N.: Early Cognitive Vision: Feedback Mechanisms for the Disambiguation of Early Visual Representation. VDM Verlag Dr. Müller, Germany (2008)

  39. Pugeault, N., Wörgötter, F., Krüger, N.: Accumulated visual representation for cognitive vision. In Proceedings of the British Machine Vision Conference (BMVC) (2008)

  40. Sabatini, S., Gastaldi, G., Solari, F., Diaz, J., Ros, E., Pauwels, K., Van Hulle, M., Pugeault, N., Krüger, N.: Compact and accurate early vision processing in the harmonic space. In: International Conference on Computer Vision Theory and Applications, Barcelona, pp. 213–220 (2007)

  41. Satish, N., Harris, M., Garland, M.: Designing efficient sorting algorithms for manycore GPUs. In: Parallel & Distributed Processing, 2009. IPDPS 2009. IEEE International Symposium, pp. 1–10 (2009). doi:10.1109/IPDPS.2009.5161005

  42. Schiele, B., Crowley, J.: Probabilistic object recognition using multidimensional receptive field histograms. Adv. Neural Info. Process. Syst. 8, 865–871 (1996)

    Google Scholar 

  43. Slama, C.C. (ed.): Manual of Photogrammetry. American Society of Photo (1980)

  44. Thompson, W., Pong, T.: Detecting moving-objects. Int. J. Comput. Vis. 4, 39–57 (1990)

    Article  Google Scholar 

  45. Tröger, P.: The Multi-Core Era—Trends and Challenges. CoRR abs/0810.5439 (2008)

  46. Vernon, D.: The space of cognitive vision. In: Cognitive Vision Systems, Part I: Foundations of Cognitive Vision Systems. LNCS, vol. 3948 (2006)

  47. Wilkinson, B.: Computer Architecture (2nd ed.): Design and Performance. Prentice-Hall, Inc., Upper Saddle River (1996)

    Google Scholar 

  48. Wörgötter, F., Krüger, N., Pugeault, N., Calow, D., Lappe, M., Pauwels, K., Hulle, M.V., Tan, S., Johnston, A.: Early cognitive vision: Using gestalt-laws for task-dependent, active image-processing. Nat. Comput. 3(3), 293–321 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  49. Zetzsche, C., Krieger, G.: Nonlinear mechanisms and higher-order statistics in biologial vision and electronic image processing: review and perspectives. J. Electron. Imaging 10(1), 56–99 (2001)

    Article  Google Scholar 

  50. Zhang, T., Tomasi, C.: On the consistency of instantaneous rigid motion estimation. Int. J. Comput. Vis. 46, 51–79 (2002)

    Article  MATH  Google Scholar 

Download references

Acknowledgments

This work was supported by the European Commission FP6 Project DRIVSCO (IST-016276-2) and the Danish project Robo-Packman.

Author information

Authors and Affiliations

  1. The Maersk Mc-Kinney Moller Institute, University of Southern Denmark, Campusvej 55, 5230, Odense M, Denmark

    Lars Baunegaard With Jensen, Anders Kjær-Nielsen, Jeppe Barsøe Jessen & Norbert Krüger

  2. Laboratorium voor Neuro- en Psychofysiologie, K.U.Leuven, O&N 2 Herestraat 49, bus 1021, 3000, Leuven, Belgium

    Karl Pauwels & Marc Van Hulle

Authors
  1. Lars Baunegaard With Jensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anders Kjær-Nielsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karl Pauwels
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeppe Barsøe Jessen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marc Van Hulle
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Norbert Krüger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lars Baunegaard With Jensen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jensen, L.B.W., Kjær-Nielsen, A., Pauwels, K. et al. A two-level real-time vision machine combining coarse- and fine-grained parallelism. J Real-Time Image Proc 5, 291–304 (2010). https://doi.org/10.1007/s11554-010-0159-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-010-0159-4

Keywords

Search

Navigation