Skip to main content
SpringerLink
Log in

Augmenting moving planar surfaces robustly with video projection and direct image alignment

  • SI: Mixed and Augmented Reality
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Augmented reality applications based on video projection, to be effective, must track moving targets and make sure that the display remains aligned even when they move, but the projection can severely alter their appearances to the point where traditional computer vision algorithms fail. Current solutions consider the displayed content as interference and largely depend on channels orthogonal to visible light. They cannot directly align projector images with real-world surfaces, even though this may be the actual goal. We propose instead to model the light emitted by projectors and reflected into cameras and to consider the displayed content as additional information useful for direct alignment. Using a color camera, our implemented software successfully tracks with subpixel accuracy a planar surface of diffuse reflectance properties at an average of eight frames per second on commodity hardware, providing a solid base for future enhancements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Audet S, Okutomi M (2009) A user-friendly method to geometrically calibrate projector-camera systems. In: 2009 IEEE conference on computer vision and pattern recognition (CVPR 2009)—workshops (Procams 2009), IEEE computer society, © 2009 IEEE, pp 47–54

  • Audet S, Okutomi M, Tanaka M (2010) Direct image alignment of projector-camera systems with planar surfaces. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR 2010), IEEE computer society, © 2010 IEEE

  • Baker S, Matthews I (2004) Lucas-Kanade 20 years on: a unifying framework. Int J Comput Vis 56(1):221–255

    Article  Google Scholar 

  • Baker S, Datta A, Kanade T (2006) Parameterizing homographies. Tech. Rep. CMU-RI-TR-06-11, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA

  • Bandyopadhyay D, Raskar R, Fuchs H (2001) Dynamic shader lamps: painting on movable objects. In: 2001 IEEE and ACM international symposium on augmented reality (ISAR 2001), IEEE computer society, p 207

  • Bartoli A (2008) Groupwise geometric and photometric direct image registration. IEEE Trans Pattern Anal Mach Intell 30(12):2098–2108

    Article  Google Scholar 

  • Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. A. K. Peters, Ltd., Natick

    Google Scholar 

  • Bradski G, Kaehler A (2008) Learning OpenCV: computer vision with the OpenCV library. O’Reilly, Cambridge

    Google Scholar 

  • Caspi D, Kiryati N, Shamir J (1998) Range imaging with adaptive color structured light. IEEE Trans Pattern Anal Mach Intell 20(5):470–480

    Article  Google Scholar 

  • Chen X, Yang X, Xiao S, Li M (2008) Color mixing property of a projector-camera system. In: Fifth international workshop on projector-camera systems (Procams 2008), ACM, pp 1–6

  • Hartley R, Zisserman A (2004) Multiple view geometry in computer vision, second edition. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • International Electrotechnical Commission (1999) IEC 61966-2-1 (1999-10-18): multimedia systems and equipment—colour measurement and management—part 2-1: colour management—default RGB colour space—sRGB

  • Johnson T, Fuchs H (2007) Real-time projector tracking on complex geometry using ordinary imagery. In: 2007 IEEE conference on computer vision and pattern recognition (CVPR 2007)—workshops (Procams 2007), IEEE computer society, pp 1–8

  • Leibe B, Starner T, Ribarsky W, Wartell Z, Krum D, Singletary B, Hodges L (2000) The perceptive workbench: towards spontaneous and natural interaction in semi-immersive virtual environments. In: 2000 IEEE virtual reality conference (VR 2000), IEEE computer society, pp 13–20

  • Moritani T, Hiura S, Sato K (2006) Real-time object tracking without feature extraction. In: 2006 IEEE conference on pattern recognition (ICPR 2006), IEEE computer society, Los Alamitos, CA, USA, vol 1, pp 747–750

  • Raskar R, Welch G, Cutts M, Lake A, Stesin L, Fuchs H (1998) The office of the future: a unified approach to image-based modeling and spatially immersive displays. In: 25th international conference on computer graphics and interactive techniques (SIGGRAPH 98). ACM Press, New York, pp 179–188

  • Raskar R, van Baar J, Beardsley P, Willwacher T, Rao S, Forlines C (2003) iLamps: geometrically aware and self-configuring projectors. In: ACM transactions on graphics (SIGGRAPH 2003), ACM, pp 809–818

  • Silveira G, Malis E (2007) Real-time visual tracking under arbitrary illumination changes. In: 2007 IEEE conference on computer vision and pattern recognition (CVPR 2007), IEEE computer society

  • Sturm P (2000) Algorithms for plane-based pose estimation. In: 2000 IEEE conference on computer vision and pattern recognition (CVPR 2000), IEEE computer society, Hilton Head Island, South Carolina, USA, pp 1706–1711

  • Sugimoto S, Okutomi M (2007) A direct and efficient method for piecewise-planar surface reconstruction from stereo images. In: 2007 IEEE conference on computer vision and pattern recognition (CVPR 2007), IEEE computer society

  • Takao N, Shi J, Baker S (2003) Tele-graffiti: a camera-projector based remote sketching system with hand-based user interface and automatic session summarization. Int J Comput Vis 53(2):115–133

    Article  Google Scholar 

  • Triggs B (1998) Autocalibration from planar scenes. In: 5th European conference on computer vision (ECCV ’98), vol I, Springer, Berlin, pp 89–105

  • Wagner D, Schmalstieg D (2007) ARToolKitPlus for pose tracking on mobile devices. In: 12th computer vision winter workshop (CVWW’07), Graz University of Technology, St. Lambrecht, Austria

  • Xiao J, Baker S, Matthews I, Kanade T (2004) Real-time combined 2D+3D active appearance models. In: 2004 IEEE conference on computer vision and pattern recognition (CVPR 2004), IEEE computer society, Washinton, DC, USA, vol 2, pp 535–542

  • Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by a scholarship from the Ministry of Education, Culture, Sports, Science and Technology (MEXT) of the Japanese Government.

Author information

Authors and Affiliations

  1. Tokyo Institute of Technology, 2-12-1 Ookayama, Meguro-ku, Tokyo, Japan

    Samuel Audet, Masatoshi Okutomi & Masayuki Tanaka

Authors
  1. Samuel Audet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Masatoshi Okutomi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Masayuki Tanaka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samuel Audet.

Electronic supplementary material

Below is the link to the electronic supplementary material.

MPG (12816 KB)

MPG (36770 KB)MPG (36770 KB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Audet, S., Okutomi, M. & Tanaka, M. Augmenting moving planar surfaces robustly with video projection and direct image alignment. Virtual Reality 17, 157–168 (2013). https://doi.org/10.1007/s10055-012-0210-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-012-0210-9

Keywords

Search

Navigation