Skip to main content

Advertisement

Springer Nature Link
Log in

Surface Prediction for Spatial Augmented Reality Applications

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

In spatial augmented reality applications, incorrect projection mapping may occur when projecting images onto moving non-rigid surfaces. This may detract from the user experience, as the image may not be perceived as originally intended. This is especially apparent when using low-cost projectors and cameras or when surfaces are moving quickly. In this paper, an algorithm is developed which predicts the motion of a non-rigid surface, so that when an image is being projected onto the surface, the projection “matches” the surface shape, while using low-cost equipment. Using an interconnected mass–spring–damper system to model the surface, the surface position is predicted using a Kalman filter-based algorithm, which also compensates for the processing delays and fast-moving surfaces. To accurately model real-world materials, the mass–spring system parameters are found using a system identification approach. When the prediction algorithm is implemented experimentally, in real-time, the results show convergent results in the sense that the surface predictions converge to the measured position of a non-rigid surface. The error results show that the algorithm is both accurate and robust and can currently be applied in spatial augmented reality applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  • Anderson BDO, Moore JB (2012) Optimal filtering. Courier Corporation, North Chelmsford

    MATH  Google Scholar 

  • Arasaratnam I (2009) Cubature Kalman filtering theory & applications. PhD thesis, McMaster University

  • Bermano AH, Billeter M, Iwai D, Grundhöfer A (2017) Makeup lamps: live augmentation of human faces via projection. In: Computer graphics forum, volume 36. Wiley Online Library, pp 311–323

  • Bradley D, Popa T, Sheffer A, Heidrich W, Boubekeur T (2008) Markerless garment capture. In: ACM transactions on graphics (TOG), volume 27. ACM, p 99

  • Bridson R, Marino S, Fedkiw R (2003) Simulation of clothing with folds and wrinkles. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation. Eurographics Association, pp 28–36

  • Chen Y, Rui Y, Huang TS et al (2006) Multicue hmm-ukf for real-time contour tracking. IEEE Trans Pattern Anal Mach Intell 28(9):1525

    Article  Google Scholar 

  • Choi K-J, Ko H-S (2005) Stable but responsive cloth. In: ACM SIGGRAPH 2005 courses, p 1. ACM

  • Eberhardt B, Weber A, Strasser W (1996) A fast, flexible, particle-system model for cloth draping. IEEE Comput Graph Appl 16(5):52–59

    Article  Google Scholar 

  • Gibson SFF, Mirtich B (1997) A survey of deformable modeling in computer graphics. Technical report, Citeseer

  • Gillette R, Peters C, Vining N, Edwards E, Sheffer A (2015) Real-time dynamic wrinkling of coarse animated cloth. In: Proceedings of the 14th ACM SIGGRAPH/eurographics symposium on computer animation, pp 17–26

  • Gomes Adam Daniel (2016) Prediction for projection on time-varying surfaces. Master’s thesis, University of Waterloo

  • Granville V, Krivánek M, Rasson J-P (1994) Simulated annealing: a proof of convergence. IEEE Trans Pattern Anal Mach Intell 16(6):652–656

    Article  Google Scholar 

  • Grinspun E, Hirani AN, Desbrun M, Schröder P (2003) Discrete shells. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on computer animation. Eurographics Association, pp 62–67

  • Hilsmann A, Schneider DC, Eisert P (2010) Realistic cloth augmentation in single view video under occlusions. Comput Graph 34(5):567–574

    Article  Google Scholar 

  • Kavan L, Gerszewski D, Bargteil AW, Sloan P-P (2011) Physics-inspired upsampling for cloth simulation in games. In: ACM SIGGRAPH 2011 papers, pp 1–10

  • Kocev B, Ritter F, Linsen L (2014) Projector-based surgeon-computer interaction on deformable surfaces. Int J Comput Assist Radiol Surg 9(2):301–312

    Article  Google Scholar 

  • Livolsi B (2015) Input lag and projectors for gaming: what it is and why you should care. Projector Central. [Online]. http://www.projectorcentral.com/projector-input-lag.htm

  • Lloyd B, Székely G, Harders M (2007) Identification of spring parameters for deformable object simulation. IEEE Trans Visual Comput Graph 13(5):1081–1094

    Article  Google Scholar 

  • Narita G, Watanabe Y, Ishikawa M (2015) Dynamic projection mapping onto a deformable object with occlusion based on high-speed tracking of dot marker array. In: Proceedings of the 21st ACM symposium on virtual reality software and technology. ACM, pp 149–152

  • Nealen A, Müller M, Keiser R, Boxerman E, Carlson M (2006) Physically based deformable models in computer graphics. In: Computer graphics forum, volume 25. Wiley Online Library, pp 809–836

  • Piper B, Ratti C, Ishii H (2002) Illuminating clay: a 3-d tangible interface for landscape analysis. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, pp 355–362

  • Natural Point (2011) Optitrack. Natural Point, Inc.,[Online]. urlhttp://www.naturalpoint. com/optitrack/

  • Provot X (1995) Deformation constraints in a mass-spring model to describe rigid cloth behaviour. In: Graphics interface. Canadian Information Processing Society, pp 147–147

  • Punpongsanon P, Iwai D, Sato K (2015) Projection-based visualization of tangential deformation of nonrigid surface by deformation estimation using infrared texture. Virtual Real 19(1):45–56

    Article  Google Scholar 

  • Sakamaki S, Hashimoto N (2013) Time-delay compensation for dynamic projection mapping. In: SIGGRAPH Asia 2013 Posters, pp 1–1

  • Steimle J, Jordt A, Maes P (2013) Flexpad: highly flexible bending interactions for projected handheld displays. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 237–246

  • Teschner M, Heidelberger B, Muller M, Gross M (2004) A versatile and robust model for geometrically complex deformable solids. In: Computer graphics international, 2004. Proceedings. IEEE, pp 312–319

  • Varol A, Salzmann M, Tola E, Fua P (2009) Template-free monocular reconstruction of deformable surfaces. In 2009 IEEE 12th international conference on computer vision. IEEE, pp 1811–1818

Download references

Acknowledgements

We would like to acknowledge Madeleine Wang for her help in running experiments, Cong Yue for her work with the CKF, and Yousef Sawires for moral support.

Author information

Authors and Affiliations

  1. University of Waterloo, Waterloo, Canada

    Adam Gomes, Keegan Fernandes & David Wang

Authors
  1. Adam Gomes
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Keegan Fernandes
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. David Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Adam Gomes.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1 (MP4 42055 kb)

Supplementary material 2 (MP4 43117 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gomes, A., Fernandes, K. & Wang, D. Surface Prediction for Spatial Augmented Reality Applications. Virtual Reality 25, 761–771 (2021). https://doi.org/10.1007/s10055-020-00490-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-020-00490-2

Keywords