Skip to main content
SpringerLink
Log in

Fast and scalable 3D cyber-physical modeling for high-precision mobile augmented reality systems

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Mobile augmented reality is an emerging technique which allows users to use a mobile device’s camera to capture real-world imagery and view real-world physical objects and their associated cyber-information overlaid on top of imagery of them. One key challenge for mobile augmented reality is the fast and precisely localization of a user in order to determine what is visible in their camera view. Recent advances in Structure-from-Motion (SfM) enable the creation of 3D point clouds of physical objects from an unordered set of photographs taken by commodity digital cameras. The generated 3D point cloud can be used to identify the location and orientation of the camera relative to the point cloud. While this SfM-based approach provides complete pixel-accurate camera pose estimation in 3D without relying on external GPS or geomagnetic sensors, the preparation of initial 3D point cloud typically takes from hours to a day, making it difficult to use in mobile augmented reality applications. Furthermore, creating 3D cyber-information and associating it with the 3D point cloud is also a challenge of using SfM-based approach for mobile augmented reality. To overcome these challenges in 3D point cloud creation and cyber-physical content authoring, the paper presents a new SfM framework that is optimized for mobile augmented reality and rapidly generates a complete 3D point cloud of a target scene up to 28 times faster than prior approaches. Key improvements in the proposed SfM framework stem from the use of (1) state-of-the-art binary feature descriptors, (2) new filtering approach for accurate 3D modeling, (3) optimized point cloud structure for augmented reality, and (4) hardware/software parallelism. The paper also provides a new image-based 3D content authoring method designed specifically for the limited user interfaces of mobile devices. The proposed content authoring method generates 3D cyber-information from a single 2D image and automatically associates it with the 3D point cloud.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Agarwal S, Snavely N, Simon I, Seitz S, Szeliski R (2009) Building rome in a day. In: 2009 IEEE 12th international conference on computer vision, pp 72–79. IEEE

  2. Akula M, Dong S, Kamat V, Ojeda L, Borrell A, Borenstein J (2011) Integration of infrastructure based positioning systems and inertial navigation for ubiquitous context-aware engineering applications. Autom Constr 25(4):640–655

    Google Scholar 

  3. Alahi A, Ortiz R, Vandergheynst P (2012) Freak: fast retina keypoint. In: Proceeding of the 2012 IEEE conference on computer vision and pattern recognition (CVPR), pp 510–517

  4. Allen M, Regenbrecht H, Abbott M (2010) Smart-phone augmented reality for public participation in urban planning. In: Proceeding of the 23rd Australian computer-human interaction conference, pp 11–20

  5. Arth C, Schmalstieg D (2011) Challenges of large-scale augmented reality on smartphones. In: Proceeding of the 10th international symposium on mixed and augmented reality (ISMAR), vol 1

  6. Bae H, Golparvar-Fard M, White J (2011) Enhanced \(\text{HD}^{4}\text{AR}\) (hybrid 4-dimensional augmented reality) for ubiquitous context-aware aec/fm applications. In: Proceeding of the 12th international conference on construction applications of virtual reality (CONVR), pp 253–262

  7. Bae H, Golparvar-Fard M, White J (2013) High-precision and infrastructure-independent mobile augmented reality system for context-aware construction and facility management applications. In: Proceeding of the 2013 ASCE international workshop on computing in civil engineering (IWCCE), pp 637–644

  8. Bae H, Golparvar-Fard M, White J (2013) High-precision vision-based mobile augmented reality system for context-aware architectural, engineering, construction and facility management (aec/fm) applications. Vis Eng 1(3):1–13. doi:10.1186/2213-7459-1-3

    Google Scholar 

  9. Bay H, Ess A, Tuytelaars T, Gool L (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3):346–359

    Article  Google Scholar 

  10. Behzadan A, Kamat V (2007) Georeferenced registration of construction graphics in mobile outdoor augmented reality. J Comput Civil Eng 21(4):247–258

    Article  Google Scholar 

  11. Carozza L, Tingdahl D, Bosché F, Van Gool L (2014) Markerless vision-based augmented reality for urban planning. J Comput Aided Civil Infrastruct Eng. doi:10.1111/j.1467-8667.2012.00798.x

    Google Scholar 

  12. Chen W, Xiong Y, Gao J, Gelfand N, Grzeszczuk R (2007) Efficient extraction of robust image features on mobile devices. In: Proceeding of the 6th IEEE and ACM international symposium on mixed and augmented reality (ISMAR), pp 1–2

  13. Davison A, Reid I, Molton N, Stasse O (2007) Monoslam: real-time single camera slam. IEEE Trans Pattern Anal Mach Intell 29(6):1052–1067

    Article  Google Scholar 

  14. Dong Z, Zhang G, Jia J, Bao H (2009) Keyframe-based real-time camera tracking. In: Proceeding of the 12th IEEE international conference on computer vision (ICCV), pp 1538–1545

  15. Frahm J, Fite-Georgel P, Gallup D, Johnson T, Raguram R, Wu C, Jen Y, Dunn E, Clipp B, Lazebnik S, Pollefeys M (2010) Building Rome on a cloudless day. In: Proceedings of the 11th European conference on Computer vision (ECCV 2010). Springer, Berlin

  16. Golparvar-Fard M, Peña Mora F, Savarese S (2011) Integrated sequential as-built and as-planned representation with d\(^4\)ar tools in support of decision-making tasks in the aec/fm industry. J Constr Eng Manag 137(12):1099–1116

    Article  Google Scholar 

  17. Golparvar-Fard M, Peña-Mora F, Savarese S (2015) Automated progress monitoring using unordered daily construction photographs and IFC-based building information models. J Comput Civil Eng. doi:10.1061/(ASCE)CP.1943-5487.0000205

  18. Gordon I, Lowe D (2006) Toward category-level object recognition. Springer, Berlin

    Google Scholar 

  19. Gotow J, Zienkiewicz K, White J, Schmidt D (2010) Mobile wireless middleware, operating systems, and applications. Springer, Berlin Heidelberg

    Google Scholar 

  20. Hakkarainen M, Woodward C, Billinghurst M (2008) Augmented assembly using a mobile phone. In: Proceeding of 7th IEEE/ACM international symposium on mixed and augmented reality (ISMAR 2008), pp 167–168

  21. Hartley R, Sturm P (1997) Triangulation. Comput Vis Image Underst 68(2):146–157

    Article  Google Scholar 

  22. Hartley R, Zisserman A (2004) Multiple view geometry in computer vision. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  23. Hartmann J, Forouher D, Litza M, Klssendorff JH, Maehle E (2012) Real-time visual slam using fastslam and the microsoft kinect camera. In: Proceeding of the 7th German conference on robotics (ROBOTIK), pp 1–6

  24. Irizarry J, Gheisari M, Williams G, Walker B (2013) Infospot: a mobile augmented reality method for accessing building information through a situation awareness approach. Autom Constr 33:1–6

    Article  Google Scholar 

  25. Irschara A, Zach C, Frahm J, Bischof H (2009) From structure-from-motion point clouds to fast location recognition. In: Proceeding of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2599–2606

  26. Izkara JL, Pérez J, Basogain X, Borro D (2007) Mobile augmented reality, an advanced tool for the construction sector. In: Proceedings of the 24th W78 conference, Maribor, Slovenia, pp 190–202. Citeseer

  27. Khoury H, Kamat V (2009) High-precision identification of contextual information in location-aware engineering applications. Adv Eng Inform 23(4):483–496

    Article  Google Scholar 

  28. Klein G, Murray D (2007) Parallel tracking and mapping for small ar workspaces. In: Proceeding of the 6th IEEE and ACM international symposium on mixed and augmented reality (ISMAR), pp 225–234

  29. Lee T, Höllerer T (2008) Hybrid feature tracking and user interaction for markerless augmented reality. In: Proceeding of the IEEE virtual reality conference (VR), pp 145–152

  30. Leutenegger S, Chli M, Siegwart R (2011) Brisk: binary robust invariant scalable keypoints. In: Proceeding of the 13th IEEE international conference on computer vision (ICCV), pp 2548–2555

  31. Lim H, Sinha S, Cohen M, Uyttendaele M (2012) Real-time image-based 6-dof localization in large-scale environments. In: Proceeding of the 2012 IEEE conference on computer vision and pattern recognition (CVPR), pp 1043–1050

  32. Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  33. Muja M, Lowe D (2009) Fast approximate nearest neighbors with automatic algorithm configuration. In: Proceeding of the international conference on computer vision theory and applications (VISAPP), pp 331–340

  34. Muja M, Lowe D (2012) Fast matching of binary features. In: Proceeding of the 9th IEEE conference on computer and robot vision (CRV), pp 404–410

  35. Nistér D (2004) An efficient solution to the five-point relative pose problem. IEEE Trans Pattern Anal Mach Intell 26(6):756–777

    Article  Google Scholar 

  36. Ojeda L, Borenstein J (2007) Personal dead-reckoning system for gps-denied environments. In: Proceeding of the 2007 IEEE international workshop on safety, security and rescue robotics (SSRR): 27–29 September; Rome, Italy, pp 1–6

  37. Salas-Moreno R, Newcombe R, Strasdat H, Kelly P, Davison A (2013) Slam++: simultaneous localisation and mapping at the level of objects. In: Proceeding of the 2013 IEEE international conference on computer vision and pattern recognition (CVPR), pp 1352–1359

  38. Sattler T, Leibe B, Kobbelt L (2011) Fast image-based localization using direct 2d-to-3d matching. In: Proceeding of the 13th IEEE international conference on computer vision (ICCV), pp 667–674

  39. Shin D, Dunston P (2008) Identification of application areas for augmented reality in industrial construction based on technology suitability. Autom Constr 17(7):882–894

    Article  Google Scholar 

  40. Snavely N, Seitz S, Szeliski R (2007) Modeling the world from internet photo collections. Int J Comput Vis 80(2):189–210

    Article  Google Scholar 

  41. Strecha C, Pylvanainen T, Fua P (2010) Dynamic and scalable large scale image reconstruction. In: Proceeding of the 2010 IEEE international conference on computer vision and pattern recognition (CVPR), pp 406–413

  42. Ufkes A, Fiala M (2013) A markerless augmented reality system for mobile devices. In: Proceeding of the 2013 IEEE international conference on computer and robot vision (CRV), pp 226–233

  43. Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2010) Real-time detection and tracking for augmented reality on mobile phones. IEEE Trans Vis Comput Graphics 16(3):355–368

    Article  Google Scholar 

  44. Wang X (2008) Improving human–machine interfaces for construction equipment operations with mixed and augmented reality. In: Robotics and automation in construction: new development. I-Tech Education and Publishing, pp 349–362

  45. Woodward C, Hakkarainen M (2011) Mobile augmented reality system for construction site visualization. In: Proceeding of the international symposium on mixed and augmented reality (ISMAR), pp 1–6

  46. Woodward C, Hakkarainen M, Korkalo O, Kantonen T, Aittala M, Rainio K, Kähkönen K (2010) Mixed reality for mobile construction site visualization and communication. In: Proceeding of the 10th international conference on construction applications of virtual reality (CONVR), pp 35–44

  47. Wu C, Agarwal S, Curless B, Seitz S (2011) Multicore bundle adjustment. In: Proceeding of the 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 3057–3064

Download references

Author information

Authors and Affiliations

  1. Bradley Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA, USA

    Hyojoon Bae

  2. Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA

    Jules White, Yao Pan & Yu Sun

  3. Department of Civil and Environmental Engineering, Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL, USA

    Mani Golparvar-Fard

Authors
  1. Hyojoon Bae
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jules White
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mani Golparvar-Fard
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yao Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yu Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yao Pan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bae, H., White, J., Golparvar-Fard, M. et al. Fast and scalable 3D cyber-physical modeling for high-precision mobile augmented reality systems. Pers Ubiquit Comput 19, 1275–1294 (2015). https://doi.org/10.1007/s00779-015-0892-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-015-0892-6

Keywords

Search

Navigation