Skip to main content
SpringerLink
Log in

Toward cross-platform immersive visualization for indoor navigation and collaboration with augmented reality

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

Navigation and situation awareness in 3D environments are often required by emergency responses such as firefighters, police, and military soldiers. This paper investigates a collaborative, cross-platform immersive system to improve the team navigation through effective real-time communication. We explore a set of essential building navigation, visualization, and interaction methods to support joint tasks in the physical building environments by leveraging on device sensors and image markers. Our platform also supports efficient exchange of useful visual information to improve the coordination and situation awareness for multiple users performing real-time operations in smart buildings. We have performed a user study to evaluate different devices used in coordination tasks. Our results demonstrate the effects of immersive visualization for improving 3D navigation and coordination for on-site collaboration in real physical environment.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  • Ayyanchira A, Mahfoud E, Wang W, Lu A (2021) Cross-platform immersive visualization and navigation with augmented reality. In: Proceedings of the international symposium on visual information communication and interaction. VINCI

  • Azuma RT (1997) A survey of augmented reality. Presence Teleoper Virtual Environ 6(4):355–385

    Article  Google Scholar 

  • Baudisch P, Rosenholtz R (2003) Halo: a technique for visualizing off-screen objects. In: Proceedings of the SIGCHI conference on human factors in computing systems. CHI ’03. Association for Computing Machinery, New York, NY, USA, pp 481-488. https://doi.org/10.1145/642611.642695

  • Ben-Haim D, Lappan G, Houang RT (1985) Visualizing rectangular solids made of small cubes: analyzing and effecting students’ performance. Educ Stud Math. 16(4):389–409. https://doi.org/10.1007/BF00417194

    Article  Google Scholar 

  • Benko H, Ishak EW, Feiner S (2004) Collaborative mixed reality visualization of an archaeological excavation. In: Third IEEE and ACM international symposium on mixed and augmented reality, pp 132–140

  • Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Found Trends Hum Comput Interact 8(2–3):73–272

    Article  Google Scholar 

  • Biocca F, Tang A, Owen C, Xiao F (2006) Attention funnel: omnidirectional 3D cursor for mobile augmented reality platforms. In: Proceedings of the SIGCHI conference on human factors in computing systems. CHI ’06. Association for Computing Machinery, New York, NY, USA, pp 1115–1122. https://doi.org/10.1145/1124772.1124939

  • Butscher S, Hubenschmid S, Müller J, Fuchs J, Reiterer H (2018) Clusters, trends, and outliers: how immersive technologies can facilitate the collaborative analysis of multidimensional data. In: Proceedings of the 2018 CHI conference on human factors in computing systems. CHI ’18, pp 90:1–90:12

  • Chen H, Lee AS, Swift M, Tang JC (2015) 3D collaboration method over HoloLens\(^{TM}\) and Skype\(^{TM}\) end points. In: Proceedings of the 3rd international workshop on immersive media experiences. ACM, pp 27–30

  • Coffey D, Malbraaten N, Le TB, Borazjani I, Sotiropoulos F, Erdman AG et al (2012) Interactive slice WIM: navigating and interrogating volume data sets using a multisurface, multitouch VR interface. IEEE Trans Vis Comput Graph 18(10):1614–1626. https://doi.org/10.1109/TVCG.2011.283

    Article  Google Scholar 

  • Cui Z, Sen S, Badam SK, Elmqvist N (2018) VisHive: supporting web-based visualization through ad-hoc computational clusters of mobile devices. Inf Vis 18:195–210

    Article  Google Scholar 

  • Dong S, Kamat VR (2011) Collaborative visualization of simulated processes using tabletop fiducial augmented reality. In: Proceedings of the 2011 winter simulation conference (WSC), pp 828–837

  • Elmqvist N, Irani P (2013) Ubiquitous analytics: interacting with big data anywhere anytime. IEEE Comput 46(4):86–89

    Article  Google Scholar 

  • Faria NdO, Gabbard JL, Smith M (2020) Place in the world or place on the screen? Investigating the effects of augmented reality head -up display user interfaces on drivers’ spatial knowledge acquisition and glance behavior. In: 2020 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW), pp 762–763

  • Feiner S, MacIntyre B, Hollerer T, Webster A (1997) A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment. In: Proceedings of the 1st IEEE international symposium on wearable computers. ISWC ’97. IEEE Computer Society, Washington, DC, USA, pp 74–81. http://dl.acm.org/citation.cfm?id=851036.856454

  • Gerstweiler G (2018) Guiding people in complex indoor environments using augmented reality. In: 2018 IEEE conference on virtual reality and 3D user interfaces (VR), pp 801–802

  • Gruenefeld U, Ali AE, Boll S, Heuten W (2018) Beyond Halo and wedge: visualizing out-of-view objects on head-mounted virtual and augmented reality devices. In: Proceedings of the 20th international conference on human–computer interaction with mobile devices and services. MobileHCI ’18. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3229434.3229438

  • Gustafson S, Baudisch P, Gutwin C, Irani P (2008) Wedge: clutter-free visualization of off-screen locations. In: Proceedings of the SIGCHI conference on human factors in computing systems. CHI ’08. Association for Computing Machinery, New York, NY, USA, pp 787–796. https://doi.org/10.1145/1357054.1357179

  • Hein P, Bernhagen M, Bullinger AC (2020) Two is better than one. Improved attention guiding in AR by combining techniques. IEEE Comput Graph Appl 40(5):57–66. https://doi.org/10.1109/MCG.2020.3012274

    Article  Google Scholar 

  • Horak T, Badam SK, Elmqvist N, Dachselt R (2018) When David meets Goliath: combining smartwatches with a large vertical display for visual data exploration. In: Proceedings of the ACM conference on human factors in computing systems. http://www.umiacs.umd.edu/~elm/projects/david-goliath/david-goliath.pdf

  • Huang W, Sun M, Li S (2016) A 3D GIS-based interactive registration mechanism for outdoor augmented reality system. Expert Syst Appl 55:48–58. https://doi.org/10.1016/j.eswa.2016.01.037

    Article  Google Scholar 

  • Irizarry J, Gheisari M, Williams G, Walker BN (2013) InfoSPOT: a mobile Augmented Reality method for accessing building information through a situation awareness approach. Autom Constr 33:11–23. Augmented Reality in Architecture, Engineering, and Construction. https://doi.org/10.1016/j.autcon.2012.09.002

  • Khenak N, Vézien J, Bourdot P (2020) Spatial presence, performance, and behavior between real, remote, and virtual immersive environments. IEEE Trans Vis Comput Graph 26(12):3467–3478. https://doi.org/10.1109/TVCG.2020.3023574

    Article  Google Scholar 

  • Kim K, Billinghurst M, Bruder G, Duh HB, Welch GF (2018) Revisiting trends in augmented reality research: a review of the 2nd decade of ISMAR (2008–2017). IEEE Trans Vis Comput Graph 24(11):2947–2962. https://doi.org/10.1109/TVCG.2018.2868591

    Article  Google Scholar 

  • Kim K, Billinghurst M, Bruder G, Duh HB, Welch GF (2018) Revisiting trends in augmented reality research: a review of the 2nd Decade of ISMAR (2008–2017). IEEE Trans Vis Comput Graph. https://doi.org/10.1109/TVCG.2018.2868591

    Article  Google Scholar 

  • Kishishita N, Kiyokawa K, Orlosky J, Mashita T, Takemura H, Kruijff E (2014) Analysing the effects of a wide field of view augmented reality display on search performance in divided attention tasks. In: 2014 IEEE international symposium on mixed and augmented reality (ISMAR), pp 177–186

  • Kister U, Klamka K, Tominski C, Dachselt R (2017) GraSp: combining spatially-aware mobile devices and a display wall for graph visualization and interaction. Comput Graph Forum 36(3):503–514. https://doi.org/10.1111/cgf.13206

    Article  Google Scholar 

  • Lee Y, Choi W, Sohn BS (2018) Immersive gesture interfaces for 3D map navigation in HMD-based virtual environments. In: 2018 international conference on information networking (ICOIN), pp 963–965

  • Morar A, B\(\breve{a}\)lutoiu MA, Moldoveanu A, Moldoveanu F, Butean A, Asavei V (2020) Evaluation of the ARCore indoor localization technology. In: 2020 19th RoEduNet conference: networking in education and research (RoEduNet), pp 1–5

  • Mulloni A, Seichter H, Schmalstieg D (2011) User experiences with augmented reality aided navigation on phones. In: 2011 10th IEEE international symposium on mixed and augmented reality, pp 229–230

  • Nam JW, McCullough K, Tveite J, Espinosa MM, Perry CH, Wilson BT et al (2019) Worlds-in-wedges: combining worlds-in-miniature and portals to support comparative immersive visualization of forestry data. In: 2019 IEEE conference on virtual reality and 3D user interfaces (VR), pp 747–755

  • Nilsson S, Johansson BJE, Jonsson A (2011) Cross-organizational collaboration supported by augmented reality. IEEE Trans Vis Comput Graph 17(10):1380–1392. https://doi.org/10.1109/TVCG.2010.249

    Article  Google Scholar 

  • Pankratz F, Dippon A, Coskun T, Klinker G (2013) User awareness of tracking uncertainties in AR navigation scenarios. In: 2013 IEEE international symposium on mixed and augmented reality (ISMAR), pp 285–286

  • Perea P, Morand D, Nigay L (2017) [POSTER] Halo3D: a technique for visualizing off-screen points of interest in mobile augmented reality. In: 2017 IEEE international symposium on mixed and augmented reality (ISMAR-Adjunct), pp 170–175

  • Pillette L, Moreau G, Normand JM, Perrier M, Lecuyer A, Cogne M (2022) A systematic review of navigation assistance systems for people with dementia. IEEE Trans Vis Comput Graph. https://doi.org/10.1109/TVCG.2022.3141383

    Article  Google Scholar 

  • Roberts JC, Ritsos PD, Badam SK, Brodbeck D, Kennedy J, Elmqvist N (2014) Visualization beyond the desktop-the next big thing. IEEE Comput Graph Appl. 34(6):26–34

    Article  Google Scholar 

  • Roux WL (2011) The use of augmented reality in command and control situation awareness. Sci Mil South Afr J Mil Stud 38(1)

  • Santos MEC, Chen A, Taketomi T, Yamamoto G, Miyazaki J, Kato H (2014) Augmented reality learning experiences: survey of prototype design and evaluation. IEEE Trans Learn Technol. 7(1):38–56

    Article  Google Scholar 

  • Sayapogu T, Dsa K, Kaul P (2021) AR smart navigation system. In: 2021 2nd international conference for emerging technology (INCET), pp 1–4

  • Sereno M, Wang X, Besancon L, Mcguffin MJ, Isenberg T (2020) Collaborative work in augmented reality: a survey. IEEE Trans Vis Comput Graph. https://doi.org/10.1109/TVCG.2020.3032761

    Article  Google Scholar 

  • Speicher M, Hall BD, Yu A, Zhang B, Zhang H, Nebeling J et al (2018) XD-AR: challenges and opportunities in cross-device augmented reality application development. Proc ACM Hum Comput Interact 2(EICS):7:1-7:24. https://doi.org/10.1145/3229089

    Article  Google Scholar 

  • Stoakley R, Conway MJ, Pausch R (1995) Virtual reality on a WIM: interactive worlds in miniature. In: Proceedings of the SIGCHI conference on human factors in computing systems. CHI ’95. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, pp 265–272. https://doi.org/10.1145/223904.223938

  • Sukan M, Elvezio C, Oda O, Feiner S, Tversky B (2014) ParaFrustum: visualization techniques for guiding a user to a constrained set of viewing positions and orientations. In: Proceedings of the 27th annual ACM symposium on user interface software and technology. UIST ’14. Association for Computing Machinery, New York, NY, USA, pp 331–340. https://doi.org/10.1145/2642918.2647417

  • Sutherland IE (1968) A head-mounted three dimensional display. In: Proceedings of the December 9-11, 1968, fall joint computer conference, Part I. AFIPS ’68 (Fall, part I). ACM, New York, pp 757–764. https://doi.org/10.1145/1476589.1476686

  • Szalavári Z, Schmalstieg D, Fuhrmann A, Gervautz M (1998) Studierstube: an environment for collaboration in augmented reality. Virtual Real 3(1):37–48. https://doi.org/10.1007/BF01409796

    Article  Google Scholar 

  • Tache R, Abeykoon HA, Karunanayaka KT, Kumarasinghe JP, Roth G, Fernando ONN et al (2012) Command center: authoring tool to supervise augmented reality session. In: 2012 IEEE virtual reality workshops (VRW). IEEE, pp 99–100

  • Thangarajah A, Wu J, Madon B, Chowdhury AK (2015) Vision-based registration for augmented reality—a short survey. In: 2015 IEEE international conference on signal and image processing applications (ICSIPA), pp 463–468

  • Tonnis M, Klinker G (2006) Effective control of a car driver’s attention for visual and acoustic guidance towards the direction of imminent dangers. In: 2006 IEEE/ACM international symposium on mixed and augmented reality, pp 13–22

  • Trapp M, Dumke F, Döllner J (2019) Occlusion management techniques for the visualization of transportation networks in virtual 3D city models. In: Proceedings of the 12th international symposium on visual information communication and interaction. VINCI’2019. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3356422.3356445

  • Wang L, Wu J, Yang X, Popescu V (2019) VR exploration assistance through automatic occlusion removal. IEEE Trans Vis Comput Graph 25(5):2083–2092. https://doi.org/10.1109/TVCG.2019.2898782

    Article  Google Scholar 

  • Wang X, Lim J, T Collins R, Hanson A (1996) Automated texture extraction from multiple images to support site model refinement and visualization. In: The fourth international conference in Central Europe on computer graphics and visualization

  • Wang L, Wu W, Zhou Z, Popescu V (2020) View splicing for effective VR collaboration. In: 2020 IEEE international symposium on mixed and augmented reality (ISMAR), pp 509–519

  • Weick KE (1993) The Mann Gulch disaster—the collapse of sensemaking in organizations. Adm Sci Q 38(4):628–652

    Article  Google Scholar 

  • Weick KE (1995) South Canyon revisited—lessons from high reliability organizations. Wildfire 4:54–68

    Google Scholar 

  • Wither J, White S, Azuma R (2011) Comparing spatial understanding between touch-based and AR-style interaction. In: 2011 10th IEEE international symposium on mixed and augmented reality, pp 273–274

  • Wu ML, Popescu V (2018) Efficient VR and AR navigation through multiperspective occlusion management. IEEE Trans Vis Comput Graph 24(12):3069–3080. https://doi.org/10.1109/TVCG.2017.2778249

    Article  Google Scholar 

  • Zollmann S, Hoppe C, Langlotz T, Reitmayr G (2014) FlyAR: augmented reality supported micro aerial vehicle navigation. IEEE Trans Vis Comput Graph 20(4):560–568. https://doi.org/10.1109/TVCG.2014.24

    Article  Google Scholar 

Download references

Acknowledgements

We are thankful for the reviewers’ suggestions. This work was supported by NSF Awards 1564039, 1629913, 1661280, 1840080, 1937010 and 2126116.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, 28173, NC, USA

    Akshay Ayyanchira, Elias Mahfoud & Aidong Lu

  2. Department of Software and Information Systems, University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, 28173, NC, USA

    Weichao Wang

Authors
  1. Akshay Ayyanchira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elias Mahfoud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weichao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aidong Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aidong Lu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ayyanchira, A., Mahfoud, E., Wang, W. et al. Toward cross-platform immersive visualization for indoor navigation and collaboration with augmented reality. J Vis 25, 1249–1266 (2022). https://doi.org/10.1007/s12650-022-00852-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-022-00852-9

Keywords

Search

Navigation