skip to main content
10.1145/3544549.3585862acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
Work in Progress

Point Cloud Alignment through Mid-Air Gestures on a Stereoscopic Display

Published: 19 April 2023 Publication History

Abstract

Manual point cloud registration is often a crucial step during the mapping of 3D point clouds and usually performed on a conventional desktop setup with mouse interaction. Since 3D point clouds are inherently spatial, these 2D applications suffer from impaired depth perception and inconvenient interaction. Nonetheless, there are few efforts to improve the usability of these applications. To address this, we propose an alternative setup, consisting of a stereoscopic display and an external hand tracker, allowing for enhanced depth perception and natural interaction without the need for body-worn devices or handheld controllers. We developed interaction techniques for point cloud alignment in 3D space, including visual feedback during alignment, and implemented a proof-of-concept prototype in the context of a surgical use case. We describe the use case, design and implementation of our concepts and outline future work. Herewith we provide a user-centered alternative to desktop applications for manual point cloud registration.

Supplementary Material

Supplemental Materials (3544549.3585862-supplemental-materials.zip)
MP4 File (3544549.3585862-video-figure.mp4)
Video Figure
MP4 File (3544549.3585862-talk-video.mp4)
Pre-recorded Video Presentation
MP4 File (3544549.3585862-video-preview.mp4)
Video Preview

References

[1]
P.J. Besl and Neil D. McKay. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2 (1992), 239–256. https://doi.org/10.1109/34.121791
[2]
Alexander Bucksch and Kourosh Khoshelham. 2012. Localized Registration of Point Clouds of Botanic Trees. IEEE GEOSCIENCE AND REMOTE SENSING LETTERS 10 (01 2012), 1–5. https://doi.org/10.1109/LGRS.2012.2216251
[3]
Simon Burkard and Frank Fuchs-Kittowski. 2020. User-Aided Global Registration Method using Geospatial 3D Data for Large-Scale Mobile Outdoor Augmented Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). 104–109. https://doi.org/10.1109/ISMAR-Adjunct51615.2020.00041
[4]
Yang Chen and Gérard Medioni. 1992. Object modelling by registration of multiple range images. Image and vision computing 10, 3 (1992), 145–155.
[5]
Logan W. Clements, William C. Chapman, Benoit M. Dawant, Robert L. Galloway Jr., and Michael I. Miga. 2008. Robust surface registration using salient anatomical features for image-guided liver surgery: Algorithm and validation. Medical Physics 35, 6Part1 (2008), 2528–2540. https://doi.org/10.1118/1.2911920
[6]
Mine Dastan, Antonio Emmanuele Uva, and Michele Fiorentino. 2022. Gestalt Driven Augmented Collimator Widget for Precise 5 DOF Dental Drill Tool Positioning in 3D Space. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 187–195. https://doi.org/10.1109/ISMAR55827.2022.00033
[7]
Reuben Docea, Micha Pfeiffer, Sebastian Bodenstedt, Fiona R. Kolbinger, Lukas Höller, Ines Wittig, Ralf-Thorsten Hoffmann, Esther G. C. Troost, Carina Riediger, Jürgen Weitz, and Stefanie Speidel. 2021. Simultaneous localisation and mapping for laparoscopic liver navigation : a comparative evaluation study. In Medical Imaging 2021: Image-Guided Procedures, Robotic Interventions, and Modeling, Cristian A. Linte and Jeffrey H. Siewerdsen (Eds.). Vol. 11598. International Society for Optics and Photonics, SPIE, 115980B. https://doi.org/10.1117/12.2582121
[8]
Reuben Docea, Micha Pfeiffer, Jan Müller, Katja Krug, Matthias Hardner, Paul Riedel, Martin Menzel, Fiona R. Kolbinger, Laura Frohneberg, Jürgen Weitz, and Stefanie Speidel. 2022. A Laparoscopic Liver Navigation Pipeline with Minimal Setup Requirements. In 2022 IEEE Biomedical Circuits and Systems Conference (BioCAS). 578–582. https://doi.org/10.1109/BioCAS54905.2022.9948587
[9]
Åsmund Avdem Fretland, Davit Aghayan, and Bjørn and Edwin. 2019. Long-term survival after laparoscopic versus open resection for colorectal liver metastases.Journal of Clinical Oncology 37, 18_suppl (2019), LBA3516–LBA3516. https://doi.org/10.1200/JCO.2019.37.18_suppl.LBA3516
[10]
P Christopher Gloumeau, Wolfgang Stuerzlinger, and JungHyun Han. 2020. Pinnpivot: Object manipulation using pins in immersive virtual environments. IEEE transactions on visualization and computer graphics 27, 4 (2020), 2488–2494.
[11]
Ammar Hattab and Gabriel Taubin. 2018. 3D Rigid Registration of Cad Point-Clouds. In 2018 International Conference on Computing Sciences and Engineering (ICCSE). 1–6. https://doi.org/10.1109/ICCSE1.2018.8373991
[12]
Jon S. Heiselman, Logan W. Clements, Jarrod A. Collins, Jared A. Weis, Amber L. Simpson, Sunil K. Geevarghese M.D., T. Peter Kingham M.D., William R. Jarnagin M.D., and Michael I. Miga. 2017. Characterization and correction of intraoperative soft tissue deformation in image-guided laparoscopic liver surgery. Journal of Medical Imaging 5, 2 (2017), 021203. https://doi.org/10.1117/1.JMI.5.2.021203
[13]
Kok-Lim Low. 2004. Linear Least-Squares Optimization for Point-to-Plane Icp Surface Registration. Chapel Hill, University of North Carolina 4, 10 (2004), 1–3.
[14]
Paul Lubos, Rüdiger Beimler, Markus Lammers, and Frank Steinicke. 2014. Touching the Cloud: Bimanual annotation of immersive point clouds. In 2014 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, 191–192.
[15]
Stephen McClain. 2016. Manual Point Cloud Registration for Combined Ice Roughness and Ice Thickness Measurements. https://doi.org/10.2514/6.2016-3590
[16]
John P. McIntire, Paul R. Havig, and Eric E. Geiselman. 2012. What is 3D good for? A review of human performance on stereoscopic 3D displays. In Head- and Helmet-Mounted Displays XVII; and Display Technologies and Applications for Defense, Security, and Avionics VI, Peter L. Marasco, Paul R. Havig II, Daniel D. Desjardins, and Kalluri R. Sarma (Eds.). Vol. 8383. International Society for Optics and Photonics, SPIE, 83830X. https://doi.org/10.1117/12.920017
[17]
John P. McIntire, Paul R. Havig, and Eric E. Geiselman. 2014. Stereoscopic 3D displays and human performance: A comprehensive review. Displays 35, 1 (2014), 18–26. https://doi.org/10.1016/j.displa.2013.10.004
[18]
Daniel Mendes, Filipe Relvas, Alfredo Ferreira, and Joaquim Jorge. 2016. The Benefits of DOF Separation in Mid-Air 3D Object Manipulation(VRST ’16). Association for Computing Machinery, New York, NY, USA, 261–268. https://doi.org/10.1145/2993369.2993396
[19]
Ellon Mendes, Pierrick Koch, and Simon Lacroix. 2016. ICP-based pose-graph SLAM. In 2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). IEEE, 195–200.
[20]
Tamara Munzner. 2014. Visualization analysis and design. CRC press.
[21]
Raul Mur-Artal, J. Montiel, and Juan Tardos. 2015. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Transactions on Robotics 31 (10 2015), 1147 – 1163. https://doi.org/10.1109/TRO.2015.2463671
[22]
Raul Mur-Artal and Juan Tardos. 2016. ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras. IEEE Transactions on Robotics PP (10 2016). https://doi.org/10.1109/TRO.2017.2705103
[23]
Micha Pfeiffer, Carina Riediger, Stefan Leger, Jens-Peter Kühn, Danilo Seppelt, Ralf-Thorsten Hoffmann, Jürgen Weitz, and Stefanie Speidel. 2020. Non-Rigid Volume to Surface Registration Using a Data-Driven Biomechanical Model. (09 2020). https://doi.org/10.1007/978-3-030-59719-1_70
[24]
Gian Andrea Prevost, Benjamin Eigl, Iwan Paolucci, Tobias Rudolph, Matthias Peterhans, Stefan Weber, Guido Beldi, Daniel Candinas, and Anja Lachenmayer. 2019. Efficiency, Accuracy and Clinical Applicability of a New Image-Guided Surgery System in 3D Laparoscopic Liver Surgery. Journal of Gastrointestinal Surgery 24 (10 2019). https://doi.org/10.1007/s11605-019-04395-7
[25]
Shirin Sadri, Shalva A. Kohen, Carmine Elvezio, Shawn H. Sun, Alon Grinshpoon, Gabrielle J. Loeb, Naomi Basu, and Steven K. Feiner. 2019. Manipulating 3D Anatomic Models in Augmented Reality: Comparing a Hands-Free Approach and a Manual Approach. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 93–102. https://doi.org/10.1109/ISMAR.2019.00-21
[26]
Crispin Schneider, Moustafa Allam, Danail Stoyanov, D.J. Hawkes, K. Gurusamy, and Brian Davidson. 2021. Performance of image guided navigation in laparoscopic liver surgery – A systematic review. Surgical Oncology 38 (07 2021), 101637. https://doi.org/10.1016/j.suronc.2021.101637
[27]
Martin Schrepp and Jörg Tomaschewski. 2019. Eine Modulare Erweiterung Des User Experience Questionnaire. Mensch und Computer – Usability Professionals (2019), 148–156. https://doi.org/10.18420/muc2019-up-0108
[28]
Jonathan Dyssel Stets, Yongbin Sun, Wiley Corning, and Scott W. Greenwald. 2017. Visualization and Labeling of Point Clouds in Virtual Reality. In SIGGRAPH Asia 2017 Posters (Bangkok, Thailand) (SA ’17). Association for Computing Machinery, New York, NY, USA, Article 31, 2 pages. https://doi.org/10.1145/3145690.3145729
[29]
Mengu Sukan, Carmine Elvezio, Steven Feiner, and Barbara Tversky. 2016. Providing Assistance for Orienting 3D Objects Using Monocular Eyewear. In Proceedings of the 2016 Symposium on Spatial User Interaction (Tokyo, Japan) (SUI ’16). Association for Computing Machinery, New York, NY, USA, 89–98. https://doi.org/10.1145/2983310.2985764
[30]
Stephen Thompson, Johannes Totz, Yi Song, Stian Johnsen, Danail Stoyanov, Sébastien Ourselin, Kurinchi Gurusamy, Crispin Schneider, Brian Davidson, David Hawkes, and Matthew J. Clarkson. 2015. Accuracy validation of an image guided laparoscopy system for liver resection. In Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling, Robert J. Webster III and Ziv R. Yaniv (Eds.). Vol. 9415. International Society for Optics and Photonics, SPIE, 941509. https://doi.org/10.1117/12.2080974
[31]
Florian Wirth, Jannik Quehl, Jeffrey Ota, and Christoph Stiller. 2019. PointAtMe: Efficient 3D Point Cloud Labeling in Virtual Reality. In 2019 IEEE Intelligent Vehicles Symposium (IV). 1693–1698. https://doi.org/10.1109/IVS.2019.8814115

Cited By

View all
  • (2024)Annorama: Enabling Immersive At-Desk Annotation Experiences in Virtual Reality with 3D Point Cloud DioramasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682081(1-13)Online publication date: 7-Oct-2024
  • (2023)Discussing Facets of Hybrid User Interfaces for the Medical Domain2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00059(257-260)Online publication date: 16-Oct-2023
  • (2023)Fantastic Hybrid User Interfaces and How to Define Them2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00057(247-250)Online publication date: 16-Oct-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems
April 2023
3914 pages
ISBN:9781450394222
DOI:10.1145/3544549
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Check for updates

Author Tags

  1. image guidance navigation systems
  2. mid-air interaction
  3. point cloud alignment
  4. point cloud registration
  5. stereoscopic display

Qualifiers

  • Work in progress
  • Research
  • Refereed limited

Funding Sources

Conference

CHI '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)106
  • Downloads (Last 6 weeks)11
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Annorama: Enabling Immersive At-Desk Annotation Experiences in Virtual Reality with 3D Point Cloud DioramasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682081(1-13)Online publication date: 7-Oct-2024
  • (2023)Discussing Facets of Hybrid User Interfaces for the Medical Domain2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00059(257-260)Online publication date: 16-Oct-2023
  • (2023)Fantastic Hybrid User Interfaces and How to Define Them2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00057(247-250)Online publication date: 16-Oct-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media