skip to main content
10.1145/2702123.2702222acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces

Published: 18 April 2015 Publication History

Abstract

We present an augmented reality system for large scale 3D reconstruction and recognition in outdoor scenes. Unlike existing prior work, which tries to reconstruct scenes using active depth cameras, we use a purely passive stereo setup, allowing for outdoor use and extended sensing range. Our system not only produces a map of the 3D environment in real-time, it also allows the user to draw (or 'paint') with a laser pointer directly onto the reconstruction to segment the model into objects. Given these examples our system then learns to segment other parts of the 3D map during online acquisition. Unlike typical object recognition systems, ours therefore very much places the user 'in the loop' to segment particular objects of interest, rather than learning from predefined databases. The laser pointer additionally helps to 'clean up' the stereo reconstruction and final 3D map, interactively. Using our system, within minutes, a user can capture a full 3D map, segment it into objects of interest, and refine parts of the model during capture. We provide full technical details of our system to aid replication, as well as quantitative evaluation of system components. We demonstrate the possibility of using our system for helping the visually impaired navigate through spaces. Beyond this use, our system can be used for playing large-scale augmented reality games, shared online to augment streetview data, and used for more detailed car and person navigation.

Supplementary Material

suppl.mov (pn0525-file3.m4v)
Supplemental video
MP4 File (p3317-miksik.mp4)

References

[1]
Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S. M., and Szeliski, R. Building Rome in a Day. CACM (2011).
[2]
Chen, D. M., Baatz, G., Köser, K., Tsai, S. S., Vedantham, R., Pylvänäinen, T., Roimela, K., Chen, X., Bach, J., Pollefeys, M., Girod, B., and Grzeszczuk, R. City-scale landmark identification on mobile devices. In CVPR (2011), 737--744.
[3]
Curless, B., and Levoy, M. A volumetric method for building complex models from range images. In SIGGRAPH (1996), 303--312.
[4]
Davison, A. J., Reid, I. D., Molton, N. D., and Stasse, O. MonoSLAM: Real-Time Single Camera SLAM. PAMI 29, 6 (2007).
[5]
Engel, J., Schöps, T., and Cremers, D. LSD-SLAM: Large-Scale Direct Monocular SLAM. In ECCV (2014).
[6]
Engel, J., Sturm, J., and Cremers, D. Semi-Dense Visual Odometry for a Monocular Camera. In ICCV (2013).
[7]
Fischler, M. A., and Bolles, R. C. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. CACM 24, 6 (1981).
[8]
Froissard, B., Konik, H., Trmeau, A., and Dinet, . Contribution of augmented reality solutions to assist visually impaired people in their mobility. In Universal Access in Human-Computer Interaction. Design for All and Accessibility Practice. Springer, 2014, 182--191.
[9]
Furukawa, Y., Curless, B., Seitz, S. M., and Szeliski, R. Reconstructing Building Interiors from Images. In ICCV (2009).
[10]
Geiger, A., Ziegler, J., and Stiller, C. StereoScan: Dense 3d Reconstruction in Real-time. In IVS (2011).
[11]
Habbecke, M., and Kobbelt, L. LaserBrush: A Flexible Device for 3D Reconstruction of Indoor Scenes. In SPM (2008).
[12]
Hane, C., Zach, C., Cohen, A., Angst, R., and Pollefeys, M. Joint 3d scene reconstruction and class segmentation. In CVPR (2013), 97--104.
[13]
Hartley, R., and Zisserman, A. Multiple view geometry in computer vision. Cambridge university press, 2003.
[14]
Hicks, S. L., Wilson, I., van Rheede, J. J., MacLaren, R. E., Downes, S. M., and Kennard, C. Improved mobility with depth-based residual vision glasses. Investigative Ophthalmology & Visual Science 55, 5 (2014).
[15]
Huang, A. S., Bachrach, A., Henry, P., Krainin, M., Maturana, D., Fox, D., and Roy, N. Visual Odometry and Mapping for Autonomous Flight Using an RGB-D Camera. In ISRR (2011).
[16]
Iannacci, F., Turnquist, E., Avrahami, D., and Patel, S. N. The Haptic Laser: Multi-Sensation Tactile Feedback for At-a-Distance Physical Space Perception and Interaction. In CHI (2011).
[17]
Jr., D. R. O., and Nielsen, T. Laser Pointer Interaction. In CHI (2001).
[18]
Klein, G., and Murray, D. W. Parallel tracking and mapping for small ar workspaces. In ISMAR (2007).
[19]
Ladicky, L., Russell, C., Kohli, P., and Torr, P. H. S. Associative Hierarchical CRFs for Object Class Image Segmentation. In ICCV (2009).
[20]
Mariotti, S. P. Global Data on Visual Impairments 2010. Tech. rep., World Health Organization, 2010.
[21]
Munoz, D., Bagnell, J. A., and Hebert, M. Stacked Hierarchical Labeling. In ECCV (2010).
[22]
Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohli, P., Shotton, J., Hodges, S., and Fitzgibbon, A. KinectFusion: Real-Time Dense Surface Mapping and Tracking. In ISMAR (2011).
[23]
Newcombe, R. A., Lovegrove, S. J., and Davison, A. J. DTAM: Dense Tracking and Mapping in Real-Time. In ICCV (2011).
[24]
Nguyen, T., Grasset, R., Schmalstieg, D., and Reitmayr, G. Interactive syntactic modeling with a single-point laser range finder and camera. In ISMAR (2013).
[25]
Nießner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. Real-time 3d reconstruction at scale using voxel hashing. TOG 32, 6 (2013), 169.
[26]
Qin, Y., Shi, Y., Jiang, H., and Yu, C. Structured Laser Pointer: Enabling Wrist-Rolling Movements as a New Interactive Dimension. In AVI (2010).
[27]
Rosten, E., and Drummond, T. Machine learning for high-speed corner detection. In ECCV (2006).
[28]
Salas-Moreno, R. F., Newcombe, R. A., Strasdat, H., Kelly, P. H. J., and Davison, A. J. SLAM++: SLAM at the Level of Objects. In CVPR (2013).
[29]
Sengupta, S., Greveson, E., Shahrokni, A., and Torr, P. H. S. Urban 3d semantic modelling using stereo vision. In ICRA (2013), 580--585.
[30]
Taneja, A., Ballan, L., and Pollefeys, M. City-scale change detection in cadastral 3d models using images. In CVPR (2013), 113--120.
[31]
Triggs, B., McLauchlan, P. F., Hartley, R. I., and Fitzgibbon, A. W. Bundle adjustment - a modern synthesis. In Workshop on Vision Algorithms (1999).
[32]
Valentin, J., Vineet, V., Cheng, M.-M., Kim, D., Shotton, J., Kohli, P., Niessner, M., Criminisi, A., Izadi, S., and Torr, P. H. S. SemanticPaint: Interactive 3D Labeling and Learning at your Fingertips. ACM TOG (2015).
[33]
Valentin, J. P. C., Sengupta, S., Warrell, J., Shahrokni, A., and Torr, P. H. S. Mesh based semantic modelling for indoor and outdoor scenes. In CVPR (2013), 2067--2074.
[34]
Whelan, T., Johannsson, H., Kaess, M., Leonard, J. J., and Mcdonald, J. Robust real-time visual odometry for dense rgb-d mapping. In ICRA (2013).
[35]
Wienss, C., Nikitin, I., Goebbels, G., Troche, K., Göbel, M., Nikitina, L., and Müller, S. Sceptre -- An Infrared Laser Tracking System for Virtual Environments. In VRST (2006).
[36]
Xiong, X., Munoz, D., Bagnell, J. A., and Hebert, M. 3-D Scene Analysis via Sequenced Predictions over Points and Regions. In ICRA (2011).

Cited By

View all
  • (2024)An Open Spatial Computing Platform2024 IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom)10.1109/MetaCom62920.2024.00046(239-246)Online publication date: 12-Aug-2024
  • (2024)Enhancing Replicability in XR HCI Studies: A Survey-Based Approach2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct64951.2024.00020(42-46)Online publication date: 21-Oct-2024
  • (2024)3D dataset generation using virtual reality for forest biodiversityInternational Journal of Digital Earth10.1080/17538947.2024.242298417:1Online publication date: 3-Nov-2024
  • Show More Cited By

Index Terms

  1. The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems
    April 2015
    4290 pages
    ISBN:9781450331456
    DOI:10.1145/2702123
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 18 April 2015

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. 3d reconstruction
    2. augmented reality
    3. laser pointer interaction
    4. semantic segmentation
    5. stereo
    6. visually impaired

    Qualifiers

    • Research-article

    Conference

    CHI '15
    Sponsor:
    CHI '15: CHI Conference on Human Factors in Computing Systems
    April 18 - 23, 2015
    Seoul, Republic of Korea

    Acceptance Rates

    CHI '15 Paper Acceptance Rate 486 of 2,120 submissions, 23%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)44
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)An Open Spatial Computing Platform2024 IEEE International Conference on Metaverse Computing, Networking, and Applications (MetaCom)10.1109/MetaCom62920.2024.00046(239-246)Online publication date: 12-Aug-2024
    • (2024)Enhancing Replicability in XR HCI Studies: A Survey-Based Approach2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct64951.2024.00020(42-46)Online publication date: 21-Oct-2024
    • (2024)3D dataset generation using virtual reality for forest biodiversityInternational Journal of Digital Earth10.1080/17538947.2024.242298417:1Online publication date: 3-Nov-2024
    • (2023)iLabel: Revealing Objects in Neural FieldsIEEE Robotics and Automation Letters10.1109/LRA.2022.32314988:2(832-839)Online publication date: Feb-2023
    • (2023)Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)10.1109/ICCVW60793.2023.00200(1849-1859)Online publication date: 2-Oct-2023
    • (2023)Background Features-Based Novel Visual Ego-Motion EstimationComputer Vision and Machine Intelligence Paradigms for SDGs10.1007/978-981-19-7169-3_16(175-189)Online publication date: 1-Jan-2023
    • (2022)Outside Where? A Survey of Climates and Built Environments in Studies of HCI outdoorsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3507656(1-15)Online publication date: 29-Apr-2022
    • (2022)Trans4Trans: Efficient Transformer for Transparent Object and Semantic Scene Segmentation in Real-World Navigation AssistanceIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2022.316114123:10(19173-19186)Online publication date: Oct-2022
    • (2022)VIBUS: Data-efficient 3D scene parsing with VIewpoint Bottleneck and Uncertainty-Spectrum modelingISPRS Journal of Photogrammetry and Remote Sensing10.1016/j.isprsjprs.2022.10.013194(302-318)Online publication date: Dec-2022
    • (2022)LiDAR MetricsLarge-Scale Simultaneous Localization and Mapping10.1007/978-981-19-1972-5_8(171-229)Online publication date: 14-Jun-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media