Skip to main content
SpringerLink
Log in

Real-time enhancement of sparse 3D maps using a parallel segmentation scheme based on superpixels

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In this work, we focus on the problem of feature-based 3D mapping of environments with large textureless regions, which generates sparse 3D maps that may not represent well the mapped scene. To deal with this problem, based on our previous work (Cruz Martinez et al. in 2016 IEEE international symposium on mixed and augmented reality (ISMAR) adjunct proceedings, IEEE, 2016), we propose to enhance sparse 3D maps by using a superpixel-based segmentation with the aim of generating denser 3D maps of the scene in real time. Superpixels are middle-level features, which represent similar regions in an image, which can be connected in order to segment textureless areas. We propose a graphics processor unit architecture for (1) superpixel extraction considering chromatic and depth information, (2) superpixel-based segmentation, generation of connectivity matrix to compute the connected components algorithm and (3) mapping of segmented regions to 3D points. We use the ORB-SLAM system (Mur-Artal et al. in IEEE Trans Robot 31(5):1147–1163, 2015) to generate a sparse 3D map and to project the textureless segments onto it at 27 frames per second. We assessed our approach in terms of segmentation and map quality. Regarding the latter, covered area by the generated map, depth accuracy, and computational performance are reported.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  1. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)

    Article  Google Scholar 

  2. Alcantarilla, P., Stent, S., Ros, G., Arroyo, R., Gherardi, R.: Street-view change detection with deconvolutional networks. In: Robotics: Science and Systems (RSS), Michigan, USA (2016)

  3. Choi, K.S.., Oh, K.W..: Fast simple linear iterative clustering by early candidate cluster elimination. In: Iberian Conference on Pattern Recognition and Image Analysis, Springer, pp. 579–586 (2015)

  4. Concha, A., Civera, J.: Using superpixels in monocular SLAM. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 365–372 (2014)

  5. Concha, A., Civera, J.: DPPTAM: Dense piecewise planar tracking and mapping from a monocular sequence. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp. 5686–5693 (2015)

  6. Concha, A., Hussain, W., Montano, L., Civera, J.: Incorporating scene priors to dense monocular mapping. Auton. Robots 39(3), 279–292 (2015)

    Article  Google Scholar 

  7. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms. MIT Press, Cambridge (2009)

    MATH  Google Scholar 

  8. Cruz Martinez, C., Martinez Carranza, J., Mayol-Cuevas, W., Arias Estrada, M.O.: Enhancing 3d mapping via real-time superpixel-based segmentation. In: 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Adjunct Proceedings, IEEE, pp. 90–95 (2016)

  9. Durrant-Whyte, H., Bailey, T.: Simultaneous localization and mapping: part I. IEEE Robot. Autom. Mag. 13(2), 99–110 (2006)

    Article  Google Scholar 

  10. Eade, E., Drummond, T.: Edge landmarks in monocular SLAM. In: Proceedings of British Machine Vision Conference, Citeseer (2006)

  11. Engel, J., Sturm, J., Cremers, D.: Semi-dense visual odometry for a monocular camera. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1449–1456 (2013)

  12. Engel, J., Schöps, T., Cremers, D.: LSD-SLAM: large-scale direct monocular SLAM. In: Computer Vision–ECCV 2014, pp. 834–849. Springer (2014)

  13. Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181 (2004)

    Article  Google Scholar 

  14. Forster, C., Pizzoli, M., Scaramuzza, D.: SVO: fast semi-direct monocular visual odometry. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 15–22 (2014)

  15. Hartley, R., Andrew, Z.: Multiple View Geometry in Computer Vision. Cambridge University Press, Espaa (2003)

    MATH  Google Scholar 

  16. Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: RGB-D mapping: using kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robot. Res. 31(5), 647–663 (2012)

    Article  Google Scholar 

  17. Hornung, A., Wurm, K.M., Bennewitz, M., Stachniss, C., Burgard, W.: OctoMap: an efficient probabilistic 3D mapping framework based on octrees. Auton. Robots 34(3), 189–206 (2013)

    Article  Google Scholar 

  18. Jiang, L., Lu, H., Koch, A., Zell, A.: Superpixel segmentation based gradient maps on RGB-D dataset. In: 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, pp. 1359–1364 (2015)

  19. Karlsson, N., Di Bernardo, E., Ostrowski, J., Goncalves, L., Pirjanian, P., Munich, M.E.: The vSLAM algorithm for robust localization and mapping. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, 2005. ICRA 2005, IEEE, pp. 24–29 (2005)

  20. Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, 2007. ISMAR 2007. IEEE, pp. 225–234 (2007)

  21. Klein, G., Murray, D.: Improving the agility of keyframe-based SLAM. In: European Conference on Computer Vision, Springer, pp. 802–815 (2008)

  22. Li, M., Mourikis, A.I.: High-precision, consistent EKF-based visual-inertial odometry. Int. J. Robot. Res. 32(6), 690–711 (2013)

    Article  Google Scholar 

  23. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Eighth IEEE International Conference on Computer Vision, 2001. ICCV 2001. Proceedings. IEEE, vol. 2, pp. 416–423 (2001)

  24. Müller, A.C., Behnke, S.: Learning depth-sensitive conditional random fields for semantic segmentation of rgb-d images. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 6232–6237 (2014)

  25. Mur-Artal, R., Tardós, J.D.: Probabilistic semi-dense mapping from highly accurate feature-based monocular slam. In: Proceedings of Robotics: Science and Systems, Rome, Italy 1 (2015)

  26. Mur-Artal, R., Montiel, J., Tardos, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans. Robot. 31(5), 1147–1163 (2015)

    Article  Google Scholar 

  27. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor Segmentation and Support Inference from RGBD Images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) Computer Vision – ECCV 2012. ECCV 2012. Lecture Notes in Computer Science, vol 7576. Springer, Berlin, Heidelberg (2012)

  28. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: KinectFusion: real-time dense surface mapping and tracking. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, pp. 127–136 (2011)

  29. Newcombe, R.A., Lovegrove, S.J., Davison, A.J., DTAM: dense tracking and mapping in real-time. In: 2011 IEEE International Conference on Computer Vision (ICCV), IEEE, pp. 2320–2327 (2011)

  30. Pizzoli, M., Forster, C., Scaramuzza, D.: REMODE: probabilistic, monocular dense reconstruction in real time. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 2609–2616 (2014)

  31. Ren, C.Y., Reid, I.: gSLIC: a real-time implementation of SLIC superpixel segmentation. University of Oxford, Department of Engineering, technical report (2011)

  32. Ren, C.Y., Prisacariu, V.A.: Reid ID gSLICr: SLIC superpixels at over 250 Hz. arXiv preprint arXiv:150904232 (2015)

  33. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: 2011 International conference on computer vision, IEEE, pp. 2564–2571 (2011)

  34. Rusu, R.B., Cousins, S.: 3d is here: point cloud library (PCL). In: 2011 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 1–4 (2011)

  35. Salas-Moreno, R.F., Newcombe, R., Strasdat, H., Kelly, P.H., Davison, A.J., et al.: Slam++: simultaneous localisation and mapping at the level of objects. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 1352–1359 (2013)

  36. Schneider, P., Eberly, D.H.: Geometric Tools for Computer Graphics. Morgan Kaufmann, San Francisco (2002)

    Google Scholar 

  37. Schöps, T., Engel, J., Cremers, D. Semi-dense visual odometry for AR on a smartphone. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, pp. 145–150 (2014)

  38. Steinbrücker, F., Sturm, J., Cremers, D.: Volumetric 3d mapping in real-time on a cpu. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp. 2021–2028 (2014)

  39. Stühmer, J., Gumhold, S., Cremers, D.: Real-time dense geometry from a handheld camera. In: Joint Pattern Recognition Symposium, Springer, pp. 11–20 (2010)

  40. Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of rgb-d slam systems. In: Proceedings of the International Conference on Intelligent Robot Systems (IROS) (2012)

  41. Weikersdorfer, D., Gossow, D., Beetz, M.: Depth-adaptive superpixels. In: 2012 21st International Conference on Pattern Recognition (ICPR), IEEE, pp. 2087–2090 (2012)

Download references

Acknowledgements

The first author is supported by the Mexican National Council for Science and Technology (CONACyT) studentship number 624142. This work has been partially funded by the Royal Society through the Newton Advanced Fellowship with reference NA140454.

Author information

Authors and Affiliations

  1. National Institute of Astrophysics, Optics and Electronics, Puebla, Mexico

    Claudia Cruz-Martinez & José Martínez-Carranza

  2. University of Bristol, Bristol, UK

    Walterio Mayol-Cuevas

Authors
  1. Claudia Cruz-Martinez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. José Martínez-Carranza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Walterio Mayol-Cuevas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claudia Cruz-Martinez.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cruz-Martinez, C., Martínez-Carranza, J. & Mayol-Cuevas, W. Real-time enhancement of sparse 3D maps using a parallel segmentation scheme based on superpixels. J Real-Time Image Proc 14, 667–683 (2018). https://doi.org/10.1007/s11554-017-0707-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-017-0707-2

Keywords

Search

Navigation