Teo T. Niemirepo
ABSTRACT
This paper presents the first entirely open-source and cross-platform software called Open3DGen for reconstructing photorealistic textured 3D models from RGB-D images. The proposed software pipeline consists of nine main stages: 1) RGB-D acquisition; 2) 2D feature extraction; 3) camera pose estimation; 4) point cloud generation; 5) coarse mesh reconstruction; 6) optional loop closure; 7) fine mesh reconstruction; 8) UV unwrapping; and 9) texture projection. This end-to-end scheme combines multiple state-of-the-art techniques and provides an easy-to-use software package for real-time 3D model reconstruction and offline texture mapping. The main innovation lies in various Structure-from-Motion (SfM) techniques that are used with additional depth data to yield high-quality 3D models in real-time and at low cost. The functionality of Open3DGen has been validated on AMD Ryzen 3900X CPU and Nvidia GTX1080 GPU. This proof-of-concept setup attains an average processing speed of 15 fps for 720p (1280x720) RGBD input without the offline backend. Our solution is shown to provide competitive 3D mesh quality and execution performance with the state-of-the-art commercial and academic solutions.
- Agisoft Metashape. [Online]. Available: https://www.agisoft.com/.Google Scholar
- AliceVision Meshroom. [Online]. Available: https://alicevision.org#meshroom.Google Scholar
- S. Ullman, "The interpretation of structure from motion," in Proc. R. Soc. Lond., B203, Jan. 1979.Google Scholar
- P. Henry, M. Krainin, E. Herbst, X. Ren, and D. Fox, "RGB-D mapping: using Kinect-style depth cameras for dense 3D modeling of indoor environments," in Proc. Int. Symp. on Experimental Robot., Dec. 2010, New Delhi and Agra, India.Google Scholar
- D. R. dos Santos, M. A. Basso, K. Khoshelham, E. de Oliveira, N. L. Pavan, and G. Vosselman, "Mapping indoor spaces by adaptive coarse-to-fine registration of RGB-D data," IEEE Geosci. Remote Sens. Lett., vol. 13, no. 2, Feb. 2016, pp. 262--266.Google ScholarCross Ref
- A. Nüchter, K. Lingemann, J. Hertzberg, and H. Surmann, "6D SLAM---3D mapping outdoor environments," J. Field Robot., vol. 24, no. 8-9, Aug. 2007, pp. 699--722.Google ScholarDigital Library
- P. Newman, D. Cole, and K. Ho, "Outdoor SLAM using visual appearance and laser ranging," in Proc. IEEE Int. Conf. Robot. Autom., May 2006, Orlando, Florida, USA.Google Scholar
- D. M. Cole and P. M. Newman, "Using laser range data for 3D SLAM in outdoor environments," in Proc. IEEE Int. Conf. Robot. Autom., May 2006, Orlando, Florida, USA.Google Scholar
- J. Engel, T. Schöps, and D. Cremers, "LSD-SLAM: large-scale direct monocular SLAM," in Proc. European Conf. Comp. Vision, Sept. 2014, Zürich, Switzerland.Google ScholarCross Ref
- R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós, "ORB-SLAM: a versatile and accurate monocular SLAM system," IEEE Trans. on Robotics, vol. 31, no. 5, Oct. 2015, pp. 1147--1163.Google ScholarDigital Library
- R. Mur-Artal and J. D. Tardós. "ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras," IEEE Trans. on Robotics, vol. 33, no. 5, Oct. 2017, pp. 1255--1262.Google ScholarDigital Library
- C. Campos, R. Elvira, J. J. Gómez, J. M. M. Montiel, and J. D. Tardós, "ORB-SLAM3: an accurate open-source library for visual, visual-inertial and multi-map SLAM," arXiv preprint arXiv:2007.11898, July 2020.Google Scholar
- A. Rosinol, M. Abate, Y. Chang, and L. Carlone, "Kimera: an open-source library for real-time metric-semantic localization and mapping," in Proc. IEEE Int. Conf. Robot. Autom., Aug. 2020, Paris, France.Google ScholarCross Ref
- A. Dai, M. Niessner, M. Zollöfer, S. Izadi, and C. Theobalt, "BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration," ACM Trans. Graph., vol. 36, no. 3, June 2017, pp. 24:1--24:18.Google ScholarDigital Library
- M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger, "Real-time 3D reconstruction at scale using voxel hashing," ACM Trans. Graph., vol. 32, no. 6, Nov. 2013, pp. 169:1--169:11.Google ScholarDigital Library
- peel 2 3D scanner. [Online]. available: https://peel-3d.com/products/peel-2?variant=33046758522903.Google Scholar
- Artec Eva. [Online]. Available: https://www.artec3d.com/portable-3d-scanners/artec-eva-v2.Google Scholar
- S. Pertuz and D. Puig, "Analysis of focus measure operators for shape-from-focus," Pattern Recognition, vol. 46, no. 5, May 2013, pp. 1415--1432.Google ScholarDigital Library
- OpenCV: Open Computer Vision Library. [Online]. Available: https://opencv.org/.Google Scholar
- D. Lowe, "Distinctive image features from scale invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, Nov. 2004, pp. 91--110.Google ScholarDigital Library
- P. F. Alcantarilla, J. Nuevo, and A. Bartoli, "Fast explicit diffusion for accelerated features in nonlinear scale spaces," in Proc. British Mach. Vis. Conf., Sept. 2013, Bristol, United Kingdom.Google Scholar
- E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, "ORB: an efficient alternative to SIFT or SURF," in Proc. IEEE Int. Conf. Comp. Vis., Mar. 2011, Barcelona, Spain.Google ScholarDigital Library
- O. Chum, T. Pajdla, and P. Sturm, "The geometric error for homographies," Comput. Vis. Image Understanding, vol. 97, no. 1, Jan. 2005, pp. 86--102.Google ScholarDigital Library
- V. Lepetit, M. Moreno-Noguer, and P. Fua, "EPnP: an accurate O(n) solution to the PnP problem," Int. J. Comput. Vis., vol. 81, no. 2, Feb. 2009, pp. 155--166.Google ScholarDigital Library
- D. Nister, "An efficient solution to the five-point relative pose problem," IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 6, June 2004, pp. 756--777.Google ScholarDigital Library
- H. Aghajan and A. Cavallaro, "Multi-Camera Networks: Principles and Applications," Academic Press, Apr. 2009, Orlando, Florida, USA.Google Scholar
- M. Kazhdan, M. Bolitho, and H. Hoppe, "Poisson surface reconstruction," in Proc. Eurographics Symp. on Geometry Process., June 2006, Cagliari, Sardinia, Italy.Google Scholar
- J. Wenzel, M. Tarini, D. Panozzo, and O. Sorkine-Hornung, "Instant field-aligned meshes," ACM Trans. Graph., vol. 34, no. 6, Oct. 2015, pp. 189:1--189:15.Google Scholar
- F. Dellaert, "Factor graphs and GTSAM: A hands-on introduction," Georgia Institute of Technology, Sept. 2012.Google Scholar
- Q. Zhou, J. Park, and V. Koltun, "Open3D: a modern library for 3D data processing," arxiv.org/abs/1801.09847, Jan. 2018.Google Scholar
- F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, and G. Taubin, "The ball-pivoting algorithm for surface reconstruction," IEEE Trans. Vis. Comput. Graph., vol. 5, no. 4, Nov. 1999, pp. 349--359.Google ScholarDigital Library
- Xatlas. [Online]. Available: https://github.com/jpcy/xatlas.Google Scholar
- EuRoC MAV Dataset. [Online]. Available: https://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets.Google Scholar
Index Terms
- Open3DGen: open-source software for reconstructing textured 3D models from RGB-D images
Recommendations
TextureMe: High-Quality Textured Scene Reconstruction in Real Time
Three-dimensional (3D) reconstruction using an RGB-D camera has been widely adopted for realistic content creation. However, high-quality texture mapping onto the reconstructed geometry is often treated as an offline step that should run after geometric ...
Direct Reconstruction of Displaced Subdivision Surface from Unorganized Points
PG '01: Proceedings of the 9th Pacific Conference on Computer Graphics and ApplicationsIn this paper we propose a new mesh reconstruction algorithm that produces a displaced subdivision mesh directly from unorganized points. The displaced subdivision surface is a new mesh representation that defines a detailed mesh with a displacement map ...
A Real-Time and Globally Consistent Meshing Reconstruction Without GPU
Intelligent Robotics and ApplicationsAbstractReal-time 3D reconstruction is vital for various applications, such as human-robot interaction, virtual reality, and environment perception. The prevalence of low power devices and the rapid advancement of human-robot interaction techniques have ...
Comments