Abm Tariqul Islam
Oliver Staadt
ABSTRACT
The depth images from RGB-D cameras contain a substantial amount of artifacts such as holes and flickering. Moreover, for fast moving objects in successive frames, we perceive ghosting artifacts on the depth images. Hence, the poor quality of the depth images limits them to be used in various applications. Here, we propose a gradient based spatial and temporal method of depth enhancement (gSMOOTH) using least median of squares, which deals with these artifacts. For each depth pixel over a sequence of frames, we look for invalid or unstable or drastically changed depth values and use our approach to replace those values with stable and more feasible depth values. Our approach removes the ghosting artifacts and flickering, and attenuates the amount of temporal noise significantly in real time. We conduct experiments with our own- and reference datasets and evaluate our method against reference methods. Experimental results show improvements for both static and dynamic scenes.
- Abdenour Amamra and Nabil Aouf. 2014. GPU-based real-time RGBD data filtering. Journal of Real-Time Image Processing (2014), 1--18. Google Scholar
Digital Library
- Razmik Avetisyan, Christian Rosenke, Martin Luboschik, and Oliver Staadt. 2016. Temporal Filtering of Depth Images using Optical Flow. In 24th WSCG'16.Google Scholar
- S. Beck, A. Kunert, A. Kulik, and B. Froehlich. 2013. Immersive Group-to-Group Telepresence. IEEE Transactions on Visualization and Computer Graphics 19, 4 (April 2013), 616--625. Google ScholarDigital Library
- Massimo Camplani, Tomas Mantecon, and Luis Salgado. 2013. Depth-color fusion strategy for 3-d scene modeling with kinect. IEEE Transactions on Cybernetics 43, 6 (2013), 1560--1571.Google ScholarCross Ref
- Massimo Camplani and Luis Salgado. 2012. Efficient spatiotemporal hole filling strategy for Kinect depth maps. In Proc. SPIE, Vol. 8290. 82900E--82900E--10.Google Scholar
- Li Chen, Hui Lin, and Shutao Li. 2012. Depth image enhancement for Kinect using region growing and bilateral filter. In 21st International Conference on Pattern Recognition. 3070--3073.Google Scholar
- J Dolson, Jongmin Baek, C Plagemann, and S Thrun. 2010. Up-sampling Range Data in Dynamic Environments. In IEEE Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Mingsong Dou and H. Fuchs. 2014. Temporally enhanced 3D capture of room-sized dynamic scenes with commodity depth cameras. In IEEE Virtual Reality (VR). 39--44.Google Scholar
- F. Garcia, D. Aouada, T. Solignac, B. Mirbach, and B. Ottersten. 2013. Real-time depth enhancement by fusion for RGB-D cameras. Computer Vision, IET 7, 5 (October 2013), 1--11.Google ScholarCross Ref
- Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided Image Filtering. In 11th European Conference on Computer Vision (ECCV), Springer Berlin Heidelberg. 1--14.Google Scholar
- Tak-Wai Hui and King Ngi Ngan. 2014. Motion-Depth: RGB-D Depth Map Enhancement with Motion and Depth in Complement. In IEEE CVPR. 3962--3969. Google ScholarDigital Library
- A. B. M. T. Islam, C. Scheel, R. Pajarola, and O. Staadt. 2015. Depth Image Enhancement using 1D Least Median of Squares. In Computer Graphics International (CGI'15).Google Scholar
- A. B. M. T. Islam, C. Scheel, R. Pajarola, and O. Staadt. 2015. Robust enhancement of depth images from Kinect sensor. In 2015 IEEE Virtual Reality (VR). 197--198.Google Scholar
- A. B. M. T. Islam, Christian Scheel, Renato Pajarola, and Oliver Staadt. 2017. Robust enhancement of depth images from depth sensors. Computers & Graphics 68, Supplement C (2017), 53--65. Google ScholarDigital Library
- Kourosh Khoshelham and Er Oude Elberink. 2012. Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12, 2 (2012), 1437--1454.Google ScholarCross Ref
- Sung-Yeol Kim, J.-H. Cho, A. Koschan, and M.A. Abidi. 2010. Spatial and Temporal Enhancement of Depth Images Captured by a Time-of-Flight Depth Sensor. In 20th International Conference on Pattern Recognition (ICPR). 2358--2361. Google ScholarDigital Library
- Bor-Shing Lin, Mei-Ju Su, Po-Hsun Cheng, Po-Jui Tseng, and Sao-Jie Chen. 2015. Temporal and Spatial Denoising of Depth Maps. Sensors 15, 8 (2015), 18506--18525.Google ScholarCross Ref
- Junyi Liu, Xiaojin Gong, and Jilin Liu. 2012. Guided inpainting and filtering for Kinect depth maps. In 21st International Conference on Pattern Recognition (ICPR). 2055--2058.Google Scholar
- Andrew Maimone, Jonathan Bidwell, Kun Peng, and Henry Fuchs. 2012. Enhanced personal autostereoscopic telepresence system using commodity depth cameras. Computers & Graphics 36, 7 (2012), 791--807. Google ScholarDigital Library
- Andrew Maimone and Henry Fuchs. 2011. Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras. In 10th IEEE ISMAR. 137--146. Google ScholarDigital Library
- Sergey Matyunin, Dmitriy Vatolin, Yury Berdnikov, and Michail Smirnov. 2011. Temporal filtering for depth maps generated by kinect depth camera. In 3DTV Conference. 1--4.Google Scholar
- Middlebury Datasets. 2013. http://vision.middlebury.edu/stereo/data/. Accessed: 16-01-2018.Google Scholar
- Simone Milani and Giancarlo Calvagno. 2016. Correction and interpolation of depth maps from structured light infrared sensors. Signal Processing: Image Communication 41 (2016), 28--39. Google ScholarDigital Library
- C.V. Nguyen, S. Izadi, and D. Lovell. 2012. Modeling Kinect Sensor Noise for Improved 3D Reconstruction and Tracking. In 2nd International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (3DIMPVT). 524--530. Google ScholarDigital Library
- Fei Qi, Junyu Han, Pengjin Wang, Guangming Shi, and Fu Li. 2013. Structure Guided Fusion for Depth Map Inpainting. Pattern Recognition Letter 34, 1 (Jan. 2013), 70--76. Google ScholarDigital Library
- Christian Richardt, Carsten Stoll, Neil A. Dodgson, Hans-Peter Seidel, and Christian Theobalt. 2012. Coherent Spatiotemporal Filtering, Upsampling and Rendering of RGBZ Videos. Computer Graphics Forum 31, 2 (2012), 247--256. Google ScholarDigital Library
- Peter J Rousseeuw. 1984. Least median of squares regression. Journal of the American statistical association 79, 388 (1984), 871--880.Google ScholarCross Ref
- P. J. Rousseeuw and A. M. Leroy. 1987. Robust Regression and Outlier Detection. John Wiley & Sons, Inc., New York, NY, USA. Google ScholarDigital Library
- L. Sheng and K. N. Ngan. 2013. Depth enhancement based on hybrid geometric hole filling strategy. In IEEE ICIP. 2173--2176.Google Scholar
- Zhongyuan Wang, Jinhui Hu, ShiZheng Wang, and Tao Lu. 2015. Trilateral Constrained Sparse Representation for Kinect Depth Hole Filling. Pattern Recognition Letter 65, C (2015), 95--102. Google ScholarDigital Library
- J. Wasza, S. Bauer, and J. Hornegger. 2011. Real-time preprocessing for dense 3-D range imaging on the GPU: Defect interpolation, bilateral temporal averaging and guided filtering. In International Conference on Computer Vision Workshops. 1221--1227.Google Scholar
- J. Yang, X. Ye, K. Li, C. Hou, and Y. Wang. 2014. Color-Guided Depth Recovery From RGB-D Data Using an Adaptive Autoregressive Model. IEEE Transactions on Image Processing 23, 8 (Aug 2014), 3443--3458.Google ScholarCross Ref
- Qingxiong Yang, N. Ahuja, Ruigang Yang, Kar-Han Tan, J. Davis, B. Culbertson, J. Apostolopoulos, and Gang Wang. 2013. Fusion of Median and Bilateral Filtering for Range Image Upsampling. IEEE Transactions on Image Processing 22, 12 (Dec 2013), 4841--4852. Google ScholarDigital Library
- You Yang, Qiong Liu, Rongrong Ji, and Yue Gao. 2012. Dynamic 3D Scene Depth Reconstruction via Optical Flow Field Rectification. Public Library of Science 7, 11 (2012), e47041.Google Scholar
- L. Zhang, P. Shen, S. Zhang, J. Song, and G. Zhu. 2016. Depth enhancement with improved exemplar-based inpainting and joint trilateral guided filtering. In IEEE ICIP. 4102--4106.Google Scholar
- Jiejie Zhu, Liang Wang, Ruigang Yang, and J. Davis. 2008. Fusion of time-of-flight depth and stereo for high accuracy depth maps. In IEEE CVPR. 1--8.Google Scholar
Index Terms
- gSMOOTH: A Gradient based Spatial and Temporal Method of Depth Image Enhancement
Recommendations
Robust enhancement of depth images from depth sensors
Depth images from depth cameras contain artifacts due to noisy depth measurements.A new 1D LMedS (Least Median of Squares) method of depth image enhancement.A fast and computationally inexpensive data traversal method for 1D LMedS.Enhancement of both ...
Accurate depth image generation via overfit training of point cloud registration using local frame sets
AbstractAccurate three-dimensional perception is a fundamental task in various computer vision applications. Recently, commercial RGB-depth (RGB-D) cameras have been widely adopted as single-view depth-sensing devices owing to their efficient ...
Highlights- We propose a method for a high-quality real-world depth dataset using only RGB-D stream dataset.
Spatial and Temporal Enhancement of Depth Images Captured by a Time-of-Flight Depth Sensor
ICPR '10: Proceedings of the 2010 20th International Conference on Pattern RecognitionIn this paper, we present a new method to enhance depth images captured by a time-of-flight (TOF) depth sensor spatially and temporally. In practice, depth images obtained from TOF depth sensors have critical problems, such as optical noise existence, ...
Comments