Abstract
Robotic manipulation systems frequently utilize RGB-D cameras based on infrared projection to perceive three-dimensional environments. Unfortunately, this technique often fails on transparent objects such as glasses, bottles and plastic containers. We present methods to exploit the perceived infrared camera images to detect and reconstruct volumetric shapes of arbitrary transparent objects. Our reconstruction pipeline first segments transparent surfaces based on pattern scattering and absorption, followed by optimization-based multi-view reconstruction of volumetric object models. Outputs from the segmentation stage can also be utilized for single-view transparent object detection. The presented methods improve on previous work by analyzing infrared camera images directly and by successfully reconstructing cavities in objects such as drinking glasses. A dataset of recorded transparent objects, autonomously gathered by a robotic camera-in-hand setup, is published together with this work.
This research was funded by the German Research Foundation (DFG) and the National Science Foundation of China in project Crossmodal Learning, TRR-169.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Albrecht, S.: Transparent object reconstruction and registration confidence measures for 3D point clouds based on data inconsistency and viewpoint analysis. Ph.D. thesis, Osnabrück University (2018)
Chen, G., Han, K., Wong, K.Y.K.: TOM-Net: learning transparent object matting from a single image. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9233–9241, June 2018
Eren, G., et al.: Scanning from heating: 3D shape estimation of transparent objects from local surface heating. Opt. Express 17, 11457–11468 (2009). https://doi.org/10.1364/OE.17.011457
Ham, C., Singh, S., Lucey, S.: Occlusions are fleeting – texture is forever: moving past brightness constancy. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 273–281 (2017). https://doi.org/10.1109/WACV.2017.37
Ihrke, I., Kutulakos, K., Lensch, H., Magnor, M., Heidrich, W.: Transparent and specular object reconstruction. Comput. Graph. Forum 29, 2400–2426 (2010). https://doi.org/10.1111/j.1467-8659.2010.01753.x
Ji, Y., Xia, Q., Zhang, Z.: Fusing depth and silhouette for scanning transparent object with RGB-D sensor. Int. J. Opt. 2017, 9796127 (2017). https://doi.org/10.1155/2017/9796127
Klank, U., Carton, D., Beetz, M.: Transparent object detection and reconstruction on a mobile platform. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 5971–5978 (2011). https://doi.org/10.1109/ICRA.2011.5979793
Kriegel, S., Rink, C., Bodenmüller, T., Suppa, M.: Efficient next-best-scan planning for autonomous 3D surface reconstruction of unknown objects. J. Real-Time Image Proc. 10(4), 611–631 (2013). https://doi.org/10.1007/s11554-013-0386-6
Lysenkov, I., Eruhimov, V., Bradski, G.: Recognition and pose estimation of rigid transparent objects with a Kinect sensor. In: Proceedings of Robotics: Science and Systems, Sydney, Australia, July 2012. https://doi.org/10.15607/RSS.2012.VIII.035
Milan, A., et al.: Semantic segmentation from limited training data. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1908–1915 (2018). https://doi.org/10.1109/ICRA.2018.8461082
Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7263–7271, July 2017
Sajjan, S.S., et al.: ClearGrasp: 3D shape estimation of transparent objects for manipulation (2019)
Saygili, G., van der Maaten, L., Hendriks, E.: Hybrid Kinect depth map refinement for transparent objects. In: Proceedings of the International Conference on Pattern Recognition, pp. 2751–2756 (2014). https://doi.org/10.1109/ICPR.2014.474
Tsai, D., Dansereau, D.G., Peynot, T., Corke, P.: Distinguishing refracted features using light field cameras with application to structure from motion. IEEE Robot. Autom. Lett. 4(2), 177–184 (2019). https://doi.org/10.1109/LRA.2018.2884765
Wu, B., Zhou, Y., Qian, Y., Cong, M., Huang, H.: Full 3D reconstruction of transparent objects. ACM Trans. Graph. 37(4), 103:1–103:11 (2018). https://doi.org/10.1145/3197517.3201286
Yun, Y., Seo, D., Kim, D.: Recognition of transparent objects using 3D depth camera. In: 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), pp. 882–883, June 2017. https://doi.org/10.1109/URAI.2017.7992854
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Ruppel, P., Görner, M., Hendrich, N., Zhang, J. (2021). Detection and Reconstruction of Transparent Objects with Infrared Projection-Based RGB-D Cameras. In: Sun, F., Liu, H., Fang, B. (eds) Cognitive Systems and Signal Processing. ICCSIP 2020. Communications in Computer and Information Science, vol 1397. Springer, Singapore. https://doi.org/10.1007/978-981-16-2336-3_53
Download citation
DOI: https://doi.org/10.1007/978-981-16-2336-3_53
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-2335-6
Online ISBN: 978-981-16-2336-3
eBook Packages: Computer ScienceComputer Science (R0)