Abstract
Drivable road detection is a fundamental problem for autonomous vehicles. RGB cameras and LiDAR are the mostly used data sources in road detection. While cameras provide lots of useful visual information, LiDARs can provide precise altitude information without being affected by the ambient light. However, these sensors create images at different space and this causes a challenging fusion task when they are intended to be used together. In this study, a U-Net-based novel fusion set is developed to fuse the RGB and LiDAR images for road detection. The LiDAR images are pre-processed and transferred to the 2D image space before fusion. Then, U-NET model, which is effectively used in image segmentation applications, is adapted for three different fusion techniques: early fusion, late fusion and cross-fusion. Models are evaluated on the KITTI road detection dataset, and the developed early fusion model which fuses the RGB and altitude difference image achieved the highest MaxF score on road detection. The obtained results are also at a competitive level with state-of-the-art models.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
Assidiq, A.A.M., Khalifa, O.O., Islam, Md. R., Khan, S.: Real time lane detection for autonomous vehicles. In: 2008 International Conference on Computer and Communication Engineering, pp. 82–88 (2008)
Wang, H., Fan, R., Cai, P., Liu, M.: Sne-roadseg+: rethinking depth-normal translation and deep supervision for freespace detection. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1140–1145 (2021)
Muñoz-Bulnes, J., Fernandez, C., Parra, I., Fernández-Llorca, D., Sotelo, M.A.: Deep fully convolutional networks with random data augmentation for enhanced generalization in road detection. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 366–371 (2017)
Hillel, A.B., Lerner, R., Levi, D., Raz, G.: Recent progress in road and lane detection: a survey. Mach. Vis. Appl. 25, 04 (2014)
Gu, S., Zhang, Y., Tang, J., Yang, J., Kong, H.: Road detection through CRF based Lidar-camera fusion. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 3832–3838 (2019)
Han, X., Wang, H., Lu, J., Zhao, C.: Road detection based on the fusion of lidar and image data. Int. J. Adv. Robot. Syst. (2017). https://doi.org/10.1177/1729881417738102
Caltagirone, Luca, Bellone, Mauro, Svensson, Lennart, Wahde, Mattias: Lidar-camera fusion for road detection using fully convolutional neural networks. Robot. Auton. Syst. 111, 11 (2018)
Chen, Zhe, Zhang, Jing, Tao, Dacheng: Progressive lidar adaptation for road detection. IEEE/CAA J. Autom. Sin. 6(3), 693–702 (2019)
Gu, S., Yang, J., Kong, H.: A cascaded lidar-camera fusion network for road detection. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13308–13314 (2021)
Fritsch, J., Kühnl, T., Geiger, A.: A new performance measure and evaluation benchmark for road detection algorithms. In: 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013), pp. 1693–1700 (2013)
Chen, Liang-Chieh., Papandreou, George, Kokkinos, Iasonas, Murphy, Kevin, Yuille, Alan L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2018)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988 (2017)
Wu, H, Zhang, J., Huang, K., Liang, K., Yu, Y.: Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation. arXiv:1903.11816 (2019)
Öztürk, O., Sariturk, B., Seker, D.: Comparison of fully convolutional networks (FCN) and u-net for road segmentation from high resolution imageries. Int. J. Environ. Geoinform. 7, 272–279 (2020)
Abderrahim, N.Y.Q., Abderrahim, S., Rida, A.: Road segmentation using u-net architecture. In: 2020 IEEE International conference of Moroccan Geomatics (Morgeo), pp. 1–4 (2020)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention (MICCAI), vol. 9351 of LNCS, pp. 234–241. Springer, Berlin (2015) (available on arXiv:1505.04597 [cs.CV])
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. This study was supervised by Habil Kalkan. Implementation and evaluation were performed by Arda Taha Candan. The first draft of the manuscript was written by Arda Taha Candan and reviewed and edited by Habil Kalkan. All the authors have read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no conflicts of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Candan, A.T., Kalkan, H. U-Net-based RGB and LiDAR image fusion for road segmentation. SIViP 17, 2837–2843 (2023). https://doi.org/10.1007/s11760-023-02502-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-023-02502-5