Skip to main content
SpringerLink
Log in

Regional feature fusion for on-road detection of objects using camera and 3D-LiDAR in high-speed autonomous vehicles

  • Application of soft computing
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Autonomous vehicles require accurate, and fast decision-making perception systems to know the driving environment. The 2D object detection is critical in allowing the perception system to know the environment. However, 2D object detection lacks depth information, which are crucial for understanding the driving environment. Therefore, 3D object detection is essential for the perception system of autonomous vehicles to predict the location of objects and understand the driving environment. The 3D object detection also faces challenges because of scale changes, and occlusions. Therefore in this study, a novel object detection method is presented that fuses the complementary information of 2D and 3D object detection to accurately detect objects in autonomous vehicles. Firstly, the aim is to project the 3D-LiDAR data into image space. Secondly, the regional proposal network (RPN) to produce a region of interest (ROI) is utilised. The ROI pooling network is used to map the ROI into ResNet50 feature extractor to get a feature map of fixed size. To accurately predict the dimensions of all the objects, we fuse the features of the 3D-LiDAR with the regional features obtained from camera images. The fused features from 3D-LiDAR and camera images are employed as input to the faster-region based convolution neural network (Faster-RCNN) network for the detection of objects. The assessment results on the KITTI object detection dataset reveal that the method can accurately predict car, van, truck, pedestrian and cyclist with an average precision of 94.59%, 82.50%, 79.60%, 85.31%, 86.33%, respectively, which is better than most of the previous methods. Moreover, the average processing time of the proposed method is only 70 ms which meets the real-time demand of autonomous vehicles. Additionally, the proposed model runs at 15.8 frames per second (FPS), which is faster than state-of-the-art fusion methods for 3D-LiDAR and camera.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

References

  • Bashir F, Porikli F (2006) Performance evaluation of object detection and tracking systems. In: Proceedings 9th IEEE International Workshop on PETS pp 7–14

  • Cai Y, Luan T, Gao H et al. (2021) YOLOv4–5D: an effective and efficient object detector for autonomous driving, In: IEEE Transactions on Instrumentation and Measurement, vol. 70

  • Cao Z, Liu J, Zhou W, Jiao X, Yang D (2021) LiDAR-based object detection failure tolerated autonomous driving planning system, In: 2021 IEEE Intelligent Vehicles Symposium (IV), Nagoya, Japan, pp 122–128

  • Carranza-García M, Lara-Benítez P, García-Gutiérrez J, Riquelme JC (2021) “Enhancing object detection for autonomous driving by optimizing anchor generation and addressing class imbalance. Neurocomputing 449:229–244

    Article  Google Scholar 

  • Chandra R et al (2020) Forecasting Trajectory and behavior of road-agents using spectral clustering in graph-LSTMs. IEEE Robot Auto Lett 5(3):4882–4890

    Article  Google Scholar 

  • Chen J, Bai T (2020) Saanet: Spatial adaptive alignment network for object detection in automatic driving. Image vis Comput 94:103873

    Article  Google Scholar 

  • Chen X, Kundu K, Zhu Y, Ma H, Fidler S, Urtasun R (2018) 3D object proposals using stereo imagery for accurate object class detection. IEEE Trans Pattern Anal Mach Intell 40(5):1259–1272

    Article  Google Scholar 

  • Chen LC, Hermans A, Papandreou C, Schroff F, Wang P, Adam H (2018) Masklab: Instance segmentation by refining object detection with semantic and direction features. In: Proceedings of the IEEE conference on computer vision and pattern recognition pp 4013–4022.

  • Chen K et al. MVLidarNet: real-time multi-class scene understanding for autonomous driving using multiple views. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA, 2020, pp. 2288–2294,2020.

  • Choi JD, Kim MY (2023) A sensor fusion system with thermal infrared camera and LiDAR for autonomous vehicles and deep learning based object detection. ICT Express, 9(2): 222–227

  • Daniel A, Chandru Vignesh J, Muthu C (2023) Fully convolutional neural networks for LIDAR–camera fusion for pedestrian detection in autonomous vehicle. Multimed Tools Applications 82(25107–25130)

  • Ennajar A, Khouja N, Boutteau R, Tlili F (2021) Deep multi-modal object detection for autonomous driving. In: 2021 18th International Multi-Conference on Systems, Signals & Devices (SSD) (pp 7–11). IEEE

  • https://www.nuscenes.org/

  • http://www.cvlibs.net/datasets/kitti

  • Huang Y, Zhang H, A safety vehicle detection mechanism based on YOLOv5. In: 2021 IEEE 6th International Conference on Smart Cloud (SmartCloud) (pp 1–6). IEEE, November, 2021

  • Islam MM, Newaz AA (2020) Pedestrian detection for autonomous cars: occlusion handling by classifying body parts, In: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC) IEEE, 2020, pp 1433–1438

  • Jamuna S, Murthy Gm, Lai SC, Parameshachari BD, Sujata N, Patil KL, Hemalatha (2022) ObjectDetect: a real-time object detection framework for advanced driver assistant systems using YOLOv5, Wireless Communications and Mobile Computing, 2022, 9444360, 1–10.

  • Kaican L et al. Coda: a real-world road corner case dataset for object detection in autonomous driving.In: European Conference on Computer Vision. Cham: Springer Nature Switzerland, pp.406–423 2022.

  • Kiran BR, Saboh I, Talpeart V, Sallab A (2022) Deep reinforcement learning for autonomous driving: a survey. IEEE Trans Intell Transp Syst 23(6):4909–4926

    Article  Google Scholar 

  • Ku J, Mozifian M, Lee J, Harakeh A, Waslander SL (2018) Joint 3D proposal generation and object detection from view aggregation. IEEE Int Conf Intell Robot Syst 5750–5757:2018

    Google Scholar 

  • Kumar VR, Eising C, Witt C, Yogamani SK (2023) Surround-view fisheye camera perception for automated driving: overview, survey & challenges. IEEE Trans Intell Transp Syst 24(4):3638–3659

    Article  Google Scholar 

  • Li G, Fan W, Xie H, Qu X (2022) Detection of road objects based on camera sensors for autonomous driving in various traffic situations, In: IEEE Sensors Journal, 22(24): 24253–24263

  • Li G, Ji Z, Qu X, Rui Z, and Cao D (2022) Cross-domain object detection for autonomous driving: A stepwise domain adaptative YOLO approach. IEEE Transactions on Intelligent Vehicles 7(3): 603–615.

  • Li P, Zhao H (2021) Monocular 3D object detection using dual quadric for autonomous driving. Neurocomputing 441:151–160

    Article  Google Scholar 

  • Lim T-Y, Ansari A, Major B, Fontijne D, Hamilton M, Gowaikar R, Subramanian S Radar and camera early fusion for vehicle detection in advanced driver assistance systems. In: Machine learning for autonomous driving workshop at the 33rd conference on neural information processing systems, vol. 2, 2019

  • Luo C, Xiaodong C, Alan Yuille YQ (2021) Exploring Simple 3D Multi-Object Tracking for Autonomous Driving. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10468–10477

  • Meyer M, Kuschk G (2019) Deep learning based 3d object detection for automotive radar and camera, In: 2019 16th European Radar Conference (EuRAD). IEEE, 2019, pp. 133–136,https://www.nuscenes.org/

  • Muhammad K, Ullah A, Lloret J, Ser JD, de Albuquerque VHC (2021) Deep learning for safe autonomous driving: current challenges and future directions. IEEE Trans Intell Transp Syst 22(7):4316–4336

    Article  Google Scholar 

  • Nabati R, Qi H Center fusion: Center-based radar and camera fusion for 3d object detection. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2021, pp 1527–1536.

  • Ni J, Shen K, Chen Y, Cao W, Yang SX (2022) An improved deep network-based scene classification method for self-driving cars. IEEE Trans Instrum Meas 71:1–14

    Google Scholar 

  • Ni J, Shen K, Chen Y, Cao W, Yang SX An improved deep network-based scene classification method for self-driving cars, In: IEEE Transactions on Instrumentation and Measurement, 71, pp 1–14.,2022.

  • Niranjan DR, VinayKarthik BC, Mohana (2021) Deep learning based object detection model for autonomous driving research using CARLA simulator. In: 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 2021, pp 1251–1258

  • Nobis F, Shafiei E, Karle P, Betz J, Lienkamp M (2021) Radar voxel fusion for 3d object detection. Appl Sci 11(12):5598

    Article  Google Scholar 

  • Peng L, Wang H, Li J (2021) “Uncertainty evaluation of object detection algorithms for autonomous vehicles. Automotive Innovation 4(3):241–252

    Article  Google Scholar 

  • Rani S, Ghai D, Kumar SS Object detection and recognition using contour based edge detection and fast R-CNN. Multimedia Tools Applications 81(42183–42207) 2022.

  • Ruchay A, Dorofeev K, Kober A (2018) 3D object reconstruction using multiple Kinect sensors and initial estimation of sensor parameters. In: Applications of Digital Image Processing XLI (Vol. 10752, pp. 639–646). SPIE

  • Shi S, Wang X, Li H (2019) PointRCNN: 3D object proposal generation and detection from point cloud. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, 2019, pp. 770–779

  • Shi W, Rajkumar R (2020) Point-GNN: graph neural network for 3D object detection in a point cloud. Proc IEEE Comput Soc Conf Comput vis Pattern Recognit 2020:1708–1716

    Google Scholar 

  • Sun P, Kretzschmar H, Dotiwalla X, Chouard A, Patnaik V, Tsui P, Guo J, Zhou Y, Chai Y, Caine B et al. (2020) Scalability in perception for autonomous driving: Waymo open dataset, In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

  • Uribe C,, Méndez-Monroy AA (2022) U19-Net: a deep learning approach for obstacle detection in self-driving cars. Soft Comput 26: 5195–5207. http://www.cvlibs.net/datasets/kitti

  • Wang R, Wang Z, Xu Z, Wang C, Li Q, Zhang Y, Li H (2021) A real-time object detector for autonomous vehicles based on YOLOv4. Comput Intell Neurosci 2021(9218137):1–11

    Google Scholar 

  • Wang L et al (2023) Global perception-based robust parking space detection using a low-cost camera. IEEE Trans Intell Vehicles 8(2):1439–1448

    Article  Google Scholar 

  • Wang Z, Jia K, and Frustum (2019) ConvNet: sliding frustums to aggregate local point-wise features for amodal, IEEE Int. Conf. Intell. Robot. Syst. (Mar. 2019) 1742–1749

  • Wang Y, Mao Q, Zhu H, Deng J, Zhang Y (2023) Multi-modal 3d object detection in autonomous driving: a survey. Int J Comput Vis pp 1–31

  • Wen L-H, Jo K-H (2021) Fast and accurate 3D object detection for lidar-camera-based autonomous vehicles using one shared voxel-based backbone. IEEE Access 9:22080–22089

    Article  Google Scholar 

  • Yang B, Luo W, and Urtasun (2018) Pixor: Real-time 3d object detection from point clouds. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 7652–7660)

  • Yuxuan L, Yixuan Y, Liu M (2021) Ground-aware monocular 3d object detection for autonomous driving. In: IEEE Robotics and Automation Letters 6(2):919–926

  • Yuan Z, Song X, Bai L, Wang Z, Ouyang W (2022) Temporal-channel transformer for 3D lidar-based video object detection for autonomous driving. IEEE Trans Circuits Syst Video Technol 32(4):2068–2078

    Article  Google Scholar 

  • Zhao X, Sun P, Xu Z, Min H, Yu H (2020) Fusion of 3D LIDAR and camera data for object detection in autonomous vehicle applications. In: IEEE Sensors Journal, 20(9): 4901–4913

Download references

Funding

No funding was provided for the completion of this study.

Author information

Authors and Affiliations

  1. University of Chinese Academy of Sciences, Beijing, 100049, China

    Qingyu Wu & Xiaoxiao Li

  2. China Mobile (Zhejiang) Innovation Research Co., Ltd., Hangzhou, 310030, Zhejiang, China

    Kang Wang

  3. Department of Automation, University of Science and Technology of China, Hefei, 2300271, China

    Hazrat Bilal

Authors
  1. Qingyu Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoxiao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hazrat Bilal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoxiao Li.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose. The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any study with human participants or animals performed by the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Q., Li, X., Wang, K. et al. Regional feature fusion for on-road detection of objects using camera and 3D-LiDAR in high-speed autonomous vehicles. Soft Comput 27, 18195–18213 (2023). https://doi.org/10.1007/s00500-023-09278-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-023-09278-3

Keywords

Search

Navigation