Skip to main content
Springer Nature Link
Log in

LEROjD: Lidar Extended Radar-Only Object Detection

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15118))

Included in the following conference series:

  • 248 Accesses

Abstract

Accurate 3D object detection is vital for automated driving. While lidar sensors are well suited for this task, they are expensive and have limitations in adverse weather conditions. 3+1D imaging radar sensors offer a cost-effective, robust alternative but face challenges due to their low resolution and high measurement noise. Existing 3+1D imaging radar datasets include radar and lidar data, enabling cross-modal model improvements. Although lidar should not be used during inference, it can aid the training of radar-only object detectors. We explore two strategies to transfer knowledge from the lidar to the radar domain and radar-only object detectors: 1. multi-stage training with sequential lidar point cloud thin-out, and 2. cross-modal knowledge distillation. In the multi-stage process, three thin-out methods are examined. Our results show significant performance gains of up to 4.2% points in mean Average Precision with multi-stage training and up to 3.9% points with knowledge distillation by initializing the student with the teacher’s weights. The main benefit of these approaches is their applicability to other 3D object detection networks without altering their architecture, as we show by analyzing it on two different object detectors. Our code is available at https://github.com/rst-tu-dortmund/lerojd.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Afouras, T., Chung, J.S., Zisserman, A.: ASR is all you need: cross-modal distillation for lip reading. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2143–2147. IEEE (2020)

    Google Scholar 

  2. Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48 (2009)

    Google Scholar 

  3. Brisken, S., Ruf, F., Höhne, F.: Recent evolution of automotive imaging radar and its information content. IET Radar Sonar Navig. 12(10), 1078–1081 (2018)

    Article  Google Scholar 

  4. Caesar, H., et al.: nuScenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11621–11631 (2020)

    Google Scholar 

  5. Chen, X., Zhang, T., Wang, Y., Wang, Y., Zhao, H.: FUTR3D: a unified sensor fusion framework for 3D detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 172–181 (2023)

    Google Scholar 

  6. Chen, Y., Liu, J., Zhang, X., Qi, X., Jia, J.: Voxelnext: fully sparse voxelnet for 3D object detection and tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21674–21683 (2023)

    Google Scholar 

  7. Cheng, H., Han, X., Jiang, H., He, D., Xiao, G.: PCB-randnet: rethinking random sampling for lidar semantic segmentation in autonomous driving scene. arXiv preprint arXiv:2209.13797 (2022)

  8. Choi, M., et al.: MSC-RAD4R: ROS-based automotive dataset with 4D radar. IEEE Robot. Autom. Lett. 8(11), 7194–7201 (2023)

    Article  Google Scholar 

  9. Danzer, A., Griebel, T., Bach, M., Dietmayer, K.: 2D car detection in radar data with pointnets. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 61–66 (2019)

    Google Scholar 

  10. Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y., Li, H.: Voxel R-CNN: towards high performance voxel-based 3D object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 1201–1209 (2021)

    Google Scholar 

  11. Deng, J., Chan, G., Zhong, H., Lu, C.X.: See beyond seeing: robust 3D object detection from point clouds via cross-modal feature augmentation. arXiv preprint arXiv:2309.17336 (2023)

  12. Ding, F., Palffy, A., Gavrila, D.M., Lu, C.X.: Hidden gems: 4D radar scene flow learning using cross-modal supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9340–9349 (2023)

    Google Scholar 

  13. Drews, F., Feng, D., Faion, F., Rosenbaum, L., Ulrich, M., Gläser, C.: Deepfusion: a robust and modular 3D object detector for lidars, cameras and radars. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 560–567. IEEE (2022)

    Google Scholar 

  14. Engels, F., Heidenreich, P., Wintermantel, M., Stäcker, L., Al Kadi, M., Zoubir, A.M.: Automotive radar signal processing: research directions and practical challenges. IEEE J. Sel. Top. Signal Process. 15(4), 865–878 (2021)

    Article  Google Scholar 

  15. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The kitti vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)

    Google Scholar 

  16. Gupta, S., Hoffman, J., Malik, J.: Cross modal distillation for supervision transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2827–2836 (2016)

    Google Scholar 

  17. Wang, H., et al.: DSVT: dynamic sparse voxel transformer with rotated sets. In: CVPR (2023)

    Google Scholar 

  18. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: Deep Learning Workshop at NIPS (2014)

    Google Scholar 

  19. Hu, Q., et al.: Randla-net: efficient semantic segmentation of large-scale point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11108–11117 (2020)

    Google Scholar 

  20. Jiang, M., et al.: 4D high-resolution imagery of point clouds for automotive mmwave radar. IEEE Trans. Intell. Transp. Syst. 1–15 (2023)

    Google Scholar 

  21. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference for Learning Representations (2015)

    Google Scholar 

  22. Klingner, M., et al.: X3KD: knowledge distillation across modalities, tasks and stages for multi-camera 3D object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13343–13353 (2023)

    Google Scholar 

  23. Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: fast encoders for object detection from point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  24. Lang, I., Manor, A., Avidan, S.: Samplenet: differentiable point cloud sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7578–7588 (2020)

    Google Scholar 

  25. Li, D., Wei, Y., Zhu, R.: A comparative study on point cloud down-sampling strategies for deep learning-based crop organ segmentation. Plant Methods 19(1), 124 (2023)

    Article  Google Scholar 

  26. Li, J., et al.: BEV-LGKD: a unified lidar-guided knowledge distillation framework for multi-view BEV 3D object detection. IEEE Trans. Intell. Veh. 1–10 (2023)

    Google Scholar 

  27. Li, J., Luo, C., Yang, X.: Pillarnext: rethinking network designs for 3D object detection in lidar point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17567–17576 (2023)

    Google Scholar 

  28. Li, Q., Jin, S., Yan, J.: Mimicking very efficient network for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6356–6364 (2017)

    Google Scholar 

  29. Li, Z., Li, Y., Wang, Y., Xie, G., Qu, H., Lyu, Z.: A lightweight model for 3D point cloud object detection. Appl. Sci. 13(11) (2023)

    Google Scholar 

  30. Liu, J., Zhao, Q., Xiong, W., Huang, T., Han, Q.L., Zhu, B.: Smurf: spatial multi-representation fusion for 3D object detection with 4D imaging radar. IEEE Trans. Intell. Veh. 1–14 (2023)

    Google Scholar 

  31. Mao, J., et al.: One million scenes for autonomous driving: once dataset. In: Vanschoren, J., Yeung, S. (eds.) Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1 (2021)

    Google Scholar 

  32. Meyer, M., Kuschk, G.: Automotive radar dataset for deep learning based 3D object detection. In: 2019 16th European Radar Conference (EuRAD), pp. 129–132. IEEE (2019)

    Google Scholar 

  33. Nguyen, C.H., Nguyen, T.C., Tang, T.N., Phan, N.L.: Improving object detection by label assignment distillation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1005–1014 (2022)

    Google Scholar 

  34. Nobis, F., Shafiei, E., Karle, P., Betz, J., Lienkamp, M.: Radar voxel fusion for 3D object detection. Appl. Sci. 11(12) (2021)

    Google Scholar 

  35. Paek, D.H., Kong, S.H., Wijaya, K.T.: K-radar: 4D radar object detection for autonomous driving in various weather conditions. In: Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022)

    Google Scholar 

  36. Palffy, A., Pool, E., Baratam, S., Kooij, J.F.P., Gavrila, D.M.: Multi-class road user detection with 3+1D radar in the view-of-delft dataset. IEEE Robot. Autom. Lett. 7(2), 4961–4968 (2022)

    Article  Google Scholar 

  37. Palmer, P., Krueger, M., Altendorfer, R., Adam, G., Bertram, T.: Reviewing 3D object detectors in the context of high-resolution 3+1D radar. In: Workshop on 3D Vision and Robotics at the Conference on Computer Vision and Pattern Recognition 2023 (2023)

    Google Scholar 

  38. Palmer, P., Krueger, M., Altendorfer, R., Bertram, T.: Ego-motion estimation and dynamic motion separation from 3D point clouds for accumulating data and improving 3D object detection. In: AmE 2023 - Automotive meets Electronics; 14. GMM Symposium, pp. 86–91 (2023)

    Google Scholar 

  39. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  40. Rebut, J., Ouaknine, A., Malik, W., Pérez, P.: Raw high-definition radar for multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17021–17030 (2022)

    Google Scholar 

  41. Shi, S., et al.: PV-RCNN: point-voxel feature set abstraction for 3D object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10529–10538 (2020)

    Google Scholar 

  42. Shi, S., Wang, X., Li, H.: Pointrcnn: 3D object proposal generation and detection from point cloud. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–779 (2019)

    Google Scholar 

  43. Smith, L.N., Topin, N.: Super-convergence: very fast training of neural networks using large learning rates. In: Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, vol. 11006, pp. 369–386. SPIE (2019)

    Google Scholar 

  44. Stephan, M., Hazra, S., Santra, A., Weigel, R., Fischer, G.: People counting solution using an FMCW radar with knowledge distillation from camera data. In: 2021 IEEE Sensors, pp. 1–4 (2021)

    Google Scholar 

  45. Sun, P., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  46. Sun, P., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2446–2454 (2020)

    Google Scholar 

  47. Sun, S., Zhang, Y.D.: 4D automotive radar sensing for autonomous vehicles: a sparsity-oriented approach. IEEE J. Sel. Top. Signal Process. 15(4), 879–891 (2021)

    Article  Google Scholar 

  48. Tan, B., et al.: 3D object detection for multi-frame 4D automotive millimeter-wave radar point cloud. IEEE Sens. J. 23(11), 11125–11138 (2022)

    Article  Google Scholar 

  49. Tung, F., Mori, G.: Similarity-preserving knowledge distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1365–1374 (2019)

    Google Scholar 

  50. Wang, L., et al.: Multi-modal and multi-scale fusion 3D object detection of 4D radar and lidar for autonomous driving. IEEE Trans. Veh. Technol. 72(5), 5628–5641 (2023)

    Article  Google Scholar 

  51. Wang, T., Yuan, L., Zhang, X., Feng, J.: Distilling object detectors with fine-grained feature imitation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4933–4942 (2019)

    Google Scholar 

  52. Wei, Y., Wei, Z., Rao, Y., Li, J., Zhou, J., Lu, J.: Lidar distillation: bridging the beam-induced domain gap for 3D object detection. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13699, pp. 179–195. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19842-7_11

    Chapter  Google Scholar 

  53. Wu, H., Wen, C., Shi, S., Li, X., Wang, C.: Virtual sparse convolution for multimodal 3D object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21653–21662 (2023)

    Google Scholar 

  54. Xiao, A., Huang, J., Guan, D., Zhan, F., Lu, S.: Transfer learning from synthetic to real lidar point cloud for semantic segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 2795–2803 (2022)

    Google Scholar 

  55. Xiao, Y., Liu, Y., Luan, K., Cheng, Y., Chen, X., Lu, H.: Deep lidar-radar-visual fusion for object detection in urban environments. Remote Sens. 15(18) (2023)

    Google Scholar 

  56. Xiong, W., Liu, J., Huang, T., Han, Q.L., Xia, Y., Zhu, B.: LXL: lidar excluded lean 3D object detection with 4D imaging radar and camera fusion. IEEE Trans. Intell. Veh. 1–14 (2023)

    Google Scholar 

  57. Xu, B., et al.: RPFA-Net: a 4D radar pillar feature attention network for 3D object detection. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3061–3066 (2021)

    Google Scholar 

  58. Yan, Q., Wang, Y.: MVFAN: multi-view feature assisted network for 4D radar object detection. In: Luo, B., Cheng, L., Wu, Z.G., Li, H., Li, C. (eds.) ICONIP 2023. LNCS, vol. 14450, pp. 493–511. Springer, Cham (2023). https://doi.org/10.1007/978-981-99-8070-3_38

    Chapter  Google Scholar 

  59. Yang, J., Shi, S., Ding, R., Wang, Z., Qi, X.: Towards efficient 3D object detection with knowledge distillation. In: Advances in Neural Information Processing Systems, vol. 35, pp. 21300–21313. Curran Associates, Inc. (2022)

    Google Scholar 

  60. Yang, Z., Sun, Y., Liu, S., Jia, J.: 3DSSD: point-based 3D single stage object detector. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11040–11048 (2020)

    Google Scholar 

  61. You, Y., et al.: Pseudo-lidar++: accurate depth for 3D object detection in autonomous driving. In: ICLR (2020)

    Google Scholar 

  62. Zamanakos, G., Tsochatzidis, L., Amanatiadis, A., Pratikakis, I.: A comprehensive survey of lidar-based 3D object detection methods with deep learning for autonomous driving. Comput. Graph. 99, 153–181 (2021)

    Article  Google Scholar 

  63. Zhang, J., et al.: NTU4DRadLM: 4D radar-centric multi-modal dataset for localization and mapping. In: 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC) (2023)

    Google Scholar 

  64. Zhang, L., Dong, R., Tai, H.S., Ma, K.: Pointdistiller: structured knowledge distillation towards efficient and compact 3D detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21791–21801 (2023)

    Google Scholar 

  65. Zhang, X., et al.: Dual radar: a multi-modal dataset with dual 4D radar for autononous driving. arXiv preprint arXiv:2310.07602 (2023)

  66. Zhang, Y., Hu, Q., Xu, G., Ma, Y., Wan, J., Guo, Y.: Not all points are equal: learning highly efficient point-based detectors for 3D lidar point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18953–18962 (2022)

    Google Scholar 

  67. Zhao, M., et al.: Through-wall human pose estimation using radio signals. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7356–7365 (2018)

    Google Scholar 

  68. Zheng, L., et al.: Rcfusion: fusing 4-D radar and camera with bird’s-eye view features for 3-D object detection. IEEE Trans. Instrum. Meas. 72, 1–14 (2023)

    Google Scholar 

  69. Zheng, L., et al.: TJ4DRadSet: a 4D radar dataset for autonomous driving. In: 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp. 493–498 (2022)

    Google Scholar 

  70. Zhou, T., Chen, J., Shi, Y., Jiang, K., Yang, M., Yang, D.: Bridging the view disparity between radar and camera features for multi-modal fusion 3D object detection. IEEE Trans. Intell. Veh. 8(2), 1523–1535 (2023)

    Google Scholar 

  71. Zhou, Y., Tuzel, O.: Voxelnet: end-to-end learning for point cloud based 3D object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4490–4499 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Control Theory and Systems Engineering, TU Dortmund University, Dortmund, Germany

    Patrick Palmer, Martin Krüger, Stefan Schütte & Torsten Bertram

  2. ZF Group, Friedrichshafen, Germany

    Richard Altendorfer & Ganesh Adam

Authors
  1. Patrick Palmer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martin Krüger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Schütte
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard Altendorfer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ganesh Adam
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Torsten Bertram
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Patrick Palmer .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 478 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Palmer, P., Krüger, M., Schütte, S., Altendorfer, R., Adam, G., Bertram, T. (2025). LEROjD: Lidar Extended Radar-Only Object Detection. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15118. Springer, Cham. https://doi.org/10.1007/978-3-031-73027-6_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73027-6_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73026-9

  • Online ISBN: 978-3-031-73027-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics