skip to main content
10.1145/3495243.3560539acmconferencesArticle/Chapter ViewAbstractPublication PagesmobicomConference Proceedingsconference-collections
research-article

VIPS: real-time perception fusion for infrastructure-assisted autonomous driving

Published: 14 October 2022 Publication History

Abstract

Infrastructure-assisted autonomous driving is an emerging paradigm that expects to significantly improve the driving safety of autonomous vehicles. The key enabling technology for this vision is to fuse LiDAR results from the roadside infrastructure and the vehicle to improve the vehicle's perception in real time. In this work, we propose VIPS, a novel lightweight system that can achieve decimeter-level and real-time (up to 100 ms) perception fusion between driving vehicles and roadside infrastructure. The key idea of VIPS is to exploit highly efficient matching of graph structures that encode objects' lean representations as well as their relationships, such as locations, semantics, sizes, and spatial distribution. Moreover, by leveraging the tracked motion trajectories, VIPS can maintain the spatial and temporal consistency of the scene, which effectively mitigates the impact of asynchronous data frames and unpredictable communication/compute delays. We implement VIPS end-to-end based on a campus smart lamppost testbed. To evaluate the performance of VIPS under diverse situations, we also collect two new multi-view point cloud datasets using the smart lamppost testbed and an autonomous driving simulator, respectively. Experiment results show that VIPS can extend the vehicle's perception range by 140% within 58 ms on average, and delivers a 4X improvement in perception fusion accuracy and 47X data transmission saving over existing approaches. A video demo of VIPS based on the lamppost dataset is available at https://youtu.be/zW4oi_EWOu0.

References

[1]
2021. Self-driving technology: Automated Transportation Systems. https://www.tusimple.com/technology/
[2]
2021. Velodyne's HDL-32E surround lidar sensor. https://velodynelidar.com/products/hdl-32e/
[3]
2022. Nvidia TENSORRT. https://developer.nvidia.com/tensorrt
[4]
2022. The providentia++ project. https://innovation-mobility.com/en/project-providentia/
[5]
2022. Street light. https://en.wikipedia.org/wiki/Street_light
[6]
[n.d.]. Apollo. https://apollo.auto/.
[7]
[n.d.]. The Autoware Foundation - open source for autonomous driving. https://www.autoware.org/
[8]
[n.d.]. Carla Documentation. https://carla.readthedocs.io/en/stable/
[9]
[n.d.]. The MEC-view system. https://www.uni-due.de/~hp0309/index.php/en/project-issues
[10]
[n.d.]. Open Neural Network Exchange. https://onnx.ai/
[11]
[n.d.] url=https://www.livoxtech.com/horizon,journal=Livox. Horizon.
[12]
Fawad Ahmad, Hang Qiu, Ray Eells, Fan Bai, and Ramesh Govindan. 2020. {CarMap}: Fast 3D Feature Map Updates for Automobiles. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). 1063--1081.
[13]
PhD Alessandro Lori. 2019. Are self-driving cars safe? https://www.verizonconnect.com/resources/article/are-self-driving-cars-safe/
[14]
Eduardo Arnold, Mehrdad Dianati, Robert de Temple, and Saber Fallah. 2020. Cooperative perception for 3D object detection in driving scenarios using infrastructure sensors. IEEE Transactions on Intelligent Transportation Systems (2020).
[15]
Erkan Baser, Venkateshwaran Balasubramanian, Prarthana Bhattacharyya, and Krzysztof Czarnecki. 2019. Fantrack: 3d multi-object tracking with feature association network. In 2019 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1426--1433.
[16]
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. 2016. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP). IEEE, 3464--3468.
[17]
Michael Buchholz, Johannes Christian Muller, Martin Herrmann, Jan Strohbeck, Benjamin Volz, Matthias Maier, Jonas Paczia, Oliver Stein, Hubert Rehborn, and Rudiger-Walter Henn. 2021. Handling Occlusions in Automated Driving Using a Multiaccess Edge Computing Server-Based Environment Model From Infrastructure Sensors. IEEE Intelligent Transportation Systems Magazine (2021).
[18]
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. 2020. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11621--11631.
[19]
Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. 2019. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing. 88--100.
[20]
Qi Chen, Sihai Tang, Qing Yang, and Song Fu. 2019. Cooper: Cooperative perception for connected autonomous vehicles based on 3d point clouds. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, 514--524.
[21]
Titus Cieslewski, Siddharth Choudhary, and Davide Scaramuzza. 2018. Data-efficient decentralized visual SLAM. In 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2466--2473.
[22]
Christian Creß and Alois C Knoll. 2021. Intelligent Transportation Systems With The Use of External Infrastructure: A Literature Survey. arXiv preprint arXiv:2112.05615 (2021).
[23]
Vinayak V Dixit, Sai Chand, and Divya J Nair. 2016. Autonomous vehicles: disengagements, accidents and reaction times. PLoS one 11, 12 (2016), e0168054.
[24]
Zhen Dong, Fuxun Liang, Bisheng Yang, Yusheng Xu, Yufu Zang, Jianping Li, Yuan Wang, Wenxia Dai, Hongchao Fan, Juha Hyyppä, et al. 2020. Registration of large-scale terrestrial laser scanner point clouds: A review and benchmark. ISPRS Journal of Photogrammetry and Remote Sensing 163 (2020), 327--342.
[25]
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Conference on robot learning. PMLR, 1--16.
[26]
Bertrand Douillard, A Quadros, Peter Morton, James Patrick Underwood, Mark De Deuge, S Hugosson, M Hallström, and Tim Bailey. 2012. Scan segments matching for pairwise 3D alignment. In 2012 IEEE International Conference on Robotics and Automation. IEEE, 3033--3040.
[27]
Stephan Eichler. 2007. Performance evaluation of the IEEE 802.11 p WAVE communication standard. In 2007 IEEE 66th Vehicular Technology Conference. IEEE, 2199--2203.
[28]
Davi Frossard and Raquel Urtasun. 2018. End-to-end learning of multi-sensor 3d tracking by detection. In 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 635--642.
[29]
Michael Gabb, Holger Digel, Tobias Müller, and Rüdiger-Walter Henn. 2019. Infrastructure-supported perception and track-level fusion using edge computing. In 2019 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1739--1745.
[30]
Wei Gao and Russ Tedrake. 2019. Filterreg: Robust and efficient probabilistic point-set registration using gaussian filter and twist parameterization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11095--11104.
[31]
Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition. IEEE, 3354--3361.
[32]
Yuze He, Li Ma, Zhehao Jiang, Yi Tang, and Guoliang Xing. 2021. VI-eye: semantic-based 3D point cloud registration for infrastructure-assisted autonomous driving. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 573--586.
[33]
Rudolph Emil Kalman. 1960. A new approach to linear filtering and prediction problems. (1960).
[34]
Aleksandr Kim, Aljoša Ošep, and Laura Leal-Taixé. 2021. Eagermot: 3d multi-object tracking via sensor fusion. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 11315--11321.
[35]
Kenji Koide, Masashi Yokozuka, Shuji Oishi, and Atsuhiko Banno. 2021. Voxelized gicp for fast and accurate 3d point cloud registration. In 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 11054--11059.
[36]
Harold W Kuhn. 1955. The Hungarian method for the assignment problem. Naval research logistics quarterly 2, 1--2 (1955), 83--97.
[37]
Sampo Kuutti, Saber Fallah, Konstantinos Katsaros, Mehrdad Dianati, Francis Mccullough, and Alexandros Mouzakitis. 2018. A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications. IEEE Internet of Things Journal 5, 2 (2018), 829--846.
[38]
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. 2019. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12697--12705.
[39]
Marius Leordeanu and Martial Hebert. 2005. A spectral technique for correspondence problems using pairwise constraints. (2005).
[40]
E Li, Shuaijun Wang, Chengyang Li, Dachuan Li, Xiangbin Wu, and Qi Hao. 2020. SUSTech POINTS: A Portable 3D Point Cloud Interactive Annotation Platform System. In 2020 IEEE Intelligent Vehicles Symposium (IV). 1108--1115.
[41]
Jianqiang Li, Genqiang Deng, Chengwen Luo, Qiuzhen Lin, Qiao Yan, and Zhong Ming. 2016. A hybrid path planning method in unmanned air/ground vehicle (UAV/UGV) cooperative systems. IEEE Transactions on Vehicular Technology 65, 12 (2016), 9585--9596.
[42]
Kok-Lim Low. 2004. Linear least-squares optimization for point-to-plane icp surface registration. Chapel Hill, University of North Carolina 4, 10 (2004), 1--3.
[43]
Qian Luo, Yurui Cao, Jiajia Liu, and Abderrahim Benslimane. 2019. Localization and navigation in autonomous driving: Threats and countermeasures. IEEE Wireless Communications 26, 4 (2019), 38--45.
[44]
Gaurang Naik, Biplav Choudhury, and Jung-Min Park. 2019. IEEE 802.11 bd & 5G NR V2X: Evolution of radio access technologies for V2X communications. IEEE access 7 (2019), 70169--70184.
[45]
David Prokhorov, Dmitry Zhukov, Olga Barinova, Konushin Anton, and Anna Vorontsova. 2019. Measuring robustness of Visual SLAM. In 2019 16th International Conference on Machine Vision Applications (MVA). IEEE, 1--6.
[46]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 652--660.
[47]
Siegfried Seebacher, Bernd Datler, Jacqueline Erhart, Gerhard Greiner, Manfred Harrer, Peter Hrassnig, Arnold Präsent, Christian Schwarzl, and Martin Ullrich. 2019. Infrastructure data fusion for validation and future enhancements of autonomous vehicles' perception on Austrian motorways. In 2019 IEEE International Conference on Connected Vehicles and Expo (ICCVE). IEEE, 1--7.
[48]
Tixiao Shan, Brendan Englot, Drew Meyers, Wei Wang, Carlo Ratti, and Daniela Rus. 2020. Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 5135--5142.
[49]
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. 2020. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10529--10538.
[50]
Steve Shwartz. 2021. Are self-driving cars really safer than human drivers? https://thegradient.pub/are-self-driving-cars-really-safer-than-human-drivers/
[51]
Jack Stilgoe. 2020. Who Killed Elaine Herzberg? In Who's Driving Innovation? Springer, 1--6.
[52]
OpenPCDet Development Team. 2020. OpenPCDet: An Open-source Toolbox for 3D Object Detection from Point Clouds. https://github.com/open-mmlab/OpenPCDet.
[53]
Mabrouk Touahmia. 2018. Identification of risk factors influencing road traffic accidents. Engineering, Technology & Applied Science Research 8, 1 (2018), 2417--2421.
[54]
Manabu Tsukada, Takaharu Oi, Masahiro Kitazawa, and Hiroshi Esaki. 2020. Networked roadside perception units for autonomous driving. Sensors 20, 18 (2020), 5320.
[55]
Bill Vlasic and Neal E Boudette. 2016. Self-driving Tesla was involved in fatal crash, US says. New York Times 302016 (2016).
[56]
Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. 2019. Mots: Multi-object tracking and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7942--7951.
[57]
Song Wang, Jingqi Huang, and Xinyu Zhang. 2020. Demystifying millimeter-wave V2X: Towards robust and efficient directional connectivity under high mobility. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking.
[58]
Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani. 2020. 3d multi-object tracking: A baseline and new evaluation metrics. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 10359--10366.
[59]
Yan Yan, Yuxing Mao, and Bo Li. 2018. Second: Sparsely embedded convolutional detection. Sensors 18, 10 (2018), 3337.
[60]
Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. 2019. Std: Sparse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1951--1960.
[61]
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. 2021. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11784--11793.
[62]
Lijun Yu, Dawei Zhang, Xiangqun Chen, and Alexander Hauptmann. 2018. Traffic danger recognition with surveillance cameras without training data. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, 1--6.
[63]
Xumiao Zhang, Anlan Zhang, Jiachen Sun, Xiao Zhu, Y Ethan Guo, Feng Qian, and Z Morley Mao. 2021. EMP: edge-assisted multi-vehicle perception. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 545--558.
[64]
Zhihe Zhao, Zhehao Jiang, Neiwen Ling, Xian Shuai, and Guoliang Xing. 2018. ECRT: An edge computing system for real-time image-based object tracking. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. 394--395.
[65]
Feng Zhou and Fernando De la Torre. 2012. Factorized graph matching. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 127--134.
[66]
Yi Zhou and Ling Shao. 2018. Aware attentive multi-view inference for vehicle re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6489--6498.
[67]
Yin Zhou and Oncel Tuzel. 2018. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4490--4499.

Cited By

View all
  • (2025)Cross-Domain Generalization for LiDAR-Based 3D Object Detection in Infrastructure and Vehicle EnvironmentsSensors10.3390/s2503076725:3(767)Online publication date: 27-Jan-2025
  • (2025)Edge-Assisted Collaborative Perception Against Jamming and Interference in Vehicular NetworksIEEE Transactions on Wireless Communications10.1109/TWC.2024.351060124:1(860-874)Online publication date: Jan-2025
  • (2025)t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous DrivingIEEE Transactions on Mobile Computing10.1109/TMC.2024.346243724:1(135-149)Online publication date: Jan-2025
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MobiCom '22: Proceedings of the 28th Annual International Conference on Mobile Computing And Networking
October 2022
932 pages
ISBN:9781450391818
DOI:10.1145/3495243
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. infrastructure-assisted autonomous driving
  2. perception fusion
  3. vehicle mobility
  4. vehicle-infrastructure information fusion

Qualifiers

  • Research-article

Funding Sources

  • Innovation and Technology Commission - The Government of the Hong Kong Special Administrative Region of the People's republic of China
  • Centre for Perceptual and Interactive Intelligence

Conference

ACM MobiCom '22
Sponsor:

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)547
  • Downloads (Last 6 weeks)43
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Cross-Domain Generalization for LiDAR-Based 3D Object Detection in Infrastructure and Vehicle EnvironmentsSensors10.3390/s2503076725:3(767)Online publication date: 27-Jan-2025
  • (2025)Edge-Assisted Collaborative Perception Against Jamming and Interference in Vehicular NetworksIEEE Transactions on Wireless Communications10.1109/TWC.2024.351060124:1(860-874)Online publication date: Jan-2025
  • (2025)t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous DrivingIEEE Transactions on Mobile Computing10.1109/TMC.2024.346243724:1(135-149)Online publication date: Jan-2025
  • (2025)CPCS: a perception sharing scheme of vehicle-road cooperation based on cybertwinThe Journal of Supercomputing10.1007/s11227-025-06939-y81:2Online publication date: 24-Jan-2025
  • (2024)On data fabrication in collaborative vehicular perceptionProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699253(6309-6326)Online publication date: 14-Aug-2024
  • (2024)VILAMProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691925(1831-1845)Online publication date: 16-Apr-2024
  • (2024)Enhanced Perception for Autonomous Vehicles at Obstructed Intersections: An Implementation of Vehicle to Infrastructure (V2I) CollaborationSensors10.3390/s2403093624:3(936)Online publication date: 31-Jan-2024
  • (2024)Lightweight sensing-computing-decision collaboration enhancement for multi-mobile terminalsSCIENTIA SINICA Informationis10.1360/SSI-2024-008954:9(2136)Online publication date: 9-Sep-2024
  • (2024)Artificial Intelligence of Things: A SurveyACM Transactions on Sensor Networks10.1145/369063921:1(1-75)Online publication date: 30-Aug-2024
  • (2024)Stealthy Data Fabrication in Collaborative Vehicular PerceptionProceedings of the Sixth Workshop on CPS&IoT Security and Privacy10.1145/3690134.3694822(142-149)Online publication date: 19-Nov-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media