skip to main content
10.1145/3570361.3613271acmconferencesArticle/Chapter ViewAbstractPublication PagesmobicomConference Proceedingsconference-collections
research-article
Public Access

Robust Real-time Multi-vehicle Collaboration on Asynchronous Sensors

Published: 02 October 2023 Publication History

Abstract

Cooperative perception significantly enhances the perception performance of connected autonomous vehicles. Instead of purely relying on local sensors with limited range, it enables multiple vehicles and roadside infrastructures to share sensor data to perceive the environment collaboratively. Through our study, we realize that the performance of cooperative perception systems is limited in real-world deployment due to (1) out-of-sync sensor data during data fusion and (2) inaccurate localization of occluded areas. To address these challenges, we develop RAO, an innovative, effective, and lightweight cooperative perception system that merges asynchronous sensor data from different vehicles through our novel designs of motion-compensated occupancy flow prediction and on-demand data sharing, improving both the accuracy and coverage of the perception system. Our extensive evaluation, including real-world and emulation-based experiments, demonstrates that RAO outperforms state-of-the-art solutions by more than 34% in perception coverage and by up to 14% in perception accuracy, especially when asynchronous sensor data is present. RAO consistently performs well across a wide variety of map topologies and driving scenarios. RAO incurs negligible additional latency (8.5 ms) and low data transmission overhead (10.9 KB per frame), making cooperative perception feasible.

References

[1]
2022. 5G Automotive Association. https://5gaa.org/.
[2]
2022. Autoware: Open-source software for self-driving vehicles. https://github.com/Autoware-AI.
[3]
2022. Baidu Apollo. https://apollo.auto.
[4]
2022. OpenCDA. https://github.com/ucla-mobility/OpenCDA/.
[5]
2022. SUMO: Simulation of Urban Mobility. https://www.eclipse.org/sumo/.
[6]
2022. The KITTI benchmark suite. https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d.
[7]
2023. CARLA: Open-source simulator for autonomous driving research. https://carla.org/.
[8]
2023. Mcity - University of Michigan. https://mcity.umich.edu/.
[9]
2023. Speed limits in the United States. https://en.wikipedia.org/wiki/Speed_limits_in_the_United_States_by_jurisdiction.
[10]
2023. Velodyne LiDAR. https://velodynelidar.com/.
[11]
Eduardo Arnold, Mehrdad Dianati, Robert de Temple, and Saber Fallah. 2020. Cooperative perception for 3D object detection in driving scenarios using infrastructure sensors. IEEE Transactions on Intelligent Transportation Systems (2020).
[12]
Franz Aurenhammer, Rolf Klein, and Der-Tsai Lee. 2013. Voronoi diagrams and Delaunay triangulations. World Scientific Publishing Company.
[13]
David Avis and David Bremner. 1995. How good are convex hull algorithms?. In Proceedings of the eleventh annual symposium on Computational geometry. 20--28.
[14]
Hanlin Chen, Brian Liu, Xumiao Zhang, Feng Qian, Z Morley Mao, and Yiheng Feng. 2022. A Cooperative Perception Environment for Traffic Operations and Control. arXiv preprint arXiv:2208.02792 (2022).
[15]
Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. 2019. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing. 88--100.
[16]
Qi Chen, Sihai Tang, Qing Yang, and Song Fu. 2019. Cooper: Cooperative perception for connected autonomous vehicles based on 3d point clouds. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, 514--524.
[17]
Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. 2022. COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17252--17262.
[18]
Guanting Dong, Yueyi Zhang, Hanlin Li, Xiaoyan Sun, and Zhiwei Xiong. 2022. Exploiting Rigidity Constraints for LiDAR Scene Flow Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12776--12785.
[19]
Bertrand Douillard, James Underwood, Noah Kuntz, Vsevolod Vlaskine, Alastair Quadros, Peter Morton, and Alon Frenkel. 2011. On the segmentation of 3D LIDAR point clouds. In 2011 IEEE International Conference on Robotics and Automation. IEEE, 2798--2805.
[20]
Renaud Dubé, Abel Gawel, Hannes Sommer, Juan Nieto, Roland Siegwart, and Cesar Cadena. 2017. An online multi-robot SLAM system for 3D LiDARs. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 1004--1011.
[21]
Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (1981), 381--395.
[22]
Moinak Ghoshal, Z Jonny Kong, Qiang Xu, Zixiao Lu, Shivang Aggarwal, Imran Khan, Yuanjie Li, Y Charlie Hu, and Dimitrios Koutsonikolas. 2022. An in-depth study of uplink performance of 5G mmWave networks. In Proceedings of the ACM SIGCOMM Workshop on 5G and Beyond Network Measurements, Modeling, and Use Cases. 29--35.
[23]
Zan Gojcic, Or Litany, Andreas Wieser, Leonidas J Guibas, and Tolga Birdal. 2021. Weakly supervised learning of rigid 3D scene flow. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5692--5703.
[24]
In-Chul Ha. 2008. Kinematic parameter calibration method for industrial robot manipulator using the relative position. Journal of mechanical science and technology 22, 6 (2008), 1084--1090.
[25]
Ying He, Bin Liang, Jun Yang, Shunzhi Li, and Jin He. 2017. An iterative closest points algorithm for registration of 3D laser scanner point clouds with geometric features. Sensors 17, 8 (2017), 1862.
[26]
Yuze He, Li Ma, Zhehao Jiang, Yi Tang, and Guoliang Xing. 2021. VI-eye: semantic-based 3D point cloud registration for infrastructure-assisted autonomous driving. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 573--586.
[27]
Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. 2022. Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps. (2022).
[28]
John B Kenney. 2011. Dedicated short-range communications (DSRC) standards in the United States. Proc. IEEE 99, 7 (2011), 1162--1182.
[29]
ByeoungDo Kim, Chang Mook Kang, Jaekyum Kim, Seung Hi Lee, Chung Choo Chung, and Jun Won Choi. 2017. Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 399--404.
[30]
Seong-Woo Kim, Wei Liu, Marcelo H Ang, Emilio Frazzoli, and Daniela Rus. 2015. The impact of cooperative perception on decision making and planning of autonomous vehicles. IEEE Intelligent Transportation Systems Magazine 7, 3 (2015), 39--50.
[31]
Swarun Kumar, Shyamnath Gollakota, and Dina Katabi. 2012. A cloud-assisted design for autonomous driving. In Proceedings of the first edition of the MCC workshop on Mobile cloud computing. 41--46.
[32]
Swarun Kumar, Lixin Shi, Nabeel Ahmed, Stephanie Gil, Dina Katabi, and Daniela Rus. 2012. Carspeak: a content-centric network for autonomous driving. ACM SIGCOMM Computer Communication Review 42, 4 (2012), 259--270.
[33]
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. 2019. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12697--12705.
[34]
Seokju Lee, Francois Rameau, Fei Pan, and In So Kweon. 2021. Attentive and contrastive learning for joint depth and motion field estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 4862--4871.
[35]
Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. 2022. Latency-aware collaborative perception. In European Conference on Computer Vision. Springer, 316--332.
[36]
Hao Li, Manabu Tsukada, Fawzi Nashashibi, and Michel Parent. 2014. Multivehicle cooperative local mapping: A methodology based on occupancy grid map merging. IEEE Transactions on Intelligent Transportation Systems 15, 5 (2014), 2089--2100.
[37]
Ruibo Li, Guosheng Lin, and Lihua Xie. 2021. Self-point-flow: Self-supervised scene flow estimation from point clouds with optimal transport and random walk. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15577--15586.
[38]
Yiming Li, Dekun Ma, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng. 2022. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters 7, 4 (2022), 10914--10921.
[39]
Shih-Chieh Lin, Yunqi Zhang, Chang-Hong Hsu, Matt Skach, Md E Haque, Lingjia Tang, and Jason Mars. 2018. The architectural implications of autonomous driving: Constraints and acceleration. In Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems. 751--766.
[40]
Hansi Liu, Pengfei Ren, Shubham Jain, Mohannad Murad, Marco Gruteser, and Fan Bai. 2019. FusionEye: Perception Sharing for Connected Vehicles and its Bandwidth-Accuracy Trade-offs. In 2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 1--9.
[41]
Qiang Liu, Tao Han, Jiang Linda Xie, and BaekGyu Kim. 2021. LiveMap: Real-time dynamic map in automotive edge computing. In IEEE IN-FOCOM 2021-IEEE Conference on Computer Communications. IEEE, 1--10.
[42]
Guiyang Luo, Hui Zhang, Quan Yuan, and Jinglin Li. 2022. Complementarity-Enhanced and Redundancy-Minimized Collaboration Network for Multi-agent Perception. In Proceedings of the 30th ACM International Conference on Multimedia. 3578--3586.
[43]
Lili Miao, John Jethro Virtusio, and Kai-Lung Hua. 2021. PC5-based cellular-V2X evolution and deployment. Sensors 21, 3 (2021), 843.
[44]
David L Mills. 1991. Internet time synchronization: the network time protocol. IEEE Transactions on communications 39, 10 (1991), 1482--1493.
[45]
Arvind Narayanan, Eman Ramadan, Rishabh Mehta, Xinyue Hu, Qingxu Liu, Rostand AK Fezeu, Udhaya Kumar Dayalan, Saurabh Verma, Peiqi Ji, Tao Li, et al. 2020. Lumos5G: Mapping and predicting commercial mmWave 5G throughput. In Proceedings of the ACM Internet Measurement Conference. 176--193.
[46]
Arvind Narayanan, Xumiao Zhang, Ruiyang Zhu, Ahmad Hassan, Shuowei Jin, Xiao Zhu, Xiaoxuan Zhang, Denis Rybkin, Zhengxuan Yang, Zhuoqing Morley Mao, et al. 2021. A variegated look at 5G in the wild: performance, power, and QoE implications. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference. 610--625.
[47]
Mohammad Naserian, Curtis L Hay, Sayyed Rouhollah Jafari Tafti, and Allan K Lewis. 2021. Process and system for sensor sharing for an autonomous lane change. US Patent App. 16/695,897.
[48]
Anshul Paigwar, Özgür Erkent, David Sierra-Gonzalez, and Christian Laugier. 2020. Gndnet: Fast ground plane estimation and point cloud segmentation for autonomous vehicles. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2150--2156.
[49]
D Qiao and F Zulkernine. 2022. Adaptive Feature Fusion for Cooperative Perception using LiDAR Point Clouds. arXiv preprint arXiv:2208.00116 (2022).
[50]
Hang Qiu, Fawad Ahmad, Fan Bai, Marco Gruteser, and Ramesh Govindan. 2018. AVR: Augmented vehicular reality. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 81--95.
[51]
Hang Qiu, Pohan Huang, Namo Asavisanu, Xiaochen Liu, Konstantinos Psounis, and Ramesh Govindan. 2021. Autocast: Scalable infrastructure-less cooperative perception for distributed collaborative driving. arXiv preprint arXiv:2112.14947 (2021).
[52]
Alexander Rabitsch, Karl-Johan Grinnemo, Anna Brunstrom, Henrik Abrahamsson, Fehmi Ben Abdesslem, Stefan Alfredsson, and Bengt Ahlgren. 2020. Utilizing multi-connectivity to reduce latency and enhance availability for vehicle to infrastructure communication. IEEE Transactions on Mobile Computing (2020).
[53]
Frédéric Rivard, Jonathan Bisson, François Michaud, and Dominic Létourneau. 2008. Ultrasonic relative positioning for multi-robot systems. In 2008 IEEE International Conference on Robotics and Automation. IEEE, 323--328.
[54]
Shuyao Shi, Jiahe Cui, Zhehao Jiang, Zhenyu Yan, Guoliang Xing, Jianwei Niu, and Zhenchao Ouyang. 2022. VIPS: real-time perception fusion for infrastructure-assisted autonomous driving. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking. 133--146.
[55]
Zhiying Song, Fuxi Wen, Hailiang Zhang, and Jun Li. 2022. An Efficient and Robust Object-Level Cooperative Perception Framework for Connected and Automated Driving. arXiv preprint arXiv:2210.06289 (2022).
[56]
Jan Erik Stellet, Fabian Straub, Jan Schumacher, Wolfgang Branz, and J Marius Zöllner. 2015. Estimating the process noise variance for vehicle motion models. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems. IEEE, 1512--1519.
[57]
Vladimir Vukadinovic, Krzysztof Bakowski, Patrick Marsch, Ian Dexter Garcia, Hua Xu, Michal Sybis, Pawel Sroka, Krzysztof Wesolowski, David Lister, and Ilaria Thibault. 2018. 3GPP C-V2X and IEEE 802.11 p for Vehicle-to-Vehicle communications in highway platooning scenarios. Ad Hoc Networks 74 (2018), 17--29.
[58]
Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. 2020. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In European Conference on Computer Vision. Springer, 605--621.
[59]
Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani. 2020. 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics. IROS (2020).
[60]
Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer. 2018. Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1887--1893.
[61]
Runsheng Xu, Jinlong Li, Xiaoyu Dong, Hongkai Yu, and Jiaqi Ma. 2022. Bridging the Domain Gap for Multi-Agent Perception. arXiv preprint arXiv:2210.08451 (2022).
[62]
Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. 2022. V2X-ViT: Vehicle-to-everything cooperative perception with vision transformer. In European Conference on Computer Vision. Springer, 107--124.
[63]
Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. 2022. OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication. In 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2583--2589.
[64]
Zhigang Xu, Xiaochi Li, Xiangmo Zhao, Michael H Zhang, and Zhongren Wang. 2017. DSRC versus 4G-LTE for connected vehicle applications: A study on field experiments of vehicular communication performance. Journal of advanced transportation 2017 (2017).
[65]
Jingang Yi, Hongpeng Wang, Junjie Zhang, Dezhen Song, Suhada Jayasuriya, and Jingtai Liu. 2009. Kinematic modeling and analysis of skid-steered mobile robots with applications to low-cost inertial-measurement-unit-based motion estimation. IEEE transactions on robotics 25, 5 (2009), 1087--1097.
[66]
Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al. 2022. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21361--21370.
[67]
Yunshuang Yuan, Hao Cheng, and Monika Sester. 2022. Keypoints-Based Deep Feature Fusion for Cooperative Vehicle Detection of Autonomous Driving. IEEE Robotics and Automation Letters 7, 2 (2022), 3054--3061.
[68]
Dimitris Zermas, Izzat Izzat, and Nikolaos Papanikolopoulos. 2017. Fast segmentation of 3d point clouds: A paradigm on lidar data for autonomous vehicle applications. In 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 5067--5073.
[69]
Xumiao Zhang, Anlan Zhang, Jiachen Sun, Xiao Zhu, Y Ethan Guo, Feng Qian, and Z Morley Mao. 2021. EMP: edge-assisted multi-vehicle perception. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 545--558.

Cited By

View all
  • (2024)On data fabrication in collaborative vehicular perceptionProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699253(6309-6326)Online publication date: 14-Aug-2024
  • (2024)Stealthy Data Fabrication in Collaborative Vehicular PerceptionProceedings of the Sixth Workshop on CPS&IoT Security and Privacy10.1145/3690134.3694822(142-149)Online publication date: 19-Nov-2024
  • (2024)Demo: Enabling Efficient Perception Sharing via Infrastructure-to-Road BeamformingProceedings of the ACM SIGCOMM 2024 Conference: Posters and Demos10.1145/3672202.3673713(89-91)Online publication date: 4-Aug-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ACM MobiCom '23: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking
October 2023
1605 pages
ISBN:9781450399906
DOI:10.1145/3570361
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 October 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. cooperative perception
  2. autonomous cars
  3. vehicular networks
  4. LiDAR

Qualifiers

  • Research-article

Funding Sources

Conference

ACM MobiCom '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)925
  • Downloads (Last 6 weeks)84
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)On data fabrication in collaborative vehicular perceptionProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699253(6309-6326)Online publication date: 14-Aug-2024
  • (2024)Stealthy Data Fabrication in Collaborative Vehicular PerceptionProceedings of the Sixth Workshop on CPS&IoT Security and Privacy10.1145/3690134.3694822(142-149)Online publication date: 19-Nov-2024
  • (2024)Demo: Enabling Efficient Perception Sharing via Infrastructure-to-Road BeamformingProceedings of the ACM SIGCOMM 2024 Conference: Posters and Demos10.1145/3672202.3673713(89-91)Online publication date: 4-Aug-2024
  • (2024)Boosting Collaborative Vehicular Perception on the Edge with Vehicle-to-Vehicle CommunicationProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699328(141-154)Online publication date: 4-Nov-2024
  • (2024)VRF: Vehicle Road-side Point Cloud FusionProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661874(547-560)Online publication date: 3-Jun-2024
  • (2024)Foes or Friends: Embracing Ground Effect for Edge Detection on Lightweight DronesProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3690699(1377-1392)Online publication date: 4-Dec-2024
  • (2024)SwissCheese: Fine-Grained Channel-Spatial Feature Filtering for Communication-Efficient Cooperative PerceptionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2024.348035925:12(21047-21059)Online publication date: Dec-2024
  • (2024)A Review of Cloud-Edge SLAM: Toward Asynchronous Collaboration and Implicit Representation TransmissionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2024.343816525:11(15437-15453)Online publication date: Nov-2024
  • (2024)Edge Intelligence for Internet of Vehicles: A SurveyIEEE Transactions on Consumer Electronics10.1109/TCE.2024.337850970:2(4858-4877)Online publication date: May-2024
  • (2024)PhD Forum Abstract: Cooperative Perception System with Roadside Assistance2024 23rd ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN)10.1109/IPSN61024.2024.00064(325-326)Online publication date: 13-May-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media