skip to main content
10.1145/3581783.3612080acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

CenterLPS: Segment Instances by Centers for LiDAR Panoptic Segmentation

Published: 27 October 2023 Publication History

Abstract

This paper focuses on LiDAR Panoptic Segmentation (LPS), which has attracted more attention recently due to its broad application prospect for autonomous driving and robotics. The mainstream LPS approaches either adopt a top-down strategy relying on 3D object detectors to discover instances or utilize time-consuming heuristic clustering algorithms to group instances in a bottom-up manner. Inspired by the center representation and kernel-based segmentation, we propose a new detection-free and clustering-free framework called CenterLPS, with the center-based instance encoding and decoding paradigm. Specifically, we propose a sparse center proposal network to generate the sparse 3D instance centers, as well as center feature embedding, which can well encode characteristics of instances. Then a center-aware transformer is applied to collect the context between different center feature embedding and around centers. Moreover, we generate the kernel weights based on the enhanced center feature embedding and initialize dynamic convolutions to decode the final instance masks. Finally, a mask fusion module is devised to unify the semantic and instance predictions and improve the panoptic quality. Extensive experiments on SemanticKITTI and nuScenes demonstrate the effectiveness of our proposed center-based framework CenterLPS.

References

[1]
Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. 2019. SemanticKITTI: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE Int'l Conf. on Computer Vision. 9297--9307.
[2]
Jens Behley, Andres Milioto, and Cyrill Stachniss. 2021. A Benchmark for LiDAR-based Panoptic Segmentation based on KITTI. In 2021 IEEE Int'l Conf. on Robotics and Automation (ICRA). IEEE, 13596--13603.
[3]
Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. 2018. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition. 4413--4421.
[4]
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conf., Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16. Springer, 213--229.
[5]
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. 2022. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 1290--1299.
[6]
Bowen Cheng, Alex Schwing, and Alexander Kirillov. 2021. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, Vol. 34 (2021), 17864--17875.
[7]
Fabian Duerr, Hendrik Weigel, and Jürgen Beyerer. 2022. RangeBird: Multi View Panoptic Segmentation of 3D Point Clouds with Neighborhood Attention. In 2022 Int'l Conf. on Robotics and Automation (ICRA). IEEE, 11131--11137.
[8]
Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. 2022. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters, Vol. 7, 2 (2022), 3795--3802.
[9]
Stefano Gasperini, Mohammad-Ali Nikouei Mahani, Alvaro Marcos-Ramiro, Nassir Navab, and Federico Tombari. 2021. Panoster: End-to-end panoptic segmentation of lidar point clouds. IEEE Robotics and Automation Letters, Vol. 6, 2 (2021), 3216--3223.
[10]
Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conf. on computer vision and pattern recognition. IEEE, 3354--3361.
[11]
Yi Gu, Yuming Huang, Chengzhong Xu, and Hui Kong. 2022. MaskRange: A Mask-classification Model for Range-view based LiDAR Segmentation. arXiv preprint arXiv:2206.12073 (2022).
[12]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE Int'l Conf. on computer vision. 2961--2969.
[13]
Tong He, Chunhua Shen, and Anton Van Den Hengel. 2021. Dyco3d: Robust instance segmentation of 3d point clouds through dynamic convolution. In Proceedings of the IEEE/CVF Conf. on computer vision and pattern recognition. 354--363.
[14]
Fangzhou Hong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. 2021. Lidar-based panoptic segmentation via dynamic shifting network. In Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 13090--13099.
[15]
Juana Valeria Hurtado, Rohit Mohan, Wolfram Burgard, and Abhinav Valada. 2020. Mopt: Multi-object panoptic tracking. arXiv preprint arXiv:2004.08189 (2020).
[16]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[17]
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. 2019. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conf. on computer vision and pattern recognition. 12697--12705.
[18]
Enxu Li, Ryan Razani, Yixuan Xu, and Bingbing Liu. 2021a. CPSeg: Cluster-free Panoptic Segmentation of 3D LiDAR Point Clouds. arXiv preprint arXiv:2111.01723 (2021).
[19]
Enxu Li, Ryan Razani, Yixuan Xu, and Bingbing Liu. 2022b. Smac-seg: Lidar panoptic segmentation via sparse multi-directional attention clustering. In 2022 Int'l Conf. on Robotics and Automation (ICRA). IEEE, 9207--9213.
[20]
Jinke Li, Xiao He, Yang Wen, Yuan Gao, Xiaoqiang Cheng, and Dan Zhang. 2022a. Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap. In Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 11809--11818.
[21]
Xiaoyan Li, Gang Zhang, Boyue Wang, Yongli Hu, and Baocai Yin. 2023. Center Focusing Network for Real-Time LiDAR Panoptic Segmentation. In Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 13425--13434.
[22]
Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. 2021b. Fully convolutional networks for panoptic segmentation. In Proceedings of the IEEE/CVF Conf. on computer vision and pattern recognition. 214--223.
[23]
Minzhe Liu, Qiang Zhou, Hengshuang Zhao, Jianing Li, Yuan Du, Kurt Keutzer, Li Du, and Shanghang Zhang. 2022. Prototype-Voxel Contrastive Learning for LiDAR Point Cloud Panoptic Segmentation. In 2022 Int'l Conf. on Robotics and Automation (ICRA). IEEE, 9243--9250.
[24]
Rodrigo Marcuzzi, Lucas Nunes, Louis Wiesmann, Jens Behley, and Cyrill Stachniss. 2023. Mask-Based Panoptic LiDAR Segmentation for Autonomous Driving. IEEE Robotics and Automation Letters (2023).
[25]
Jianbiao Mei, Mengmeng Wang, Yeneng Lin, Yi Yuan, and Yong Liu. 2021. Transvos: Video object segmentation with transformers. arXiv preprint arXiv:2106.00588 (2021).
[26]
Jianbiao Mei, Yu Yang, Mengmeng Wang, Xiaojun Hou, Laijian Li, and Yong Liu. 2023. PANet: LiDAR Panoptic Segmentation with Sparse Instance Proposal and Aggregation. arXiv preprint arXiv:2306.15348 (2023).
[27]
Andres Milioto, Jens Behley, Chris McCool, and Cyrill Stachniss. 2020. Lidar panoptic segmentation for autonomous driving. In 2020 IEEE/RSJ Int'l Conf. on Intelligent Robots and Systems (IROS). IEEE, 8505--8512.
[28]
Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. 2019. Rangenet: Fast and accurate lidar semantic segmentation. In 2019 IEEE/RSJ Int'l Conf. on intelligent robots and systems (IROS). IEEE, 4213--4220.
[29]
Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, and Peter Kontschieder. 2019. Seamless scene segmentation. In Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 8277--8286.
[30]
Ryan Razani, Ran Cheng, Enxu Li, Ehsan Taghavi, Yuan Ren, and Liu Bingbing. 2021. GP-S3Net: Graph-based panoptic sparse semantic segmentation network. In Proceedings of the IEEE/CVF Int'l Conf. on Computer Vision. 16076--16085.
[31]
Kshitij Sirohi, Rohit Mohan, Daniel Büscher, Wolfram Burgard, and Abhinav Valada. 2021. Efficientlps: Efficient lidar panoptic segmentation. IEEE Transactions on Robotics, Vol. 38, 3 (2021), 1894--1914.
[32]
Shihao Su, Jianyun Xu, Huanyu Wang, Zhenwei Miao, Xin Zhan, Dayang Hao, and Xi Li. 2023. PUPS: Point Cloud Unified Panoptic Segmentation. arXiv preprint arXiv:2302.06185 (2023).
[33]
Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. 2020. Searching efficient 3d architectures with sparse point-voxel convolution. In Computer Vision-ECCV 2020: 16th European Conf., Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII. Springer, 685--702.
[34]
Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, Francc ois Goulette, and Leonidas J Guibas. 2019. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF Int'l Conf. on computer vision. 6411--6420.
[35]
Zhi Tian, Chunhua Shen, and Hao Chen. 2020. Conditional convolutions for instance segmentation. In Computer Vision-ECCV 2020: 16th European Conf., Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16. Springer, 282--298.
[36]
Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, and Weicai Zhong. 2022. 3D Instances as 1D Kernels. In Computer Vision-ECCV 2022: 17th European Conf., Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX. Springer, 235--252.
[37]
Zeqi Xiao, Wenwei Zhang, Tai Wang, Chen Change Loy, Dahua Lin, and Jiangmiao Pang. 2023. Position-Guided Point Cloud Panoptic Segmentation Transformer. arXiv preprint arXiv:2303.13509 (2023).
[38]
Shuangjie Xu, Rui Wan, Maosheng Ye, Xiaoyi Zou, and Tongyi Cao. 2022. Sparse cross-scale attention network for efficient lidar panoptic segmentation. In Proceedings of the AAAI Conf. on Artificial Intelligence, Vol. 36. 2920--2928.
[39]
Yixuan Xu, Hamidreza Fazlali, Yuan Ren, and Bingbing Liu. 2023. AOP-Net: All-in-One Perception Network for Joint LiDAR-based 3D Object Detection and Panoptic Segmentation. arXiv preprint arXiv:2302.00885 (2023).
[40]
Dongqiangzi Ye, Zixiang Zhou, Weijia Chen, Yufei Xie, Yu Wang, Panqu Wang, and Hassan Foroosh. 2022b. Lidarmultinet: Towards a unified multi-task network for lidar perception. arXiv preprint arXiv:2209.09385 (2022).
[41]
Maosheng Ye, Rui Wan, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. 2022a. Efficient Point Cloud Segmentation with Geometry-Aware Sparse Networks. In Computer Vision-ECCV 2022: 17th European Conf., Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX. Springer, 196--212.
[42]
Maosheng Ye, Shuangjie Xu, Tongyi Cao, and Qifeng Chen. 2021. Drinet: A dual-representation iterative learning network for point cloud segmentation. In Proceedings of the IEEE/CVF Int'l Conf. on computer vision. 7447--7456.
[43]
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. 2021. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF Conf. on computer vision and pattern recognition. 11784--11793.
[44]
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. 2021. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, Vol. 34 (2021), 10326--10338.
[45]
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. 2019. Objects as points. arXiv preprint arXiv:1904.07850 (2019).
[46]
Zixiang Zhou, Yang Zhang, and Hassan Foroosh. 2021. Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation. In Proceedings of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 13194--13203.
[47]
Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. 2022. Centerformer: Center-based transformer for 3d object detection. In Computer Vision-ECCV 2022: 17th European Conf., Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII. Springer, 496--513.

Cited By

View all
  • (2024)Camera-Based 3D Semantic Scene Completion With Sparse Guidance NetworkIEEE Transactions on Image Processing10.1109/TIP.2024.346198933(5468-5481)Online publication date: 1-Jan-2024
  • (2024)PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01379(14554-14564)Online publication date: 16-Jun-2024

Index Terms

  1. CenterLPS: Segment Instances by Centers for LiDAR Panoptic Segmentation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '23: Proceedings of the 31st ACM International Conference on Multimedia
    October 2023
    9913 pages
    ISBN:9798400701085
    DOI:10.1145/3581783
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 October 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. center-aware transformer
    2. lidar panoptic segmentation
    3. mask fusion
    4. sparse center proposal network

    Qualifiers

    • Research-article

    Funding Sources

    • This work was supported by a Grant from The National Natural Science Foundation of China

    Conference

    MM '23
    Sponsor:
    MM '23: The 31st ACM International Conference on Multimedia
    October 29 - November 3, 2023
    Ottawa ON, Canada

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)84
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Camera-Based 3D Semantic Scene Completion With Sparse Guidance NetworkIEEE Transactions on Image Processing10.1109/TIP.2024.346198933(5468-5481)Online publication date: 1-Jan-2024
    • (2024)PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01379(14554-14564)Online publication date: 16-Jun-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media