skip to main content
10.1145/3552469.3555713acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

No-reference Point Clouds Quality Assessment using Transformer and Visual Saliency

Published: 10 October 2022 Publication History

Abstract

Quality estimation of 3D objects/scenes represented by cloud point is a crucial and challenging task in computer vision. In real-world applications, reference data is not always available, which motivates the development of new point cloud quality assessment (PCQA) metrics that do not require the original 3D point cloud (3DPC). This family of methods is called no-reference or blind PCQA. In this context, we propose a deep-learning-based approach that benefits from the advantage of the self-attention mechanism in transformers to accurately predict the perceptual quality score for each degraded 3DPC. Additionally, we introduce the use of saliency maps to reflect the human visual system behavior that is attracted to some specific regions compared to others during the evaluation. To this end, we first render 2D projections (i.e. views) of a 3DPC from different viewpoints. Then, we weight the obtained projected images with their corresponding saliency maps. After that, we discard the majority of the background information by extracting sub-salient images. The latter is introduced as a sequential input of the vision transformer in order to extract the global contextual information and to predict the quality scores of the sub-images. Finally, we average the scores of all the salient sub-images to obtain the perceptual 3DPC quality score. We evaluate the performance of our model on the ICIP2020 and SJTU point cloud quality assessment benchmarks. Experimental results show that our model achieves promising performance compared to the state-of-the-art point cloud quality assessment metrics.

Supplementary Material

MP4 File (QoEVMA22-qoevma04.mp4)
Presentation Video

References

[1]
Evangelos Alexiou and Touradj Ebrahimi. 2018a. Point Cloud Quality Assessment Metric Based on Angular Similarity. In 2018 IEEE International Conference on Multimedia and Expo (ICME). 1--6. https://doi.org/10.1109/ICME.2018.8486512
[2]
Evangelos Alexiou and Touradj Ebrahimi. 2018b. Point Cloud Quality Assessment Metric Based on Angular Similarity. In 2018 IEEE International Conference on Multimedia and Expo (ICME). 1--6. https://doi.org/10.1109/ICME.2018.8486512
[3]
Evangelos Alexiou and Touradj Ebrahimi. 2020. Towards a Point Cloud Structural Similarity Metric. In 2020 IEEE International Conference on Multimedia Expo Workshops (ICMEW). 1--6. https://doi.org/10.1109/ICMEW46912.2020.9106005
[4]
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. 2019. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision. 3286--3295.
[5]
Chao Cao, Marius Preda, and Titus Zaharia. 2019. 3D point cloud compression. In Web3D 2019: 24th International Conference on 3D Web Technology. ACM Press, Los Angeles, United States, 1--9. https://doi.org/10.1145/3329714.3338130
[6]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[7]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020a. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[8]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020b. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[9]
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. 2022. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence (2022).
[10]
Alireza Javaheri, Catarina Brites, Fernando Pereira, and Jo ao Ascenso. 2020a. A generalized Hausdorff distance based quality metric for point cloud geometry. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 1--6.
[11]
Alireza Javaheri, Catarina Brites, Fernando Pereira, and João Ascenso. 2020b. Mahalanobis Based Point to Distribution Metric for Point Cloud Geometry Quality Evaluation. IEEE Signal Processing Letters, Vol. 27 (2020), 1350--1354. https://doi.org/10.1109/LSP.2020.3010128
[12]
Alireza Javaheri, Catarina Brites, Fernando Pereira, and Jo ao Ascenso. 2021. A Point-to-Distribution Joint Geometry and Color Metric for Point Cloud Quality Assessment. arXiv preprint arXiv:2108.00054 (2021).
[13]
Le Kang, Peng Ye, Yi Li, and David Doermann. 2014. Convolutional neural networks for no-reference image quality assessment. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1733--1740.
[14]
Alexander Kroner, Mario Senden, Kurt Driessens, and Rainer Goebel. 2020. Contextual encoder--decoder network for visual saliency prediction. Neural Networks, Vol. 129 (2020), 261--270.
[15]
RN Mekuria, Zhu Li, C Tulvan, and P Chou. 2016. Evaluation criteria for PCC (point cloud compression). (2016).
[16]
Gabriel Meynet, Yana Nehmé, Julie Digne, and Guillaume Lavoué. 2020. PCQM: A Full-Reference Quality Metric for Colored 3D Point Clouds. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). 1--6. https://doi.org/10.1109/QoMEX48832.2020.9123147
[17]
Stuart Perry, Huy Phi Cong, Luís A. da Silva Cruz, João Prazeres, Manuela Pereira, Antonio Pinheiro, Emil Dumic, Evangelos Alexiou, and Touradj Ebrahimi. 2020. Quality Evaluation Of Static Point Clouds Encoded Using MPEG Codecs. In 2020 IEEE International Conference on Image Processing (ICIP). 3428--3432. https://doi.org/10.1109/ICIP40778.2020.9191308
[18]
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. 2021. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 (2021).
[19]
Dong Tian, Hideaki Ochimizu, Chen Feng, Robert Cohen, and Anthony Vetro. 2017a. Geometric distortion metrics for point cloud compression. In 2017 IEEE International Conference on Image Processing (ICIP). 3460--3464. https://doi.org/10.1109/ICIP.2017.8296925
[20]
Dong Tian, Hideaki Ochimizu, Chen Feng, Robert Cohen, and Anthony Vetro. 2017b. Geometric distortion metrics for point cloud compression. In 2017 IEEE International Conference on Image Processing (ICIP). 3460--3464. https://doi.org/10.1109/ICIP.2017.8296925
[21]
Marouane Tliba, Mohamed A Kerkouri, Bashir Ghariba, Aladine Chetouani, Arzu cC öltekin, Mohamed Sami Shehata, and Alessandro Bruno. 2022. SATSal: A Multi-Level Self-Attention Based Architecture for Visual Saliency Prediction. IEEE Access, Vol. 10 (2022), 20701--20713.
[22]
Yubing Tong, Hubert Konik, Faouzi Cheikh, and Alain Tremeau. 2010. Full reference image quality assessment based on saliency map analysis. Journal of Imaging Science and Technology, Vol. 54, 3 (2010), 30503--1.
[23]
Irene Viola and Pablo Cesar. 2020. A reduced reference metric for visual quality evaluation of point cloud contents. IEEE Signal Processing Letters, Vol. 27 (2020), 1660--1664.
[24]
Irene Viola, Shishir Subramanyam, and Pablo Cesar. 2020. A color-based objective quality metric for point cloud contents. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 1--6.
[25]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, Vol. 13, 4 (2004), 600--612.
[26]
Qi Yang, Hao Chen, Zhan Ma, Yiling Xu, Rongjun Tang, and Jun Sun. 2020. Predicting the Perceptual Quality of Point Cloud: A 3D-to-2D Projection-Based Exploration. IEEE Transactions on Multimedia (2020), 1--1. https://doi.org/10.1109/TMM.2020.3033117
[27]
Junyong You and Jari Korhonen. 2021. Transformer for image quality assessment. In 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 1389--1393.

Cited By

View all
  • (2024)Visual-Saliency Guided Multi-modal Learning for No Reference Point Cloud Quality AssessmentProceedings of the 3rd Workshop on Quality of Experience in Visual Multimedia Applications10.1145/3689093.3689183(39-47)Online publication date: 28-Oct-2024
  • (2024)Saliency and Depth-Aware Full Reference 360-Degree Image Quality AssessmentInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S021800142351022938:01Online publication date: 9-Feb-2024
  • (2024)Point Cloud Quality Assessment Using Multi-Level FeaturesIEEE Access10.1109/ACCESS.2024.338353612(47755-47767)Online publication date: 2024
  • Show More Cited By

Index Terms

  1. No-reference Point Clouds Quality Assessment using Transformer and Visual Saliency

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    QoEVMA '22: Proceedings of the 2nd Workshop on Quality of Experience in Visual Multimedia Applications
    October 2022
    75 pages
    ISBN:9781450394994
    DOI:10.1145/3552469
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 October 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. 3d point clouds
    2. attention
    3. objective quality assessment
    4. transformer
    5. visual saliency

    Qualifiers

    • Research-article

    Conference

    MM '22
    Sponsor:

    Acceptance Rates

    QoEVMA '22 Paper Acceptance Rate 8 of 14 submissions, 57%;
    Overall Acceptance Rate 14 of 20 submissions, 70%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)51
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 08 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Visual-Saliency Guided Multi-modal Learning for No Reference Point Cloud Quality AssessmentProceedings of the 3rd Workshop on Quality of Experience in Visual Multimedia Applications10.1145/3689093.3689183(39-47)Online publication date: 28-Oct-2024
    • (2024)Saliency and Depth-Aware Full Reference 360-Degree Image Quality AssessmentInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S021800142351022938:01Online publication date: 9-Feb-2024
    • (2024)Point Cloud Quality Assessment Using Multi-Level FeaturesIEEE Access10.1109/ACCESS.2024.338353612(47755-47767)Online publication date: 2024
    • (2023)Bitstream-Based Perceptual Quality Assessment of Compressed 3D Point CloudsIEEE Transactions on Image Processing10.1109/TIP.2023.325325232(1815-1828)Online publication date: 1-Jan-2023
    • (2023)No-Reference Point Cloud Quality Assessment via Contextual Point-Wise Deep Learning NetworkCognitive Systems and Information Processing10.1007/978-981-99-8021-5_17(218-233)Online publication date: 5-Nov-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media