Skip to main content
SpringerLink
Log in

OVPT: Optimal Viewset Pooling Transformer for 3D Object Recognition

  • Conference paper
  • First Online:
Computer Vision – ACCV 2022 (ACCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13841))

Included in the following conference series:

Abstract

The current methods for multi-view-based 3D object recognition have the problem of losing the correlation between views and rendering 3D objects with multi-view redundancy. This makes it difficult to improve recognition performance and unnecessarily increases the computational cost and running time of the network. Especially in the case of limited computing resources, the recognition performance is further affected. Our study developed an optimal viewset pooling transformer (OVPT) method for efficient and accurate 3D object recognition. The OVPT method constructs the optimal viewset based on information entropy to reduce the redundancy of the multi-view scheme. We used convolutional neural network (CNN) to extract the multi-view low-level local features of the optimal viewset. Embedding class token into the headers of multi-view low-level local features and splicing with position encoding generates local-view token sequences. This sequence was trained parallel with a pooling transformer to generate a local view information token sequence. At the same time, the global class token captured the global feature information of the local view token sequence. The two were aggregated next into a single compact 3D global feature descriptor. On two public benchmarks, ModelNet10 and ModelNet40, for each 3D object we only need a smaller number of optimal viewsets, achieving an overall recognition accuracy (OA) of 99.33% and 97.48%, respectively. Compared with other deep learning methods, our method still achieves state-of-the-art performance with limited computational resources. Our source code is available at https://github.com/shepherds001/OVPT.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Liang, Z., et al.: Stereo matching using multi-level cost volume and multi-scale feature constancy. IEEE Trans. Pattern Anal. Mach. Intell. 43(1), 300–315 (2021). https://doi.org/10.1109/TPAMI.2019.2928550

    Article  Google Scholar 

  2. Li, Y., et al.: Deep learning for lidar point clouds in autonomous driving: a review. IEEE Trans. Neural Netw. Learn. Syst. 32(8), 3412–3432 (2021). https://doi.org/10.1109/TNNLS.2020.3015992

    Article  Google Scholar 

  3. Kästner, L., Frasineanu, V.C., Lambrecht, J.: A 3D-deep-learning-based augmented reality calibration method for robotic environments using depth sensor data. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 1135–1141 (2020). https://doi.org/10.1109/ICRA40945.2020.9197155

  4. Lee, D., et al.: Large-scale localization datasets in crowded indoor spaces. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3226–3235 (2021). https://doi.org/10.1109/CVPR46437.2021.00324

  5. Maturana, D., Scherer, S.: Voxnet: a 3D convolutional neural network for real-time object recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922–928 (2015). https://doi.org/10.1109/IROS.2015.7353481

  6. Wu, Z., et al.: 3D shapenets: a deep representation for volumetric shapes. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1912–1920 (2015). https://doi.org/10.1109/CVPR.2015.7298801

  7. Riegler, G., Ulusoy, A.O., Geiger, A.: Octnet: learning deep 3D representations at high resolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6620–6629 (2017). https://doi.org/10.1109/CVPR.2017.701

  8. Wang, P.S., Liu, Y., Guo, Y.X., Sun, C.Y., Tong, X.: O-cnn: octree-based convolutional neural networks for 3d shape analysis. ACM Trans. Graph. 36(4) (2017). https://doi.org/10.1145/3072959.3073608

  9. Klokov, R., Lempitsky, V.: Escape from cells: Deep kd-networks for the recognition of 3D point cloud models. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 863–872 (2017). https://doi.org/10.1109/ICCV.2017.99

  10. Zeng, W., Gevers, T.: 3DContextNet: K-d tree guided hierarchical learning of point clouds using local and global contextual cues. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 314–330. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_24

    Chapter  Google Scholar 

  11. Le, T., Duan, Y.: Pointgrid: a deep network for 3D shape understanding. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9204–9214 (2018). https://doi.org/10.1109/CVPR.2018.00959

  12. Meng, H.Y., Gao, L., Lai, Y.K., Manocha, D.: Vv-net: voxel vae net with group convolutions for point cloud segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8499–8507 (2019). https://doi.org/10.1109/ICCV.2019.00859

  13. Charles, R.Q., Su, H., Kaichun, M., Guibas, L.J.: Pointnet: Deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77–85 (2017). https://doi.org/10.1109/CVPR.2017.16

  14. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/d8bf84be3800d12f74d8b05e9b89836f-Paper.pdf

  15. Jiang, M., Wu, Y., Zhao, T., Zhao, Z., Lu, C.: Pointsift: a sift-like network module for 3D point cloud semantic segmentation (2018). https://doi.org/10.48550/ARXIV.1807.00652

  16. Zhao, H., Jiang, L., Fu, C.W., Jia, J.: Pointweb: enhancing local neighborhood features for point cloud processing. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5560–5568 (2019). https://doi.org/10.1109/CVPR.2019.00571

  17. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3141–3149 (2019). https://doi.org/10.1109/CVPR.2019.00326

  18. Feng, M., Zhang, L., Lin, X., Gilani, S.Z., Mian, A.: Point attention network for semantic segmentation of 3D point clouds. Pattern Recogn. 107, 107446 (2020). https://doi.org/10.1016/j.patcog.2020.107446, https://www.sciencedirect.com/science/article/pii/S0031320320302491

  19. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph cnn for learning on point clouds. ACM Trans. Graph. 38(5) (2019). https://doi.org/10.1145/3326362

  20. Zhang, K., Hao, M., Wang, J., de Silva, C.W., Fu, C.: Linked dynamic graph cnn: learning on point cloud via linking hierarchical features (2019). https://doi.org/10.48550/ARXIV.1904.10014, https://arxiv.org/abs/1904.10014

  21. Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view convolutional neural networks for 3D shape recognition. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 945–953 (2015). https://doi.org/10.1109/ICCV.2015.114

  22. Wang, C., Pelillo, M., Siddiqi, K.: Dominant set clustering and pooling for multi-view 3d object recognition (2019). https://doi.org/10.48550/ARXIV.1906.01592, https://arxiv.org/abs/1906.01592

  23. Feng, Y., Zhang, Z., Zhao, X., Ji, R., Gao, Y.: Gvcnn: group-view convolutional neural networks for 3D shape recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 264–272 (2018). https://doi.org/10.1109/CVPR.2018.00035

  24. Kanezaki, A., Matsushita, Y., Nishida, Y.: Rotationnet: joint object categorization and pose estimation using multiviews from unsupervised viewpoints. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5010–5019 (2018). https://doi.org/10.1109/CVPR.2018.00526

  25. Esteves, C., Xu, Y., Allec-Blanchette, C., Daniilidis, K.: Equivariant multi-view networks. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1568–1577 (2019). https://doi.org/10.1109/ICCV.2019.00165

  26. Han, Z.: 3d2seqviews: aggregating sequential views for 3D global feature learning by CNN with hierarchical attention aggregation. IEEE Trans. Image Process. 28(8), 3986–3999 (2019). https://doi.org/10.1109/TIP.2019.2904460

    Article  MathSciNet  MATH  Google Scholar 

  27. He, X., Huang, T., Bai, S., Bai, X.: View n-gram network for 3D object retrieval. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7514–7523 (2019). https://doi.org/10.1109/ICCV.2019.00761

  28. Yang, Z., Wang, L.: Learning relationships for multi-view 3D object recognition. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7504–7513 (2019). https://doi.org/10.1109/ICCV.2019.00760

  29. Chen, S., Zheng, L., Zhang, Y., Sun, Z., Xu, K.: Veram: view-enhanced recurrent attention model for 3D shape classification. IEEE Trans. Visualization Comput. Graph. 25(12), 3244–3257 (2019). https://doi.org/10.1109/TVCG.2018.2866793

    Article  Google Scholar 

  30. Liu, A.A., et al.: Hierarchical multi-view context modelling for 3D object classification and retrieval. Inf. Sci. 547, 984–995 (2021). https://doi.org/10.1016/j.ins.2020.09.057, https://www.sciencedirect.com/science/article/pii/S0020025520309671

  31. Lin, C., Li, C., Liu, Y., Chen, N., Choi, Y.K., Wang, W.: Point2skeleton: learning skeletal representations from point clouds. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4275–4284 (2021). https://doi.org/10.1109/CVPR46437.2021.00426

  32. Liu, M., Zhang, X., Su, H.: Meshing point clouds with predicted intrinsic-extrinsic ratio guidance. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12353, pp. 68–84. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58598-3_5

    Chapter  Google Scholar 

  33. Rakotosaona, M.J., Guerrero, P., Aigerman, N., Mitra, N., Ovsjanikov, M.: Learning delaunay surface elements for mesh reconstruction. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22–31 (2021). https://doi.org/10.1109/CVPR46437.2021.00009

  34. Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948). https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

    Article  MathSciNet  MATH  Google Scholar 

  35. Vázquez, P.P., Feixas, M., Sbert, M., Heidrich, W.: Viewpoint selection using viewpoint entropy. In: VMV, vol. 1, pp. 273–280. Citeseer (2001)

    Google Scholar 

  36. Vázquez, P.P., Feixas, M., Sbert, M., Heidrich, W.: Automatic view selection using viewpoint entropy and its application to image-based modelling. Comput. Graph. Forum 22(4), 689–700. https://doi.org/10.1111/j.1467-8659.2003.00717.x, https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8659.2003.00717.x

  37. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848

  38. Dosovitskiy, A., et al.: An image is worth 16\(\times \)16 words: transformers for image recognition at scale (2020). https://doi.org/10.48550/ARXIV.2010.11929, https://arxiv.org/abs/2010.11929

  39. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization (2016). https://doi.org/10.48550/ARXIV.1607.06450, https://arxiv.org/abs/1607.06450

  40. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  41. Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus) (2016). https://doi.org/10.48550/ARXIV.1606.08415, https://arxiv.org/abs/1606.08415

  42. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269 (2017). https://doi.org/10.1109/CVPR.2017.243

  43. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). https://doi.org/10.48550/ARXIV.1412.6980, https://arxiv.org/abs/1412.6980

  44. Xu, Y., Zheng, C., Xu, R., Quan, Y., Ling, H.: Multi-view 3D shape recognition via correspondence-aware deep learning. IEEE Trans. Image Process. 30, 5299–5312 (2021). https://doi.org/10.1109/TIP.2021.3082310

    Article  Google Scholar 

  45. Chen, S., Yu, T., Li, P.: Mvt: multi-view vision transformer for 3D object recognition (2021). https://doi.org/10.48550/ARXIV.2110.13083, https://arxiv.org/abs/2110.13083

  46. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jegou, H.: Training data-efficient image transformers & distillation through attention. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 10347–10357. PMLR (2021). https://proceedings.mlr.press/v139/touvron21a.html

  47. Yu, T., Meng, J., Yuan, J.: Multi-view harmonized bilinear network for 3D object recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 186–194 (2018). https://doi.org/10.1109/CVPR.2018.00027

Download references

Acknowledgments

The fnancial support for this work was sponsored by Natural Science Foundation of Shanghai under Grant No. 19ZR1435900.

Author information

Authors and Affiliations

  1. University of Shanghai for Science and Technology, Shanghai, 200093, China

    Wenju Wang, Gang Chen, Haoran Zhou & Xiaolin Wang

Authors
  1. Wenju Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haoran Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaolin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gang Chen .

Editor information

Editors and Affiliations

  1. University of Wollongong, Wollongong, NSW, Australia

    Lei Wang

  2. University of Bonn, Bonn, Germany

    Juergen Gall

  3. University of Adelaide, Adelaide, SA, Australia

    Tat-Jun Chin

  4. National Institute of Informatics, Tokyo, Japan

    Imari Sato

  5. Johns Hopkins University, Baltimore, MD, USA

    Rama Chellappa

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 468 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, W., Chen, G., Zhou, H., Wang, X. (2023). OVPT: Optimal Viewset Pooling Transformer for 3D Object Recognition. In: Wang, L., Gall, J., Chin, TJ., Sato, I., Chellappa, R. (eds) Computer Vision – ACCV 2022. ACCV 2022. Lecture Notes in Computer Science, vol 13841. Springer, Cham. https://doi.org/10.1007/978-3-031-26319-4_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-26319-4_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-26318-7

  • Online ISBN: 978-3-031-26319-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation