Skip to main content
Springer Nature Link
Log in

OPD: Single-View 3D Openable Part Detection

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13699))

Included in the following conference series:

  • 3494 Accesses

Abstract

We address the task of predicting what parts of an object can open and how they move when they do so. The input is a single image of an object, and as output we detect what parts of the object can open, and the motion parameters describing the articulation of each openable part. To tackle this task, we create two datasets of 3D objects: OPDSynth based on synthetic objects, and OPDReal based on RGBD reconstructions of real objects. We then design OpdRcnn, a neural architecture that detects openable parts and predicts their motion parameters. Our experiments show that this is a challenging task especially when considering generalization across object categories, and the limited amount of information in a single image. Our architecture outperforms baselines and prior work especially for RGB image inputs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abbatematteo, B., Tellex, S., Konidaris, G.: Learning to generalize kinematic models to novel objects. In: Proceedings of the 3rd Conference on Robot Learning (2019)

    Google Scholar 

  2. Batra, D., et al.: Rearrangement: A challenge for embodied AI. arXiv preprint arXiv:2011.01975 (2020)

  3. Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: YOLOV4: optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)

  4. Chen, X., Mottaghi, R., Liu, X., Fidler, S., Urtasun, R., Yuille, A.: Detect what you can: detecting and representing objects using holistic models and body parts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1971–1978 (2014)

    Google Scholar 

  5. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2961–2969 (2017)

    Google Scholar 

  6. Hu, R., Li, W., Van Kaick, O., Shamir, A., Zhang, H., Huang, H.: Learning to predict part mobility from a single static snapshot. ACM Trans. Graph. (TOG) 36(6), 227 (2017)

    Article  Google Scholar 

  7. Hu, R., Savva, M., van Kaick, O.: Functionality representations and applications for shape analysis. Comput. Graph. Forum 37(2), 603–624 (2018)

    Article  Google Scholar 

  8. Huang, J., et al.: MultiBodySync: multi-body segmentation and motion estimation via 3D scan synchronization. arXiv preprint arXiv:2101.06605 (2021)

  9. Jain, A., Lioutikov, R., Niekum, S.: ScrewNet: Category-independent articulation model estimation from depth images using screw theory. arXiv preprint arXiv:2008.10518 (2020)

  10. Li, H., Wan, G., Li, H., Sharf, A., Xu, K., Chen, B.: Mobility fitting using 4D RANSAC. Comput. Graph. Forum 35(5), 79–88 (2016)

    Article  Google Scholar 

  11. Li, X., Wang, H., Yi, L., Guibas, L., Abbott, A.L., Song, S.: Category-level articulated object pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  12. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  13. Liu, L., Xu, W., Fu, H., Qian, S., Han, Y., Lu, C.: AKB-48: a real-world articulated object knowledge base. arXiv preprint arXiv:2202.08432 (2022)

  14. Liu, Q., Qiu, W., Wang, W., Hager, G.D., Yuille, A.L.: Nothing but geometric constraints: a model-free method for articulated object pose estimation. arXiv preprint arXiv:2012.00088 (2020)

  15. Lu, C., et al.: Beyond holistic object recognition: enriching image understanding with part states. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6955–6963 (2018)

    Google Scholar 

  16. Martín-Martín, R., Eppner, C., Brock, O.: The RBO dataset of articulated objects and interactions. Int. J. Robot. Res. 38(9), 1013–1019 (2019)

    Article  Google Scholar 

  17. Mitra, N.J., Yang, Y.L., Yan, D.M., Li, W., Agrawala, M.: Illustrating how mechanical assemblies work. ACM Trans. Graph.-TOG 29(4), 58 (2010)

    Google Scholar 

  18. Mittal, M., Hoeller, D., Farshidian, F., Hutter, M., Garg, A.: Articulated object interaction in unknown scenes with whole-body mobile manipulation. arXiv preprint arXiv:2103.10534 (2021)

  19. Mo, K., et al.: PartNet: a large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 909–918 (2019)

    Google Scholar 

  20. Sharf, A., Huang, H., Liang, C., Zhang, J., Chen, B., Gong, M.: Mobility-trees for indoor scenes manipulation. Comput. Graph. Forum 33(1), 2–14 (2014)

    Article  Google Scholar 

  21. Shridhar, M., et al.: Alfred: a benchmark for interpreting grounded instructions for everyday tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10740–10749 (2020)

    Google Scholar 

  22. Srivastava, S., et al.: BEHAVIOR: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In: Conference on Robot Learning, pp. 477–490. PMLR (2022)

    Google Scholar 

  23. Szot, A., et al.: Habitat 2.0: training home assistants to rearrange their habitat. Adv. Neural. Inf. Process. Syst. 34, 251–266 (2021)

    Google Scholar 

  24. Varol, G., et al.: Learning from synthetic humans. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 109–117 (2017)

    Google Scholar 

  25. Waechter, M., Moehrle, N., Goesele, M.: Let there be color! Large-scale texturing of 3d reconstructions. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 836–850. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_54

    Chapter  Google Scholar 

  26. Wang, H., Sridhar, S., Huang, J., Valentin, J., Song, S., Guibas, L.J.: Normalized object coordinate space for category-level 6D object pose and size estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2642–2651 (2019)

    Google Scholar 

  27. Wang, J., Yuille, A.L.: Semantic part segmentation using compositional model combining shape and appearance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1788–1797 (2015)

    Google Scholar 

  28. Wang, P., Shen, X., Lin, Z., Cohen, S., Price, B., Yuille, A.L.: Joint object and part segmentation using deep learned potentials. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1573–1581 (2015)

    Google Scholar 

  29. Wang, X., Zhou, B., Fang, H., Chen, X., Zhao, Q., Xu, K.: Learning to group and label fine-grained shape components. ACM Trans. Graph. (TOG) 37(6), 1–14 (2018)

    Google Scholar 

  30. Wang, X., Zhou, B., Shi, Y., Chen, X., Zhao, Q., Xu, K.: Shape2Motion: joint analysis of motion parts and attributes from 3D shapes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8876–8884 (2019)

    Google Scholar 

  31. Weihs, L., Deitke, M., Kembhavi, A., Mottaghi, R.: Visual room rearrangement. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  32. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019). https://github.com/facebookresearch/detectron2

  33. Xia, F., Wang, P., Chen, X., Yuille, A.L.: Joint multi-person pose estimation and semantic part segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6769–6778 (2017)

    Google Scholar 

  34. Xiang, F., et al.: SAPIEN: a simulated part-based interactive environment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11097–11107 (2020)

    Google Scholar 

  35. Xu, W., et al.: Joint-aware manipulation of deformable models. ACM Trans. Graph. (TOG) 28(3), 1–9 (2009)

    Article  Google Scholar 

  36. Yan, Z., et al.: RPM-Net: recurrent prediction of motion and parts from point cloud. ACM Trans. Graph. (TOG) 38(6), 240 (2019)

    Article  Google Scholar 

  37. Yi, L., Huang, H., Liu, D., Kalogerakis, E., Su, H., Guibas, L.: Deep part induction from articulated object pairs. ACM Trans. Graph. (TOG) 37(6), 209 (2019)

    Google Scholar 

  38. Yi, L., et al.: A scalable active framework for region annotation in 3D shape collections. ACM Trans. Graph. (TOG) 35(6), 1–12 (2016)

    Article  Google Scholar 

  39. Zeng, V., Lee, T.E., Liang, J., Kroemer, O.: Visual identification of articulated object parts. arXiv preprint arXiv:2012.00284 (2020)

  40. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ADE20K dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  41. Zhou, Q.Y., Park, J., Koltun, V.: Open3D: a modern library for 3D data processing. arXiv:1801.09847 (2018)

Download references

Acknowledgements

This work was funded in part by a Canada CIFAR AI Chair, a Canada Research Chair and NSERC Discovery Grant, and enabled in part by support from WestGrid and Compute Canada. We thank Sanjay Haresh for help with scanning and video narration, Yue Ruan for scanning and data annotation, and Supriya Pandhre, Xiaohao Sun, and Qirui Wu for help with data annotation.

Author information

Authors and Affiliations

  1. Simon Fraser University, Burnaby, Canada

    Hanxiao Jiang, Yongsen Mao, Manolis Savva & Angel X. Chang

Authors
  1. Hanxiao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongsen Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manolis Savva
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Angel X. Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanxiao Jiang .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 7932 KB)

Supplementary material 2 (mp4 5472 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jiang, H., Mao, Y., Savva, M., Chang, A.X. (2022). OPD: Single-View 3D Openable Part Detection. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13699. Springer, Cham. https://doi.org/10.1007/978-3-031-19842-7_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19842-7_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19841-0

  • Online ISBN: 978-3-031-19842-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics