Nikolas Lamb
ABSTRACT
3D printing offers the opportunity to perform automated restoration of objects to reduce household waste, restore objects of cultural heritage, and automate repair in medical and manufacturing domains. We present an approach that takes a 3D model of a broken object and retrieves proxy 3D models of corresponding complete objects from a library of 3D models, with the goal of using the complete proxy to repair the broken object. We input multi-view renders and point cloud representations of the query to neural networks that output learned visual and geometric feature encodings. Our approach returns complete proxies that are visually and geometrically similar to the broken query object model by searching for the learned encodings in the complete models library. We demonstrate results for retrieval of complete proxies for broken object models with breaks generated synthetically using models from the ShapeNet dataset, and from publicly available datasets of scanned everyday objects and cultural heritage objects. By combining visual and geometric features, our approach shows consistently lower Chamfer distance than when either feature is used alone. Our approach outperforms the existing state-of-the-art method in retrieval of proxies for broken objects in terms of the Chamfer distance. The 3D proxies returned by our approach enable understanding of object geometry to identify object portions requiring repair, to incorporate user preferences, and to generate 3D printable restoration components. Our code to perform broken object model generation, feature extraction, and object retrieval is available at https://git.io/JuKaJ.
Supplemental Material
- Masaki Aono and Wataru Iwabuchi. 2020. Part-In-Whole Type 3d Partial Shape Retreival Based On Connected Faces With Pointnet Features. In 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, Piscataway, NJ, 1207–1215.Google Scholar
- Lucia Arbace, Elisabetta Sonnino, Marco Callieri, Matteo Dellepiane, Matteo Fabbri, Antonio Iaccarino Idelson, and Roberto Scopigno. 2013. Innovative uses of 3D digital technologies to assist the restoration of a fragmented terracotta statue. Journal of Cultural Heritage 14, 4 (2013), 332–345.Google ScholarCross Ref
- Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X Chang, and Matthias Nießner. 2019a. Scan2cad: Learning cad model alignment in rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 2614–2623.Google ScholarCross Ref
- Armen Avetisyan, Angela Dai, and Matthias Nießner. 2019b. End-to-end cad model retrieval and 9dof alignment in 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, Piscataway, NJ, 2551–2560.Google ScholarCross Ref
- Song Bai, Xiang Bai, Zhichao Zhou, Zhaoxiang Zhang, and Longin Jan Latecki. 2016. Gift: A real-time and scalable 3d shape search engine. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 5023–5032.Google ScholarCross Ref
- Vincent Barra and Silvia Biasotti. 2013. 3D shape retrieval using kernels on extended Reeb graphs. Pattern Recognition 46, 11 (2013), 2985–2999.Google ScholarCross Ref
- Silvia Biasotti, Daniela Giorgi, Michela Spagnuolo, and Bianca Falcidieno. 2008. Reeb graphs for shape analysis and applications. Theoretical computer science 392, 1-3 (2008), 5–22.Google Scholar
- Mario Botsch and Olga Sorkine. 2007. On linear variational surface deformation methods. IEEE transactions on visualization and computer graphics 14, 1(2007), 213–230.Google Scholar
- Alexander M Bronstein, Michael M Bronstein, Leonidas J Guibas, and Maks Ovsjanikov. 2011. Shape google: Geometric words and expressions for invariant shape retrieval. ACM Transactions on Graphics (TOG) 30, 1 (2011), 1–20.Google ScholarDigital Library
- Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, 2015. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 1, 1 (2015), 1–11.Google Scholar
- Yang Chen, Guanlan Liu, Yaming Xu, Pai Pan, and Yin Xing. 2021. PointNet++ Network Architecture with Individual Point Level and Global Features on Centroid for ALS Point Cloud Classification. Remote Sensing 13, 3 (2021), 472.Google ScholarCross Ref
- Sungjoon Choi, Qian-Yi Zhou, Stephen Miller, and Vladlen Koltun. 2016. A Large Dataset of Object Scans. arXiv:1602.02481 1, 1 (2016), 1–7.Google Scholar
- Christopher Choy and Junha Lee. 2019. Open Universal Correspondence Network. https://github.com/chrischoy/open-ucn.Google Scholar
- Christopher Choy, Jaesik Park, and Vladlen Koltun. 2019. Fully convolutional geometric features. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, Piscataway, NJ, 8958–8966.Google ScholarCross Ref
- Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias Nießner. 2018. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 4578–4587.Google ScholarCross Ref
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 248–255.Google ScholarCross Ref
- Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. 2018. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, Berlin, Germany, 52–68.Google ScholarDigital Library
- Martin A Fischler and Robert C Bolles. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6 (1981), 381–395.Google ScholarDigital Library
- Rao Fu, Jie Yang, Jiawei Sun, Fang-Lue Zhang, Yu-Kun Lai, and Lin Gao. 2020. RISA-Net: Rotation-Invariant Structure-Aware Network for Fine-Grained 3D Shape Retrieval. arXiv preprint arXiv:2010.00973 1, 1 (2020), 1–23.Google Scholar
- Thomas Funkhouser, Michael Kazhdan, Philip Shilane, Patrick Min, William Kiefer, Ayellet Tal, Szymon Rusinkiewicz, and David Dobkin. 2004. Modeling by example. ACM transactions on graphics (TOG) 23, 3 (2004), 652–663.Google Scholar
- Takahiko Furuya and Ryutarou Ohbuchi. 2016. Deep Aggregation of Local 3D Geometric Features for 3D Model Retrieval.. In BMVC, Vol. 7. The British Machine Vision Association and Society for Pattern Recognition, Durham, UK, 8.Google Scholar
- Robert Gregor, Ivan Sipiran, Georgios Papaioannou, Tobias Schreck, Anthousis Andreadis, and Pavlos Mavridis. 2014. Towards Automated 3D Reconstruction of Defective Cultural Heritage Objects. In EUROGRAPHICS Workshops on Graphics and Cultural Heritag. EUROGRAPHICS Association, Geneve, Switzerland, 135–144.Google Scholar
- Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 2018. A papier-mâché approach to learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 216–224.Google ScholarCross Ref
- Zhizhong Han, Honglei Lu, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL Philip Chen. 2019. 3D2SeqViews: Aggregating sequential views for 3D global feature learning by CNN with hierarchical attention aggregation. IEEE Transactions on Image Processing 28, 8 (2019), 3986–3999.Google ScholarCross Ref
- Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai. 2018. Triplet-center loss for multi-view 3d object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 1945–1954.Google ScholarCross Ref
- Wataru Iwabuchi and Masaki Aono. 2018. 3d cnn based partial 3d shape retrieval focusing on local features. In 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, Piscataway, NJ, 1523–1529.Google ScholarCross Ref
- Jianwen Jiang, Di Bao, Ziqiang Chen, Xibin Zhao, and Yue Gao. 2019. MLVCNN: Multi-loop-view convolutional neural network for 3D shape retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI, Palo Alto, California, 8513–8520.Google ScholarDigital Library
- Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data 1, 1 (2019), 535–547.Google Scholar
- Nikolas Lamb, Sean Banerjee, and Natasha Kholgade Banerjee. 2019. Automated reconstruction of smoothly joining 3D printed restorations to fix broken objects. In Proceedings of the ACM Symposium on Computational Fabrication. ACM, New York, NY, 3.Google ScholarDigital Library
- Guillaume Lavoué. 2012. Combination of bag-of-words descriptors for robust partial shape retrieval. The Visual Computer 28, 9 (2012), 931–942.Google ScholarDigital Library
- Bo Li, Afzal Godil, and Henry Johan. 2014. Hybrid shape descriptor and meta similarity generation for non-rigid and partial 3D model retrieval. Multimedia tools and applications 72, 2 (2014), 1531–1560.Google Scholar
- Roee Litman, Alex Bronstein, Michael Bronstein, and Umberto Castellani. 2014. Supervised learning of bag-of-features shape descriptors using sparse coding. In Computer Graphics Forum, Vol. 33. Wiley, Hoboken, NJ, 127–136.Google Scholar
- David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 2 (2004), 91–110.Google ScholarDigital Library
- Pavlos Mavridis, Anthousis Andreadis, and Georgios Papaioannou. 2015. Fractured Object Reassembly via Robust Surface Registration.. In Eurographics (Short Papers). Eurographics, Geneve, Switzerland, 21–24.Google Scholar
- Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. 2019. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 4460–4470.Google ScholarCross Ref
- Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. 2019. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 909–918.Google ScholarCross Ref
- Weizhi Nie, Weijie Wang, Anan Liu, Jie Nie, and Yuting Su. 2019. HGAN: Holistic Generative Adversarial Networks for Two-dimensional Image-based Three-dimensional Object Retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15, 4 (2019), 1–24.Google ScholarDigital Library
- Georgios Papaioannou, Tobias Schreck, Anthousis Andreadis, Pavlos Mavridis, Robert Gregor, Ivan Sipiran, and Konstantinos Vardis. 2017. From reassembly to object completion: A complete systems pipeline. Journal on Computing and Cultural Heritage (JOCCH) 10, 2 (2017), 1–22.Google ScholarDigital Library
- Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 165–174.Google ScholarCross Ref
- Angelia Payne, Keenan Cole, Katie Simon, Christopher Goodmaster, and Fredrick Limp. 2009. Designing the next generation virtual museum: Making 3D artifacts available for viewing and download. In Making History Interactive: Proceedings of the 37th Annual International Conference on Computer Applications and Quantitative Methods in Archaeology (CAA), Vol. 3. CAA, 1–6.Google Scholar
- Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. 2016. Volumetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 5648–5656.Google ScholarCross Ref
- Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413 1, 1 (2017), 1–14.Google Scholar
- Jason Rock, Tanmay Gupta, Justin Thorsen, JunYoung Gwak, Daeyun Shin, and Derek Hoiem. 2015. Completing 3d object shape from one depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 2484–2493.Google ScholarCross Ref
- Muhammad Sarmad, Hyunjoo Jenny Lee, and Young Min Kim. 2019. Rl-gan-net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Piscataway, NJ, 5898–5907.Google ScholarCross Ref
- Konstantinos Sfikas, Ioannis Pratikakis, Anestis Koutsoudis, Michalis Savelonas, and Theoharis Theoharis. 2016. Partial matching of 3D cultural heritage objects using panoramic views. Multimedia Tools and Applications 75, 7 (2016), 3693–3707.Google ScholarDigital Library
- Konstantinos Sfikas, Ioannis Pratikakis, and Theoharis Theoharis. 2018. Ensemble of PANORAMA-based convolutional neural networks for 3D model classification and retrieval. Computers & Graphics 71(2018), 208–218.Google ScholarCross Ref
- Baoguang Shi, Song Bai, Zhichao Zhou, and Xiang Bai. 2015. Deeppano: Deep panoramic representation for 3-d shape recognition. IEEE Signal Processing Letters 22, 12 (2015), 2339–2343.Google ScholarCross Ref
- Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 1, 1 (2014), 1–14.Google Scholar
- Sekou Singare, Yaxiong Liu, Dichen Li, Bingheng Lu, and Sanhu He. 2008. Individually prefabricated prosthesis for maxilla reconstruction. Journal of Prosthodontics 17, 2 (2008), 135–140.Google ScholarCross Ref
- Ivan Sipiran. 2018. Completion of cultural heritage objects with rotational symmetry. In EUROGRAPHICS Workshop on 3D Object Retrieval. EUROGRAPHICS Association, Geneve, Switzerland, 87–93.Google Scholar
- Hyeontae Son and Young Min Kim. 2020. SAUM: Symmetry-Aware Upsampling Module for Consistent Point Cloud Completion. In Proceedings of the Asian Conference on Computer Vision. ACM, New York, NY, 1–17.Google Scholar
- Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and H-P Seidel. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. ACM, New York, NY, 175–184.Google ScholarDigital Library
- David Stutz and Andreas Geiger. 2020. Learning 3d shape completion under weak supervision. International Journal of Computer Vision 128, 5 (2020), 1162–1181.Google ScholarDigital Library
- Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision. IEEE, Piscataway, NJ, 945–953.Google ScholarDigital Library
- Hedi Tabia and Hamid Laga. 2015. Covariance-based descriptors for efficient 3D shape matching, retrieval, and classification. IEEE transactions on multimedia 17, 9 (2015), 1591–1603.Google ScholarDigital Library
- Roberto Toldo, Umberto Castellani, and Andrea Fusiello. 2010. The bag of words approach for retrieval and categorization of 3D objects. The Visual Computer 26, 10 (2010), 1257–1268.Google ScholarDigital Library
- Mikaela Angelina Uy, Jingwei Huang, Minhyuk Sung, Tolga Birdal, and Leonidas Guibas. 2020. Deformation-aware 3d model embedding and retrieval. In European Conference on Computer Vision. Springer, Berlin, Germany, 397–413.Google ScholarDigital Library
- Yan Wang, Jie Feng, Zhixiang Wu, Jun Wang, and Shih-Fu Chang. 2014. From low-cost depth sensors to cad: Cross-domain 3d shape retrieval via regression tree fields. In European Conference on Computer Vision. Springer, Berlin, Germany, 489–504.Google ScholarCross Ref
- Zihao Wang and Hongwei Lin. 2020. 3D shape retrieval based on Laplace operator and joint Bayesian model. Visual Informatics 4, 3 (2020), 69–76.Google ScholarCross Ref
- Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 1912–1920.Google Scholar
- Xu Yan. 2019. Pointnet/Pointnet++ Pytorch.Google Scholar
- Mohsen Yavartanoo, Eu Young Kim, and Kyoung Mu Lee. 2018. Spnet: Deep 3d object classification and retrieval using stereographic projection. In Asian Conference on Computer Vision. Springer, Berlin, Germany, 691–706.Google Scholar
- Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 2017. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Piscataway, NJ, 1802–1811.Google ScholarCross Ref
- Yuhe Zhang, Kang Li, Xiaoxue Chen, Shunli Zhang, and Guohua Geng. 2018. A multi feature fusion method for reassembly of 3D cultural heritage artifacts. Journal of Cultural Heritage 33 (2018), 191–200.Google ScholarCross Ref
- Jing Zhu, Fan Zhu, Edward K Wong, and Yi Fang. 2015. Learning pairwise neural network encoder for depth image-based 3d model retrieval. In Proceedings of the 23rd ACM international conference on Multimedia. ACM, New York, NY, 1227–1230.Google ScholarDigital Library
Recommendations
Automated reconstruction of smoothly joining 3D printed restorations to fix broken objects
SCF '19: Proceedings of the 3rd Annual ACM Symposium on Computational FabricationIn this work, we provide an approach to automatically reconstruct a 3D printed restoration piece for a broken object from 3D scanned meshes of the broken object and an original counterpart. Our approach provides two contributions to reconstruct a ...
Recognition of subjective objects based on one gold sample
SSIP'05: Proceedings of the 5th WSEAS international conference on Signal, speech and image processingHuman visual system can recognize incomplete contours and objects easily. However, these kind of recognition tasks are challenging in computer and robot vision. This paper demonstrates how combination of genetic algorithm and morphology operations can ...
Computational design and fabrication of soft pneumatic objects with desired deformations
We present an end-to-end solution for design and fabrication of soft pneumatic objects with desired deformations. Given a 3D object with its rest and deformed target shapes, our method automatically optimizes the chamber structure and material ...
Comments