skip to main content
research-article

Enhancing the Aesthetics of 3D Shapes via Reference-based Editing

Published: 19 November 2024 Publication History

Abstract

While there have been previous works that explored methods to enhance the aesthetics of images, the automated beautification of 3D shapes has been limited to specific shapes such as 3D face models. In this paper, we introduce a framework to automatically enhance the aesthetics of general 3D shapes. Our approach employs a reference-based beautification strategy. We first performed data collection to gather the aesthetics ratings of various 3D shapes to create a 3D shape aesthetics dataset. Then we perform reference-based editing to edit the input shape and beautify it by making it look more like some reference shape that is aesthetic. Specifically, we propose a reference-guided global deformation framework to coherently deform the input shape such that its structural proportions will be closer to those of the reference shape. We then optionally transplant some local aesthetic parts from the reference to the input to obtain the beautified output shapes. Comparisons show that our reference-guided 3D deformation algorithm outperforms existing techniques. Furthermore, quantitative and qualitative evaluations demonstrate that the performance of our aesthetics enhancement framework is consistent with both human perception and existing 3D shape aesthetics assessment.

References

[1]
Zarghili Arsalane, Majda Aicha, Oufkir Ayat Allah, et al. 2022. 3D facial attractiveness enhancement using free form deformation. Journal of King Saud University-Computer and Information Sciences 34, 6 (2022), 3497--3505.
[2]
Tunç Ozan Aydın, Aljoscha Smolic, and Markus Gross. 2014. Automated aesthetic analysis of photographic images. IEEE transactions on visualization and computer graphics 21, 1 (2014), 31--42.
[3]
Sai Praveen Bangaru, Michaël Gharbi, Fujun Luan, Tzu-Mao Li, Kalyan Sunkavalli, Milos Hasan, Sai Bi, Zexiang Xu, Gilbert Bernstein, and Fredo Durand. 2022. Differentiable rendering of neural sdfs through reparameterization. In SIGGRAPH Asia 2022 Conference Papers. 1--9.
[4]
Steve Bergen and Brian J Ross. 2013. Aesthetic 3D model evolution. Genetic Programming and Evolvable Machines 14 (2013), 339--367.
[5]
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. 2015. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015).
[6]
Minchan Chen and Manfred Lau. 2022. Learning 3D Shape Aesthetics Globally and Locally. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 579--588.
[7]
Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. 2020. Bsp-net: Generating compact meshes via binary space partitioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 45--54.
[8]
Zhiqin Chen and Hao Zhang. 2019. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5939--5948.
[9]
Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z Wang. 2006. Studying aesthetics in photographic images using a computational approach. In Computer Vision-ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006, Proceedings, Part III 9. Springer, 288--301.
[10]
Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2017. Image aesthetic assessment: An experimental survey. IEEE Signal Processing Magazine 34, 4 (2017), 80--106.
[11]
Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2018. Aesthetic-driven image enhancement by adversarial learning. In Proceedings of the 26th ACM international conference on Multimedia. 870--878.
[12]
Yu Deng, Jiaolong Yang, and Xin Tong. 2021. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10286--10296.
[13]
Kapil Dev and Manfred Lau. 2022. Learning perceptual aesthetics of 3D shapes from multiple views. IEEE Computer Graphics and Applications 42, 1 (2022), 20--31.
[14]
Nir Diamant, Dean Zadok, Chaim Baskin, Eli Schwartz, and Alex M Bronstein. 2019. Beholder-gan: Generation and beautification of facial images with conditioning on their beauty level. In 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 739--743.
[15]
Jakub Fišer, Paul Asente, Stephen Schiller, and Daniel Sỳkora. 2016. Advanced drawing beautification with shipshape. Computers & Graphics 56 (2016), 46--58.
[16]
Ran Gal, Olga Sorkine, Niloy J Mitra, and Daniel Cohen-Or. 2009. iWIRES: An analyze-and-edit approach to shape manipulation. In ACM SIGGRAPH 2009 papers. 1--10.
[17]
Lin Gao, Jie Yang, Yi-Ling Qiao, Yu-Kun Lai, Paul L Rosin, Weiwei Xu, and Shihong Xia. 2018. Automatic unpaired shape deformation transfer. ACM Transactions on Graphics (TOG) 37, 6 (2018), 1--15.
[18]
William Gao, Noam Aigerman, Groueix Thibault, Vladimir Kim, and Rana Hanocka. 2023. TextDeformer: Geometry Manipulation using Text Guidance. In ACM Transactions on Graphics (SIGGRAPH).
[19]
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 2019. Unsupervised cycle-consistent deformation for shape matching. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 123--133.
[20]
Guanjun Guo, Hanzi Wang, Chunhua Shen, Yan Yan, and Hong-Yuan Mark Liao. 2018. Automatic image cropping for visual aesthetic enhancement using deep neural networks and cascaded regression. IEEE Transactions on Multimedia 20, 8 (2018), 2073--2085.
[21]
Kunal Gupta and Manmohan Chandraker. 2020. Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows. Advances in Neural Information Processing Systems 33 (2020), 1747--1758.
[22]
Rana Hanocka, Noa Fish, Zhenhua Wang, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2018. Alignet: Partial-shape agnostic alignment via unsupervised learning. ACM Transactions on Graphics (TOG) 38, 1 (2018), 1--14.
[23]
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. 2022. AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1--19.
[24]
Ruizhen Hu, Wenchao Li, Oliver Van Kaick, Hui Huang, Melinos Averkiou, Daniel Cohen-Or, and Hao Zhang. 2017. Co-locating style-defining elements on 3d shapes. ACM Transactions on Graphics (TOG) 36, 3 (2017), 1--15.
[25]
Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu. 2022. Neural template: Topology-aware reconstruction and disentangled generation of 3d meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18572--18582.
[26]
Md Baharul Islam, Wong Lai-Kuan, and Wong Chee-Onn. 2017. A survey of aesthetics-driven image recomposition. Multimedia Tools and Applications 76 (2017), 9517--9542.
[27]
Chiyu Jiang, Jingwei Huang, Andrea Tagliasacchi, and Leonidas J Guibas. 2020. Shape-flow: Learnable deformation flows among 3d shapes. Advances in Neural Information Processing Systems 33 (2020), 9745--9757.
[28]
Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. 2016. Photo aesthetics ranking network with attributes and content adaptation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part I 14. Springer, 662--679.
[29]
Manfred Lau, Kapil Dev, Weiqi Shi, Julie Dorsey, and Holly Rushmeier. 2016. Tactile mesh saliency. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--11.
[30]
Tommer Leyvand, Daniel Cohen-Or, Gideon Dror, and Dani Lischinski. 2008. Data-driven enhancement of facial attractiveness. In ACM SIGGRAPH 2008 papers. 1--9.
[31]
Congcong Li, Alexander C Loui, and Tsuhan Chen. 2010. Towards aesthetics: A photo quality assessment and photo selection system. In Proceedings of the 18th ACM international conference on Multimedia. 827--830.
[32]
Jianshu Li, Chao Xiong, Luoqi Liu, Xiangbo Shu, and Shuicheng Yan. 2015. Deep face beautification. In Proceedings of the 23rd ACM international conference on Multimedia. 793--794.
[33]
Qiqi Liao, Xiaogang Jin, and Wenting Zeng. 2012. Enhancing the symmetry and proportion of 3D face geometry. IEEE transactions on visualization and computer graphics 18, 10 (2012), 1704--1716.
[34]
Dong Liu, Rohit Puri, Nagendra Kamath, and Subhabrata Bhattacharya. 2020. Composition-aware image aesthetics assessment. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3569--3578.
[35]
Ligang Liu, Renjie Chen, Lior Wolf, and Daniel Cohen-Or. 2010. Optimizing photo composition. In Computer graphics forum, Vol. 29. Wiley Online Library, 469--478.
[36]
Minghua Liu, Minhyuk Sung, Radomir Mech, and Hao Su. 2021. Deepmetahandles: Learning deformation meta-handles of 3d meshes with biharmonic coordinates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12--21.
[37]
Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, and James Z Wang. 2015. Rating image aesthetics using deep learning. IEEE Transactions on Multimedia 17, 11 (2015), 2021--2034.
[38]
Yiwei Ma, Xiaoqing Zhang, Xiaoshuai Sun, Jiayi Ji, Haowei Wang, Guannan Jiang, Weilin Zhuang, and Rongrong Ji. 2023. X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2749--2760.
[39]
Mayra D Barrera Machuca, Paul Asente, Wolfgang Stuerzlinger, Jingwan Lu, and Byungmoon Kim. 2018. Multiplanes: Assisted freehand vr sketching. In Proceedings of the 2018 ACM Symposium on Spatial User Interaction. 36--47.
[40]
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2022. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13492--13502.
[41]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 1 (2021), 99--106.
[42]
Kenjiro T Miura and Gobithaasan RU. 2014. Aesthetic curves and surfaces in computer aided geometric design. International Journal of Automation Technology 8, 3 (2014), 304--316.
[43]
Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. 2019. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 909--918.
[44]
Masashi Nishiyama, Takahiro Okabe, Imari Sato, and Yoichi Sato. 2011. Aesthetic quality classification of photographs based on color harmony. In CVPR 2011. IEEE, 33--40.
[45]
Alice J O'Toole, Theodore Price, Thomas Vetter, James C Bartlett, and Volker Blanz. 1999. 3D shape and 2D surface textures of human faces: The role of "averages" in attractiveness and age. Image and Vision Computing 18, 1 (1999), 9--19.
[46]
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Love-grove. 2019. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 165--174.
[47]
Binh Pham. 1999. Design for aesthetics: interactions of design variables and aesthetic properties. In Human Vision and Electronic Imaging IV, Vol. 3644. SPIE, 364--371.
[48]
Binh Pham and Jinglan Zhang. 2003. A fuzzy shape specification system to support design for aesthetics. Soft Computing in Measurement and Information Acquisition (2003), 39--50.
[49]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748--8763.
[50]
Jian Ren, Xiaohui Shen, Zhe Lin, Radomir Mech, and David J Foran. 2017. Personalized image aesthetics. In Proceedings of the IEEE international conference on computer vision. 638--647.
[51]
Mattia Segu, Margarita Grinvald, Roland Siegwart, and Federico Tombari. 2020. 3dsnet: Unsupervised shape-to-shape 3d style transfer. arXiv preprint arXiv:2011.13388 (2020).
[52]
Andrei Sharf, Marina Blumenkrants, Ariel Shamir, and Daniel Cohen-Or. 2006. Snap-paste: an interactive technique for easy mesh composition. The Visual Computer 22 (2006), 835--844.
[53]
I-Chao Shen. 2021. Data-Driven Sketch Beautification With Neural Feature Representation. IEEE Computer Graphics and Applications 42, 4 (2021), 72--79.
[54]
Chaoyue Song, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. 2021. 3D pose transfer with correspondence learning and mesh refinement. Advances in Neural Information Processing Systems 34 (2021), 3108--3120.
[55]
Chaoyue Song, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. 2023. Unsupervised 3d pose transfer with cross consistency and dual reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
[56]
Wei-Tse Sun, Ting-Hsuan Chao, Yin-Hsi Kuo, and Winston H Hsu. 2017. Photo filter recommendation by category-aware aesthetic learning. IEEE Transactions on Multimedia 19, 8 (2017), 1870--1880.
[57]
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. 2022a. Clipnerf: Text-and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3835--3844.
[58]
Haoran Wang, Jiaxin Li, Alexandru Telea, Jiří Kosinka, and Zizhao Wu. 2022b. USTNet: Unsupervised Shape-to-Shape Translation via Disentangled Representations. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 141--152.
[59]
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 (2021).
[60]
Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. 2019. 3dn: 3d deformation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1038--1046.
[61]
Zhijie Wu, Xiang Wang, Di Lin, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2019. Sagnet: Structure-aware generative network for 3d-shape modeling. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--14.
[62]
Qinjie Xiao, Xiangjun Tang, You Wu, Leyang Jin, Yong-Liang Yang, and Xiaogang Jin. 2020. Deep shapely portraits. In Proceedings of the 28th ACM International Conference on Multimedia. 1800--1808.
[63]
Qinjie Xiao, You Wu, Dinghong Wang, Yong-Liang Yang, and Xiaogang Jin. 2021. Beauty3DFaceNet: deep geometry and texture fusion for 3D facial attractiveness prediction. Computers & Graphics 98 (2021), 11--18.
[64]
Kai Xu, Honghua Li, Hao Zhang, Daniel Cohen-Or, Yueshan Xiong, and Zhi-Quan Cheng. 2010. Style-content separation by anisotropic part scales. In ACM SIGGRAPH Asia 2010 papers. 1--10.
[65]
Pengfei Xu, Guohang Yan, Hongbo Fu, Takeo Igarashi, Chiew-Lan Tai, and Hui Huang. 2019. Global Beautification of 2D and 3D Layouts With Interactive Ambiguity Resolution. IEEE transactions on visualization and computer graphics 27, 4 (2019), 2355--2368.
[66]
Zhicheng Yan, Hao Zhang, Baoyuan Wang, Sylvain Paris, and Yizhou Yu. 2016. Automatic photo adjustment using deep neural networks. ACM Transactions on Graphics (TOG) 35, 2 (2016), 1--15.
[67]
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. 2019. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision. 4541--4550.
[68]
Yuzhe Yang, Liwu Xu, Leida Li, Nan Qie, Yaqian Li, Peng Zhang, and Yandong Guo. 2022. Personalized image aesthetics assessment with rich attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19861--19869.
[69]
Wang Yifan, Noam Aigerman, Vladimir G Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung. 2020. Neural cages for detail-preserving 3d deformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 75--83.
[70]
Kangxue Yin, Jun Gao, Maria Shugrina, Sameh Khamis, and Sanja Fidler. 2021. 3dstylenet: Creating 3d shapes with geometric and texture style variations. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12456--12465.
[71]
Mehmet Ersin Yumer, Siddhartha Chaudhuri, Jessica K Hodgins, and Levent Burak Kara. 2015. Semantic shape editing using deformation handles. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1--12.
[72]
Hancheng Zhu, Yong Zhou, Leida Li, Yaqian Li, and Yandong Guo. 2021. Learning personalized image aesthetics from subjective and objective attributes. IEEE Transactions on Multimedia (2021).
[73]
C Lawrence Zitnick. 2013. Handwriting beautification using token means. ACM Transactions on Graphics (TOG) 32, 4 (2013), 1--8.

Index Terms

  1. Enhancing the Aesthetics of 3D Shapes via Reference-based Editing

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 43, Issue 6
      December 2024
      1828 pages
      EISSN:1557-7368
      DOI:10.1145/3702969
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 November 2024
      Published in TOG Volume 43, Issue 6

      Check for updates

      Author Tags

      1. perception
      2. shape aesthetics
      3. 3d shape beautification
      4. reference-guided editing

      Qualifiers

      • Research-article

      Funding Sources

      • Chow Sang Sang Group Research Fund/Donation
      • Research Grants Council of Hong Kong, China
      • Research Grants Council, Hong Kong, China

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 89
        Total Downloads
      • Downloads (Last 12 months)89
      • Downloads (Last 6 weeks)13
      Reflects downloads up to 13 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media