Qinjie Xiao
ABSTRACT
We present deep shapely portraits, a novel method based on deep learning, to automatically reshape an input portrait to be better proportioned and more shapely while keeping personal facial characteristics. Different from existing methods that may suffer from irrational face artifacts when dealing with portraits with large pose variations or reshaping adjustments, we utilize dense 3D face information and constraints instead of sparse facial landmarks based on 3D morphable models, resulting in better reshaped faces lying in rational face space. To this end, we first estimate the best shapely degree for the input portrait using a convolutional neural network (CNN) trained on our newly developed ShapeFaceNet dataset. Then the best shapely degree is used as the control parameter to reshape the 3D face reconstructed from the input portrait image. After that, we render the reshaped 3D face back to 2D and generate a seamless portrait image using a fast image warping optimization. Our work can deal with pose and expression free (PE-Free) portrait images and generate plausible shapely faces without noticeable artifacts, which cannot be achieved by prior work. We validate the effectiveness, efficiency, and robustness of the proposed method by extensive experiments and user studies.
Supplemental Material
Available for Download
zip
mmfp0925aux.zip (48.8 MB)
"Demo.mp4" is the demo of the paper in the "mp4" format. "Supplementary_Material.pdf" contains several supplementary results, the detailed description of our warping optimization, the details of the SHAPEFACENET Dataset, etc.
- Jascha Achenbach, Eduard Zell, and Mario Botsch. 2015. Accurate Face Reconstruction through Anisotropic Fitting and Eye Correction. In Vision, Modeling & Visualization, David Bommes, Tobias Ritschel, and Thomas Schultz (Eds.). The Eurographics Association. https://doi.org/10.2312/vmv.20151251Google Scholar
- Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), Vol. 99. 187--194. https://doi.org/10.1145/311535.311556Google ScholarDigital Library
- James Booth, Anastasios Roussos, Allan Ponniah, David Dunaway, and Stefanos Zafeiriou. 2018. Large Scale 3D Morphable Models. International Journal of Computer Vision, Vol. 126, 2--4 (2018), 233--254. https://doi.org/10.1007/s11263-017--1009--7Google ScholarDigital Library
- Chen Cao, Derek Bradley, Kun Zhou, and Thabo Beeler. 2015. Real-Time High-Fidelity Facial Performance Capture. ACM Transactions on Graphics, Vol. 34, 4, Article 46 (July 2015), 9 pages. https://doi.org/10.1145/2766943Google ScholarDigital Library
- Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2014. FaceWarehouse: A 3D Facial Expression Database for Visual Computing. IEEE Transactions on Visualization and Computer Graphics, Vol. 20, 3 (March 2014), 413--425. https://doi.org/10.1109/TVCG.2013.249Google ScholarDigital Library
- Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou. 2015. High-Quality Hair Modeling from a Single Portrait Photo. ACM Transactions on Graphics, Vol. 34, 6, Article 204 (Oct. 2015), 10 pages. https://doi.org/10.1145/2816795.2818112Google ScholarDigital Library
- Menglei Chai, Tianjia Shao, Hongzhi Wu, Yanlin Weng, and Kun Zhou. 2016. AutoHair: Fully Automatic Hair Modeling from a Single Image. ACM Transactions on Graphics, Vol. 35, 4, Article 116 (July 2016), 12 pages. https://doi.org/10.1145/2897824.2925961Google ScholarDigital Library
- Che-Han Chang, Yoichi Sato, and Yung-Yu Chuang. 2014. Shape-Preserving Half-Projective Warps for Image Stitching. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3254--3261.Google Scholar
- Fangmei Chen, Xihua Xiao, and David Zhang. 2018. Data-Driven Facial Beauty Analysis: Prediction, Retrieval and Manipulation. IEEE Transactions on Affective Computing, Vol. 9, 2 (2018), 205--216. https://doi.org/10.1109/TAFFC.2016.2599534Google ScholarCross Ref
- Yu-Sheng Chen and Yung-Yu Chuang. 2016. Natural Image Stitching with the Global Similarity Prior. In European Conference on Computer Vision. Springer, 186--201. https://doi.org/10.1007/978--3--319--46454--1_12Google Scholar
- Sven De Greef, Peter Claes, Dirk Vandermeulen, Wouter Mollemans, Paul Suetens, and Guy Willems. 2006. Large-Scale In-Vivo Caucasian Facial Soft Tissue Thickness Database for Craniofacial Reconstruction. Forensic science international, Vol. 159 (2006), S126--S146. https://doi.org/10.1016/j.forsciint.2006.02.034Google Scholar
- Facetune2. 2020. Facetune2 - Best Seflie Editor App. www.facetuneapp.com.Google Scholar
- Yangyu Fan, Shu Liu, Bo Li, Zhe Guo, Ashok Samal, Jun Wan, and Stan Z. Li. 2018. Label Distribution-Based Facial Attractiveness Computation by Deep Residual Learning. IEEE Transactions on Multimedia, Vol. 20, 8 (2018), 2196--2208. https://doi.org/10.1109/TMM.2017.2780762Google ScholarCross Ref
- Junying Gan, Lichen Li, Yikui Zhai, and Yinhua Liu. 2014. Deep Self-Taught Learning for Facial Beauty Prediction. Neurocomputing, Vol. 144 (2014), 295--303. https://doi.org/10.1016/j.neucom.2014.05.028Google ScholarDigital Library
- Lian Gao, Weixin Li, Zehua Huang, and Di Huang. 2018. Automatic Facial Attractiveness Prediction by Deep Multi-Task Learning. In International Conference on Pattern Recognition (ICPR). 3592--3597. https://doi.org/10.1109/ICPR.2018.8545033Google ScholarCross Ref
- Xiaoguang Han, Kangcheng Hou, Dong Du, Yuda Qiu, Shuguang Cui, Kun Zhou, and Yizhou Yu. 2018. CaricatureShop: Personalized and Photorealistic Caricature Sketching. IEEE Transactions on Visualization and Computer Graphics (2018), 1--1. https://doi.org/10.1109/TVCG.2018.2886007Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.Google ScholarCross Ref
- Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Jens Fursund, Iman Sadeghi, Carrie Sun, Yen-Chun Chen, and Hao Li. 2017. Avatar Digitization from a Single Image for Real-Time Rendering. ACM Transactions on Graphics, Vol. 36, 6, Article 195 (Nov. 2017), 14 pages. https://doi.org/10.1145/3130800.31310887Google ScholarDigital Library
- Patrik Huber, Guosheng Hu, Rafael Tena, Pouria Mortazavian, P Koppen, William J Christmas, Matthias Ratsch, and Josef Kittler. 2016. A Multiresolution 3D Morphable Face Model and Fitting Framework. In Proceedings of the 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. INSTICC, 79--86. https://doi.org/10.5220/0005669500790086Google ScholarCross Ref
- Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In International Conference on Learning Representations.Google Scholar
- Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8110--8119.Google ScholarCross Ref
- Vahid Kazemi and Josephine Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1867--1874.Google ScholarDigital Library
- Hyeongwoo Kim, Pablo Carrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. 2018. Deep Video Portraits. ACM Transactions on Graphics, Vol. 37, 4, Article 163 (2018), 14 pages. https://doi.org/10.1145/3197517.3201283Google ScholarDigital Library
- Tommer Leyvand, Daniel Cohen-Or, Gideon Dror, and Dani Lischinski. 2008. Data-Driven Enhancement of Facial Attractiveness. ACM Transactions on Graphics, Vol. 27, 3, Article 38 (Aug. 2008), 9 pages. https://doi.org/10.1145/1360612.1360637Google ScholarDigital Library
- Honglin Li, Masahiro Toyoura, and Xiaoyang Mao. 2019. Caricature Synthesis with Feature Deviation Matching under Example-Based Framework. The Visual Computer, Vol. 35, 5 (2019), 653--666. https://doi.org/10.1007/s00371-018--1495--9Google ScholarDigital Library
- Tianye Li, Timo Bolkart, Michael J. Black, Hao Li, and Javier Romero. 2017. Learning a Model of Facial Shape and Expression from 4D Scans. ACM Transactions on Graphics, Vol. 36, 6, Article 194 (Nov. 2017), 17 pages. https://doi.org/10.1145/3130800.3130813Google ScholarDigital Library
- Qiqi Liao, Xiaogang Jin, and Wenting Zeng. 2012. Enhancing the Symmetry and Proportion of 3D Face Geometry. IEEE Transactions on Visualization and Computer Graphics, Vol. 18, 10 (2012), 1704--1716. https://doi.org/10.1109/TVCG.2012.26Google ScholarDigital Library
- Si Liu, Xinyu Ou, Ruihe Qian, Wei Wang, and Xiaochun Cao. 2016. Makeup Like a Superstar: Deep Localized Makeup Transfer Network. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI'16). AAAI Press, 2568--2575.Google Scholar
- Xudong Liu, Tao Li, Hao Peng, Iris Chuoying Ouyang, Taehwan Kim, and Ruizhe Wang. 2019. Understanding Beauty via Deep Facial Features. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.Google ScholarCross Ref
- Yilong Liu, Chengwei Zheng, Feng Xu, Xin Tong, and Baining Guo. 2020. Data-Driven 3D Neck Modeling and Animation. IEEE Transactions on Visualization and Computer Graphics (2020). https://doi.org/10.1109/TVCG.2020.2967036Google Scholar
- Meitu. 2020. Meitu -- Beauty Cam, Easy Photo Editor. https://play.google.com/store/apps/details?id=com.mt.mtxx.mtxx&hl=en.Google Scholar
- Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning Detailed Face Reconstruction from a Single Image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1259--1268.Google ScholarCross Ref
- Rasmus Rothe, Radu Timofte, and Luc Van Gool. 2016. Some Like It Hot - Visual Guidance for Preference Prediction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5553--5561.Google Scholar
- Omry Sendik, Dani Lischinski, and Daniel Cohen-Or. 2019. What's in a Face? Metric Learning for Face Characterization. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 405--416. https://doi.org/10.1111/cgf.13647Google Scholar
- YiChang Shih, Wei-Sheng Lai, and Chia-Kai Liang. 2019. Distortion-Free Wide-Angle Portraits on Camera Phones. ACM Transactions on Graphics, Vol. 38, 4, Article 61 (2019), 12 pages. https://doi.org/10.1145/3306346.3322948Google ScholarDigital Library
- YiChang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014. Style Transfer for Headshot Portraits. ACM Transactions on Graphics, Vol. 33, 4, Article 148 (July 2014), 14 pages. https://doi.org/10.1145/2601097.2601137Google ScholarDigital Library
- Zhixin Shu, Sunil Hadap, Eli Shechtman, Kalyan Sunkavalli, Sylvain Paris, and Dimitris Samaras. 2017. Portrait Lighting Transfer Using a Mass Transport Approach. ACM Transactions on Graphics, Vol. 37, 1, Article 2 (Oct. 2017), 15 pages. https://doi.org/10.1145/3095816Google Scholar
- Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. 2019. Single Image Portrait Relighting. ACM Transactions on Graphics, Vol. 38, 4, Article 79 (July 2019), 12 pages. https://doi.org/10.1145/3306346.3323008Google ScholarDigital Library
- Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. 2019. FML: Face Model Learning from Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10812--10822.Google ScholarCross Ref
- Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni. 2017. Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5163--5172.Google ScholarCross Ref
- Duorui Xie, Lingyu Liang, Lianwen Jin, Jie Xu, and Mengru Li. 2015. SCUT-FBP: A Benchmark Dataset for Facial Beauty Perception. In 2015 IEEE International Conference on Systems, Man, and Cybernetics. 1821--1826. https://doi.org/10.1109/SMC.2015.319Google Scholar
- Jie Xu, Lianwen Jin, Lingyu Liang, Ziyong Feng, Duorui Xie, and Huiyun Mao. 2017. Facial Attractiveness Prediction Using Psychologically Inspired Convolutional Neural Network (PI-CNN). In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 1657--1661. https://doi.org/10.1109/ICASSP.2017.7952438Google ScholarCross Ref
- Haiming Zhao, Xiaogang Jin, Xiaojian Huang, Menglei Chai, and Kun Zhou. 2018. Parametric Reshaping of Portrait Images for Weight-Change. IEEE Computer Graphics and Applications, Vol. 38, 1 (2018), 77--90. https://doi.org/10.1109/MCG.2018.011461529Google ScholarCross Ref
- Shizhe Zhou, Hongbo Fu, Ligang Liu, Daniel Cohen-Or, and Xiaoguang Han. 2010. Parametric Reshaping of Human Bodies in Images. ACM Transactions on Graphics, Vol. 29, 4, Article 126 (2010), 10 pages. https://doi.org/10.1145/1778765.1778863Google ScholarDigital Library
Index Terms
- Deep Shapely Portraits
Recommendations
Parametric Reshaping of Portraits in Videos
MM '21: Proceedings of the 29th ACM International Conference on MultimediaSharing short personalized videos to various social media networks has become quite popular in recent years. This raises the need for digital retouching of portraits in videos. However, applying portrait image editing directly on portrait video frames ...
Deep face recognition using imperfect facial data
AbstractToday, computer based face recognition is a mature and reliable mechanism which is being practically utilised for many access control scenarios. As such, face recognition or authentication is predominantly performed using ‘perfect’ ...
Highlights- We show the performance of machine learning for face recognition using partial faces and other manipulations of the face such as rotation and zooming which ...
Deep Learning Identity-Preserving Face Space
ICCV '13: Proceedings of the 2013 IEEE International Conference on Computer VisionFace recognition with large pose and illumination variations is a challenging problem in computer vision. This paper addresses this challenge by proposing a new learning based face representation: the face identity-preserving (FIP) features. Unlike ...
Comments