skip to main content
research-article
Open access

MichiGAN: multi-input-conditioned hair image generation for portrait editing

Published: 12 August 2020 Publication History

Abstract

Despite the recent success of face image generation with GANs, conditional hair editing remains challenging due to the under-explored complexity of its geometry and appearance. In this paper, we present MichiGAN (Multi-Input-Conditioned Hair Image GAN), a novel conditional image generation method for interactive portrait hair manipulation. To provide user control over every major hair visual factor, we explicitly disentangle hair into four orthogonal attributes, including shape, structure, appearance, and background. For each of them, we design a corresponding condition module to represent, process, and convert user inputs, and modulate the image generation pipeline in ways that respect the natures of different visual attributes. All these condition modules are integrated with the backbone generator to form the final end-to-end network, which allows fully-conditioned hair generation from multiple user inputs. Upon it, we also build an interactive portrait hair editing system that enables straightforward manipulation of hair by projecting intuitive and high-level user inputs such as painted masks, guiding strokes, or reference photos to well-defined condition representations. Through extensive experiments and evaluations, we demonstrate the superiority of our method regarding both result quality and user controllability.

Supplemental Material

MP4 File
Presentation video
Transcript for: Presentation video
MP4 File
ZIP File
Code for the paper "MichiGAN: multi-input-conditioned hair image generation for portrait editing" presented in SIGGRAPH 2020 and published in ACM Transactions on Graphics (TOG). The code is also available via GitHub: https://github.com/tzt101/MichiGAN

References

[1]
Rameen Abdal, Yipeng Qin, and Peter Wonka. 2019. Image2StyleGAN: How to Embed Images into the StyleGAN Latent Space?. In ICCV 2019. 4431--4440.
[2]
David Bau, Hendrik Strobelt, William S. Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. 2019. Semantic Photo Manipulation with A Generative Image Prior. ACM Trans. Graph. 38, 4 (2019), 59:1--59:11.
[3]
Dmitri Bitouk, Neeraj Kumar, Samreen Dhillon, Peter N. Belhumeur, and Shree K. Nayar. 2008. Face Swapping: Automatically Replacing Faces in Photographs. ACM Trans. Graph. 27, 3 (2008), 39.
[4]
Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. In SIGGRAPH 1999. 187--194.
[5]
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2019. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In ICLR 2019.
[6]
Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2014. FaceWarehouse: A 3D Facial Expression Database for Visual Computing. IEEE Trans. Vis. Comput. Graph. 20, 3 (2014), 413--425.
[7]
Menglei Chai, Linjie Luo, Kalyan Sunkavalli, Nathan Carr, Sunil Hadap, and Kun Zhou. 2015. High-Quality Hair Modeling from A Single Portrait Photo. ACM Trans. Graph. 34, 6 (2015), 204:1--204:10.
[8]
Menglei Chai, Tianjia Shao, Hongzhi Wu, Yanlin Weng, and Kun Zhou. 2016. AutoHair: Fully Automatic Hair Modeling from A Single Image. ACM Trans. Graph. 35, 4 (2016), 116:1--116:12.
[9]
Menglei Chai, Lvdi Wang, Yanlin Weng, Xiaogang Jin, and Kun Zhou. 2013. Dynamic Hair Manipulation in Images and Videos. ACM Trans. Graph. 32, 4 (2013), 75:1--75:8.
[10]
Menglei Chai, Lvdi Wang, Yanlin Weng, Yizhou Yu, Baining Guo, and Kun Zhou. 2012. Single-View Hair Modeling for Portrait Manipulation. ACM Trans. Graph. 31, 4 (2012), 116:1--116:8.
[11]
Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. 2017a. Coherent Online Video Style Transfer. In ICCV 2017. 1114--1123.
[12]
Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2017b. StyleBank: An Explicit Representation for Neural Image Style Transfer. In CVPR 2017. 2770--2779.
[13]
Kevin Dale, Kalyan Sunkavalli, Micah K. Johnson, Daniel Vlasic, Wojciech Matusik, and Hanspeter Pfister. 2011. Video Face Replacement. ACM Trans. Graph. 30, 6 (2011), 130.
[14]
Carl Doersch. 2016. Tutorial on Variational Autoencoders. CoRR abs/1606.05908 (2016).
[15]
Ohad Fried, Eli Shechtman, Dan B. Goldman, and Adam Finkelstein. 2016. Perspective-Aware Manipulation of Portrait Photos. ACM Trans. Graph. 35, 4 (2016), 128:1--128:10.
[16]
Zhenglin Geng, Chen Cao, and Sergey Tulyakov. 2019. 3D Guided Fine-Grained Face Manipulation. In CVPR 2019. 9821--9830.
[17]
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NeurIPS 2014. 2672--2680.
[18]
Mingming He, Dongdong Chen, Jing Liao, Pedro V. Sander, and Lu Yuan. 2018. Deep Exemplar-Based Colorization. ACM Trans. Graph. 37, 4 (2018), 47:1--47:16.
[19]
Seunghoon Hong, Xinchen Yan, Thomas S. Huang, and Honglak Lee. 2018. Learning Hierarchical Semantic Image Manipulation through Structured Representations. In NeurIPS 2018. 2713--2723.
[20]
Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2014. Robust Hair Capture Using Simulated Examples. ACM Trans. Graph. 33, 4 (2014), 126:1--126:10.
[21]
Liwen Hu, Chongyang Ma, Linjie Luo, and Hao Li. 2015. Single-View Hair Modeling Using A Hairstyle Database. ACM Trans. Graph. 34, 4 (2015), 125:1--125:9.
[22]
Xun Huang and Serge J. Belongie. 2017. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. In ICCV 2017. 1510--1519.
[23]
Xun Huang, Ming-Yu Liu, Serge J. Belongie, and Jan Kautz. 2018. Multimodal Unsupervised Image-to-Image Translation. In ECCV 2018, Vol. 11207. 179--196.
[24]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In CVPR 2017. 5967--5976.
[25]
Youngjoo Jo and Jongyoul Park. 2019. SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color. In ICCV 2019. 1745--1753.
[26]
Neel Joshi, Wojciech Matusik, Edward H. Adelson, and David J. Kriegman. 2010. Personal Photo Enhancement Using Example Images. ACM Trans. Graph. 29, 2 (2010), 12:1--12:15.
[27]
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In ICLR 2018.
[28]
Tero Karras, Samuli Laine, and Timo Aila. 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. In CVPR 2019. 4401--4410.
[29]
Ira Kemelmacher-Shlizerman. 2016. Transfiguring Portraits. ACM Trans. Graph. 35, 4 (2016), 94:1--94:8.
[30]
Ira Kemelmacher-Shlizerman, Supasorn Suwajanakorn, and Steven M. Seitz. 2014. Illumination-Aware Age Progression. In CVPR 2014. 3334--3341.
[31]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR 2015.
[32]
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. 2019. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. CoRR abs/1907.11922 (2019).
[33]
Tommer Leyvand, Daniel Cohen-Or, Gideon Dror, and Dani Lischinski. 2008. Data-Driven Enhancement of Facial Attractiveness. ACM Trans. Graph. 27, 3 (2008), 38.
[34]
Shu Liang, Xiufeng Huang, Xianyu Meng, Kunyao Chen, Linda G. Shapiro, and Ira Kemelmacher-Shlizerman. 2018. Video to Fully Automatic 3D Hair Model. ACM Trans. Graph. 37, 6 (2018), 206:1--206:14.
[35]
Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017. Visual Attribute Transfer through Deep Image Analogy. ACM Trans. Graph. 36, 4 (2017), 120:1--120:15.
[36]
Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image Inpainting for Irregular Holes Using Partial Convolutions. In ECCV 2018, Vol. 11215. 89--105.
[37]
Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. 2019. Few-Shot Unsupervised Image-to-Image Translation. In ICCV 2019. 10550--10559.
[38]
Linjie Luo, Hao Li, and Szymon Rusinkiewicz. 2013. Structure-Aware Hair Capture. ACM Trans. Graph. 32, 4 (2013), 76:1--76:12.
[39]
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. 2017. Least Squares Generative Adversarial Networks. In ICCV 2017. 2813--2821.
[40]
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. 2017. Unrolled Generative Adversarial Networks. In ICLR 2017.
[41]
Mehdi Mirza and Simon Osindero. 2014. Conditional Generative Adversarial Nets. CoRR abs/1411.1784 (2014).
[42]
Koki Nagano, Huiwen Luo, Zejian Wang, Jaewoo Seo, Jun Xing, Liwen Hu, Lingyu Wei, and Hao Li. 2019. Deep Face Normalization. ACM Trans. Graph. 38, 6 (2019), 183:1--183:16.
[43]
Augustus Odena, Christopher Olah, and Jonathon Shlens. 2017. Conditional Image Synthesis with Auxiliary Classifier GANs. In ICML 2017, Vol. 70. 2642--2651.
[44]
Sylvain Paris, Will Chang, Oleg I. Kozhushnyan, Wojciech Jarosz, Wojciech Matusik, Matthias Zwicker, and Frédo Durand. 2008. Hair Photobooth: Geometric and Photometric Acquisition of Real Hairstyles. ACM Trans. Graph. 27, 3 (2008), 30.
[45]
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic Image Synthesis with Spatially-Adaptive Normalization. In CVPR 2019. 2337--2346.
[46]
Haonan Qiu, Chuan Wang, Hang Zhu, Xiao Zhu, Jinjin Gu, and Xiaoguang Han. 2019. Two-Phase Hair Image Synthesis by Self-Enhancing Generative Model. Comput. Graph. Forum 38, 7 (2019), 403--412.
[47]
Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative Adversarial Text to Image Synthesis. In ICML 2016, Vol. 48. 1060--1069.
[48]
Shunsuke Saito, Liwen Hu, Chongyang Ma, Hikaru Ibayashi, Linjie Luo, and Hao Li. 2018. 3D Hair Synthesis Using Volumetric Variational Autoencoders. ACM Trans. Graph. 37, 6 (2018), 208:1--208:12.
[49]
Zhixin Shu, Ersin Yumer, Sunil Hadap, Kalyan Sunkavalli, Eli Shechtman, and Dimitris Samaras. 2017. Neural Face Editing with Intrinsic Image Disentangling. In CVPR 2017. 5444--5453.
[50]
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In ICLR 2015.
[51]
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. 2019. FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery. In CVPR 2019. 6490--6499.
[52]
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. 2016. Ladder Variational Autoencoders. In NeurIPS 2016. 3738--3746.
[53]
Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E. Debevec, and Ravi Ramamoorthi. 2019. Single Image Portrait Relighting. ACM Trans. Graph. 38, 4 (2019), 79:1--79:12.
[54]
Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. In CVPR 2016. 2387--2395.
[55]
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. In CVPR 2018. 8798--8807.
[56]
Yang Wang, Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, and Dimitris Samaras. 2009. Face Relighting from A Single Image under Arbitrary Unknown Lighting Conditions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 11 (2009), 1968--1984.
[57]
Lingyu Wei, Liwen Hu, Vladimir G. Kim, Ersin Yumer, and Hao Li. 2018. Real-Time Hair Rendering Using Sequential Adversarial Networks. In ECCV 2018. 105--122.
[58]
Yanlin Weng, Lvdi Wang, Xiao Li, Menglei Chai, and Kun Zhou. 2013. Hair Interpolation for Portrait Morphing. Comput. Graph. Forum 32, 7 (2013), 79--84.
[59]
Fei Yang, Jue Wang, Eli Shechtman, Lubomir D. Bourdev, and Dimitris N. Metaxas. 2011. Expression Flow for 3D-Aware Face Component Transfer. ACM Trans. Graph. 30, 4 (2011), 60.
[60]
Shunyu Yao, Tzu-Ming Harry Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum. 2018. 3D-Aware Scene Manipulation via Inverse Graphics. In NeurIPS 2018. 1891--1902.
[61]
Guojun Yin, Bin Liu, Lu Sheng, Nenghai Yu, Xiaogang Wang, and Jing Shao. 2019. Semantics Disentangling for Text-to-Image Generation. In CVPR 2019. 2327--2336.
[62]
Han Zhang, Tao Xu, and Hongsheng Li. 2017b. StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks. In ICCV 2017. 5908--5916.
[63]
Meng Zhang, Menglei Chai, Hongzhi Wu, Hao Yang, and Kun Zhou. 2017a. A Data-Driven Approach to Four-View Image-Based Hair Modeling. ACM Trans. Graph. 36, 4 (2017), 156:1--156:11.
[64]
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. 2019. On the Continuity of Rotation Representations in Neural Networks. In CVPR 2019. 5745--5753.
[65]
Yi Zhou, Liwen Hu, Jun Xing, Weikai Chen, Han-Wei Kung, Xin Tong, and Hao Li. 2018. HairNet: Single-View Hair Reconstruction Using Convolutional Neural Networks. In ECCV 2018. 249--265.
[66]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In ICCV 2017. 2242--2251.

Cited By

View all
  • (2024)Synthetic Image Generation Using Deep Learning: A Systematic Literature ReviewComputational Intelligence10.1111/coin.7000240:5Online publication date: 21-Oct-2024
  • (2024)ETBHD‐HMF: A Hierarchical Multimodal Fusion Architecture for Enhanced Text‐Based Hair DesignComputer Graphics Forum10.1111/cgf.1519443:6Online publication date: 3-Sep-2024
  • (2024)Text2Face: Text-Based Face Generation With Geometry and Appearance ControlIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.334905030:9(6481-6492)Online publication date: 2-Jan-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 39, Issue 4
August 2020
1732 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3386569
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 August 2020
Published in TOG Volume 39, Issue 4

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. conditional hair image generation
  2. generative adversarial networks
  3. interactive portrait editing

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)150
  • Downloads (Last 6 weeks)25
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Synthetic Image Generation Using Deep Learning: A Systematic Literature ReviewComputational Intelligence10.1111/coin.7000240:5Online publication date: 21-Oct-2024
  • (2024)ETBHD‐HMF: A Hierarchical Multimodal Fusion Architecture for Enhanced Text‐Based Hair DesignComputer Graphics Forum10.1111/cgf.1519443:6Online publication date: 3-Sep-2024
  • (2024)Text2Face: Text-Based Face Generation With Geometry and Appearance ControlIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.334905030:9(6481-6492)Online publication date: 2-Jan-2024
  • (2024)MixSyn: Compositional Image Synthesis with Fuzzy Masks and Style Fusion2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00741(7460-7469)Online publication date: 17-Jun-2024
  • (2024)Diverse Semantic Image Synthesis with various conditioning modalitiesKnowledge-Based Systems10.1016/j.knosys.2024.112727(112727)Online publication date: Nov-2024
  • (2024)DeepFaceReshaping: Interactive deep face reshaping via landmark manipulationComputational Visual Media10.1007/s41095-023-0373-110:5(949-963)Online publication date: 7-Oct-2024
  • (2024)ManiCLIP: Multi-attribute Face Manipulation from TextInternational Journal of Computer Vision10.1007/s11263-024-02088-6132:10(4616-4632)Online publication date: 21-May-2024
  • (2024)Controllable image synthesis methods, applications and challenges: a comprehensive surveyArtificial Intelligence Review10.1007/s10462-024-10987-w57:12Online publication date: 18-Oct-2024
  • (2024)GAN based augmentation using a hybrid loss function for dermoscopy imagesArtificial Intelligence Review10.1007/s10462-024-10897-x57:9Online publication date: 7-Aug-2024
  • (2024)Personalized hairstyle and hair color editing based on multi-feature fusionThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-024-03468-240:7(4751-4763)Online publication date: 29-May-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media