skip to main content
10.1145/3132787.3132789acmconferencesArticle/Chapter ViewAbstractPublication Pagessiggraph-asiaConference Proceedingsconference-collections
abstract

Real-time face relighting via adaptive normal mapping

Published: 27 November 2017 Publication History

Abstract

In this paper, we present a real-time face relighting method for enhancing widely used face photographs such as `selfies'. In contrast with previous relighting methods that estimate and replace the real lighting condition with the intended one, our technique focuses on the aesthetic aspects of relighting with real-time interaction. We estimate the head pose based on the facial features and apply adaptive face normal mapping by transforming and interpolating pre-defined face normal maps. The lighting effect is calculated using adaptively mapped normals and overlaid on the original photograph. The experimental results show that the proposed method generates aesthetic face photographs without a computational overhead, enabling real-time face relighting for mobile applications.

References

[1]
David Akers, Frank Losasso, Jeff Klingner, Maneesh Agrawala, John Rick, and Pat Hanrahan. 2003. Conveying Shape and Features with Image-Based Relighting. In Proc. IEEE Visualization 2003 (VIS'03). 349--354.
[2]
Md. Akhtaruzzaman and Amir A. Shafie. 2011. Geometrical Substantiation of Phi, the Golden Ratio and the Baroque of Nature, Architecture, Design and Engineering. Int. Journal of Arts 1, 1 (2011), 1--22.
[3]
Amr Almaddah, Sadi Vural, Yasushi Mae, Kenichi Ohara, and Tatsuo Arai. 2012. 2D Spherical Spaces for Face Relighting under Harsh Illumination. Int. Journal of Computer, Electrical, Automation, Control and Information Engineering 6, 9 (2012), 1138--1143.
[4]
Amr Almaddah, Sadi Vural, Yasushi Mae, Kenichi Ohara, and Tatsuo Arai. 2014. Face Relighting using Discriminative 2D Spherical Spaces for Face Recognition. Machine Vision and Applications 25, 4 (2014), 845--857.
[5]
N K Bansode and P K Sinha. 2016. Face Shape Classification Based on Region Similarity, Correlation and Fractal Dimensions. Int. Journal of Computer Science Issues 13, 1 (2016), 24--31.
[6]
Ronen Basri and David W. Jacobs. 2003. Lambertian Reflectance and Linear Subspaces. IEEE Trans. Pattern Analysis and Machine Intelligence 25, 2 (2003), 218--233.
[7]
Sai Bi, Xiaoguang Han, and Yizhou Yu. 2015. An L1 Image Transform for Edge-Preserving Smoothing and Scene-Level Intrinsic Decomposition. ACM Trans. Graphics (Proc. SIGGRAPH) 34, 4 (2015), Article No. 78.
[8]
Volker Blanz and Thomas Vetter. 1999. A Morphable Model for the Synthesis of 3D Faces. In Proc. Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '99). 187--194.
[9]
James F. Blinn. 1978. Simulation of Wrinkled Surfaces. In Proc. Conf. Computer Graphics and Interactive Techniques. 286--292.
[10]
A. Buades, B. Coll, and J.-M. Morel. 2005. A Non-Local Algorithm for Image Denoising. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR). 60--65.
[11]
James Burnstone and Hujun Yin. 2011. Eigenlights: Recovering Illumination from Face Images. Lecture Notes in Computer Science (LNCS, Proc International Conference on Intelligent Data Engineering and Automated Learning) 6936 (2011), 490--497.
[12]
Chia-Ping Chen and Chu-Song Chen. 2005. Lighting Normalization with Generic Intrinsic Illumination Subspace for Face Recognition. In Proc. IEEE International Conference on Computer Vision (ICCV). 1089--1096.
[13]
Jiawen Chen, Sylvain Paris, and Frédo Durand. 2007. Real-Time Edge-Aware Image Processing with the Bilateral Grid. ACM Trans. Graphics (Proc. SIGGRAPH) 26, 3 (2007), Article No. 103.
[14]
Jiansheng Chen, Guangda Su, Jinping He, and Shenglan Ben. 2010. Face Image Relighting using Locally Constrained Global Optimization. In Proc. European Conf. Computer Vision 2010. 44--57.
[15]
Peter Comninos. 2011. Mathematical and Computer Programming Techniques for Computer Graphics. Springer.
[16]
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2009. BM3D Image Denoising with Shape-Adaptive Principal Component Analysis. In Proc. Workshop on Signal Processing with Adaptive Sparse Structured Representations (SPARS). 1--6.
[17]
Paul Debevec, Andreas Wenger, Chris Tchou, Andrew Gardner, Jamie Waese, and Tim Hawkins. 2002. A Lighting Reproduction Approach to Live-Action Compositing. ACM Trans. Graphics (Proc. SIGGRAPH) 21, 3 (2002), 547--556.
[18]
Frédo Durand and Julie Dorsey. 2002. Fast Bilateral Filtering for the Display of High-Dynamic-Range Images. ACM Trans. Graphics (Proc. SIGGRAPH) 21, 3 (2002), 257--266.
[19]
Gee-Sern Hsu and Ding-Yu Lin. 2013. Face Recognition using Sparse Representation with Illumination Normalization and Component Features. In Proc. Signal and Information Processing Association Annual Summit and Conference (APSIPA). 1--5.
[20]
Rein-Lien Hsu, Mohamed Abdel-Mottaleb, and Anil K. Jain. 2002. Face Detection in Color Images. IEEE Trans. Pattern Analysis and Machine Intelligence 24, 5 (2002), 696--706.
[21]
Takuya Ikeda, Francois de Sorbier, and Hideo Saito. 2013. Arbitrarily Shaped Objects Relighting Using an RGB-D Camera. In Proc. Asian Conference on Pattern Recognition (ACPR). 631--636.
[22]
Seungyong Lee, George Wolberg, and Sung Yong Shin. 1997. Scattered Data Interpolation with Multilevel B-splines. IEEE Trans. Visualization and Computer Graphics 3, 3 (1997), 228--244.
[23]
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep Photo Style Transfer. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR). 4990--4998.
[24]
Claus B. Madsen and RuneLaursen. 2005. A Model-Based Approach to Image Relighting with a Potential for Real-Time Implementation. In Proc. Vision, Video, and Graphics 2005. 237--243.
[25]
Yohei Ogura, Takuya Ikeda, Francois de Sorbier, and Hideo Saito. 2015. Illumination Estimation and Relighting using an RGB-D Camera. In Proc. International Conference on Computer Vision Theory and Applications (VISAPP). 306--312.
[26]
Sylvain Paris, Samuel W. Hasinoff, and Jan Kautz. 2011. Local Laplacian Filters: Edge-aware Image Processing with a Laplacian Pyramid. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4 (2011), Article No. 68.
[27]
Sylvain Paris, François X. Sillion, and Long Quan. 2003. Lightweight Face Relighting. In Proc. 11th Pacific Conference on Computer Graphics and Applications (PG '03). 41--50.
[28]
Vishal M. Patel, Tao Wu, Soma Biswas, P. Jonathon Phillips, and Rama Chellappa. 2011. Illumination Robust Dictionary-Based Face Recognition. In IEEE International Conference on Image Processing (ICIP). 777--780.
[29]
Venkataraman Santhanam, Vlad I. Morariu, and Larry S. Davis. 2017. Generalized Deep Image to Image Regression. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR). -.
[30]
Hyunjung Shim, Jiebo Luo, and Tsuhan Chen. 2008. A Subspace Model-Based Approach to Face Relighting Under Unknown Lighting and Poses. IEEE Trans. Image Processing 17, 8 (2008), 1331--1341.
[31]
Minjung Son, Yunjin Lee, Henry Kang, and Seungyong Lee. 2011. Structure Grid for Directional Stippling. Graphical Models 73, 3 (2011), 74--87.
[32]
Minjung Son, Yunjin Lee, Henry Kang, and Seungyong Lee. 2014. Art-Photographic Detail Enhancement. Computer Graphics Forum (Special Issue on Eurographics 2014) 33, 2 (2014), 391--400.
[33]
J.A. Stark. 2000. Adaptive Image Contrast Enhancement using Generalizations of Histogram Equalization. IEEE Trans. Image Processing 9, 5 (2000), 889--896.
[34]
Richard Szeliski. 2011. Computer Vision: Algorithms and Applications. Springer.
[35]
Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey Hinton. 2012. Deep Lambertian Networks. In Proc. International Conference on Machine Learing (ICML). 1623--1630.
[36]
G. Toderici, G. Passalis, S. Zafeiriou, G. Tzimiropoulos, M. Petrou, T. Theoharis, and I. A. Kakadiaris. 2010. Bidirectional Relighting for 3D-Aided 2D Face Recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR). 2721--2728.
[37]
Baoyuan Wang, Yizhou Yu, and Ying-Qing Xu. 2011. Example-based Image Color and Tone Style Enhancement. ACM Trans. Graphics (Proc. SIGGRAPH) 30, 4 (2011), Article No. 64.
[38]
Yang Wang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, and Dimitris Samaras. 2007. Face Re-Lighting from a Single Image under Harsh Lighting Conditions. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR). 1--8.
[39]
Yang Wang, Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, and Dimitris Samaras. 2009. Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions. IEEE Trans. Pattern Analysis and Machine Intelligence 31, 11 (2009), 1968--1984.
[40]
Phillip Ian Wilson and John Fernandez. 2006. Facial Feature Detection using Haar Classifiers. Journal of Computing Sciences in Colleges 21, 4 (2006), 127--133.
[41]
Li Xu, Cewu Lu, Yi Xu, and Jiaya Jia. 2011. Image Smoothing via L0 Gradient Minimization. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 30, 6 (2011), Article No. 174.

Cited By

View all
  • (2025) Application of Phi ( Φ ), the Golden Ratio, in Computing: A Systematic Review IEEE Access10.1109/ACCESS.2024.352220613(4621-4651)Online publication date: 2025
  • (2024)Non-Uniform Low-Light Face Image Enhancement Based on Dark Channel Prior and Image Uniform PosteriorIEEE Access10.1109/ACCESS.2024.341613012(85724-85734)Online publication date: 2024
  • (2022)Lighting-and Personal Characteristic-Aware Markov Random Field Model for Facial Image Relighting SystemIEEE Access10.1109/ACCESS.2022.315034110(20432-20444)Online publication date: 2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SA '17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications
November 2017
116 pages
ISBN:9781450354103
DOI:10.1145/3132787
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 November 2017

Check for updates

Author Tags

  1. face relighting
  2. image and video enhancement
  3. image and video processing
  4. image-based relighting
  5. real-time mobile application

Qualifiers

  • Abstract

Funding Sources

  • Samsung Electronics Co. Ltd.

Conference

SA '17
Sponsor:
SA '17: SIGGRAPH Asia 2017
November 27 - 30, 2017
Bangkok, Thailand

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)6
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025) Application of Phi ( Φ ), the Golden Ratio, in Computing: A Systematic Review IEEE Access10.1109/ACCESS.2024.352220613(4621-4651)Online publication date: 2025
  • (2024)Non-Uniform Low-Light Face Image Enhancement Based on Dark Channel Prior and Image Uniform PosteriorIEEE Access10.1109/ACCESS.2024.341613012(85724-85734)Online publication date: 2024
  • (2022)Lighting-and Personal Characteristic-Aware Markov Random Field Model for Facial Image Relighting SystemIEEE Access10.1109/ACCESS.2022.315034110(20432-20444)Online publication date: 2022
  • (2022)QRCP-based preprocessing for illumination invariant measure under severe illumination variationsSignal, Image and Video Processing10.1007/s11760-022-02283-317:3(753-760)Online publication date: 25-Jun-2022
  • (2021)RelightGAN: Instance-level Generative Adversarial Network for Face Illumination TransferIEEE Transactions on Image Processing10.1109/TIP.2021.306193330(3450-3460)Online publication date: 1-Jan-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media