Portrait Shadow Removal via Self-Exemplar Illumination Equalization
Authors: 
Qian Huang, Cheng Xu, Guiqing Li, Ziheng Wu, Shengxin Liu, Shengfeng HeAuthors Info & Claims
Qian Huang, Cheng Xu, Guiqing Li, Ziheng Wu, Shengxin Liu, Shengfeng HeAuthors Info & ClaimsPages 7474 - 7482
Abstract
We introduce the Self-Exemplar Illumination Equalization Network, designed specifically for effective portrait shadow removal. The core idea of our method is that partially shadowed portraits can find ideal exemplars within their non-shadowed facial regions. Rather than directly fusing two distinct classes of facial features, our approach utilizes non-shadowed regions as an illumination indicator to equalize the shadowed regions, generating deshadowed results without boundary-merging artifacts. Our network comprises cascaded Self-Exemplar Illumination Equalization Blocks (SExmBlock), each containing two modules: a self-exemplar feature matching module and a feature-level illumination rectification module. The former identifies and applies internal illumination exemplars to shadowed areas, producing illumination-corrected features, while the latter adjusts shadow illumination by reapplying the illumination factors from these features to the input face. Applying this series of SExmBlocks to shadowed portraits incrementally eliminates shadows and preserves clear, accurate facial details. The effectiveness of our method is demonstrated through evaluations on two public shadow portrait datasets, where it surpasses existing state-of-the-art methods in both qualitative and quantitative assessments.
References
[1]
Eli Arbel and Hagit Hel-Or. 2007. Texture-preserving shadow removal in color images containing curved surfaces. In CVPR. IEEE, 1--8.
[2]
Eli Arbel and Hagit Hel-Or. 2010. Shadow removal using intensity surfaces and texture anchor points. PAMI (2010), 1202--1216.
[3]
Ronen Basri and David W Jacobs. 2003. Lambertian reflectance and linear subspaces. PAMI, Vol. 25, 2 (2003), 218--233.
[4]
Peter N Belhumeur and David J Kriegman. 1998. What is the set of images of an object under all possible illumination conditions? IJCV, Vol. 28 (1998), 245--260.
[5]
Xiaodong Cun, Chi-Man Pun, and Cheng Shi. 2020. Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN. In AAAI, Vol. 34. 10680--10687.
[6]
N. Dalal and B. Triggs. 2005. Histograms of oriented gradients for human detection. In CVPR, Vol. 1. 886--893.
[7]
Bin Ding, Chengjiang Long, Ling Zhang, and et al. 2019. Argan: Attentive recurrent generative adversarial network for shadow detection and removal. In ICCV. 10213--10222.
[8]
Graham D Finlayson, Mark S Drew, and Cheng Lu. 2009. Entropy minimization for shadow removal. IJCV, Vol. 85, 1 (2009), 35--57.
[9]
Clement Fredembach and Graham Finlayson. 2006. Simple shadow remova. In ICPR, Vol. 1. IEEE, 832--835.
[10]
Ruiqi Guo, Qieyun Dai, and Derek Hoiem. 2012. Paired regions for shadow detection and removal. PAMI, Vol. 35, 12 (2012), 2956--2967.
[11]
Xiaoguang Han, Hao Yang, Guoqiang Xing, and et al. 2019. Asymmetric joint GANs for normalizing face illumination from a single image. TMM, Vol. 22, 6 (2019), 1619--1633.
[12]
Y He, Y Xing, T Zhang, and et al. 2021. Unsupervised portrait shadow removal via generative priors. In ACMMM. 236--244.
[13]
Jun-Wei Hsieh, Wen-Fong Hu, Chia-Jung Chang, and Yung-Sheng Chen. 2003. Shadow elimination for effective moving object detection by Gaussian shadow modeling. IVC, Vol. 21, 6 (2003), 505--516.
[14]
C H Hu, J Yu, F Wu, and et al. 2021. Face illumination recovery for the deep learning feature under severe illumination variations. PR, Vol. 111 (2021), 107724.
[15]
Xiaowei Hu, Chi-Wing Fu, Lei Zhu, Jing Qin, and Pheng-Ann Heng. 2019. Direction-aware spatial context features for shadow detection and removal. PAMI, Vol. 42, 11 (2019), 2795--2808.
[16]
Xiaowei Hu, Yitong Jiang, Chi-Wing Fu, and Pheng-Ann Heng. 2019. Mask-ShadowGAN: Learning to remove shadows from unpaired data. In ICCV. 2472--2481.
[17]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In ECCV. Springer, 694--711.
[18]
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv (2017).
[19]
Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In CVPR. 4401--4410.
[20]
Salman H Khan, Mohammed Bennamoun, Ferdous Sohel, and Roberto Togneri. 2015. Automatic shadow detection and removal from a single image. PAMI, Vol. 38, 3 (2015), 431--446.
[21]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv (2014).
[22]
Edwin H Land. 1977. The retinex theory of color vision. Scientific American, Vol. 237, 6 (1977), 108--129.
[23]
Edwin H Land and John J McCann. 1971. Lightness and retinex theory. JOSA, Vol. 61, 1 (1971), 1--11.
[24]
Hieu Le and Dimitris Samaras. 2019. Shadow removal via shadow image decomposition. In ICCV. 8578--8587.
[25]
P H Lee, S W Wu, and Y P Hung. 2012. Illumination compensation using oriented local histogram equalization and its application to face recognition. TIP, Vol. 21, 9 (2012), 4280--4289.
[26]
Yan Li, Peng Gong, and Tadashi Sasagawa. 2005. Integrated shadow removal based on photogrammetry and image analysis. International Journal of Remote Sensing, Vol. 26, 18 (2005), 3911--3929.
[27]
Jiawei Liu, Qiang Wang, Huijie Fan, Wentao Li, Liangqiong Qu, and Yandong Tang. 2023. A decoupled multi-task network for shadow removal. IEEE TMM (2023).
[28]
Jiawei Liu, Qiang Wang, Huijie Fan, Jiandong Tian, and Yandong Tang. 2023. A shadow imaging bilinear model and three-branch residual network for shadow removal. IEEE TNNLS (2023).
[29]
Yaojie Liu, Andrew Hou, Xinyu Huang, Liu Ren, and Xiaoming Liu. 2022. Blind Removal of Facial Foreign Shadows. In BMVC. London, UK.
[30]
Zhihao Liu, Hui Yin, Xinyi Wu, and et al. 2021. From shadow generation to shadow removal. In CVPR. 4927--4936.
[31]
David G Lowe. 2004. Distinctive image features from scale-invariant keypoints. IJCV, Vol. 60 (2004), 91--110.
[32]
L. Lu, W. Li, X. Tao, and et al. 2021. Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution. In CVPR. 6368--6377.
[33]
Yifang Men, Yiming Mao, Yuning Jiang, Wei-Ying Ma, and Zhouhui Lian. 2020. Controllable person image synthesis with attribute-decomposed gan. In CVPR. 5084--5093.
[34]
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic image synthesis with spatially-adaptive normalization. In CVPR. 2337--2346.
[35]
Liangqiong Qu, Jiandong Tian, Shengfeng He, and et al. 2017. Deshadownet: A multi-context embedding deep network for shadow removal. In CVPR. 4067--4075.
[36]
Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer graphics and applications, Vol. 21, 5 (2001), 34--41.
[37]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. IJCV, Vol. 115 (2015), 211--252.
[38]
Neda Salamati, Arthur Germain, and S Siisstrunk. 2011. Removing shadows from images using color and near-infrared. In ICIP. IEEE, 1713--1716.
[39]
Zhe Shu, Ersin Yumer, Sunil Hadap, and et al. 2017. Neural face editing with intrinsic image disentangling. In CVPR. 5541--5550.
[40]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv (2014).
[41]
Jifeng Wang, Xiang Li, and Jian Yang. 2018. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In CVPR. 1788--1797.
[42]
Y. Wei, M. Liu, H. Wang, and et al. 2020. Learning flow-based feature warping for face frontalization with illumination inconsistent supervision. In ECCV. Springer International Publishing, 558--574.
[43]
Tai-Pang Wu and Chi-Keung Tang. 2005. A bayesian approach for shadow extraction from a single image. In ICCV, Vol. 1. IEEE, 480--487.
[44]
Cheng Xu, Zejun Chen, Jiajie Mai, Xuemiao Xu, and Shengfeng He. 2023. Pose- and attribute-consistent person image synthesis. IEEE TOMM (2023), 1--21.
[45]
Cheng Xu, Keke Li, Xuandi Luo, Xuemiao Xu, Shengfeng He, and Kun Zhang. 2022. Fully deformable network for multiview face image synthesis. IEEE TNNLS (2022), 8854--8868.
[46]
Cheng Xu, Wei Qu, Xuemiao Xu, and Xueting Liu. 2022. Multi-scale flow-based occluding effect and content separation for cartoon animations. IEEE TVCG (2022), 4001--4014.
[47]
Yangyang Xu, Xuemiao Xu, Jianbo Jiao, Keke Li, Cheng Xu, and Shengfeng He. 2021. Multi-view face synthesis via progressive face flow. TIP, Vol. 30 (2021), 6024--6035.
[48]
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. 2018. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In ECCV. 325--341.
[49]
Ling Zhang, Qing Zhang, and Chunxia Xiao. 2015. Shadow remover: Image shadow removal based on illumination recovering optimization. TIP, Vol. 24, 11 (2015), 4623--4636.
[50]
P. Zhang, B. Zhang, D. Chen, and et al. 2020. Cross-domain correspondence learning for exemplar-based image translation. In CVPR. 5143--5153.
[51]
Xuaner Zhang, Jonathan T Barron, Yun-Ta Tsai, Rohit Pandey, Xiuming Zhang, Ren Ng, and David E Jacobs. 2020. Portrait shadow manipulation. TOG, Vol. 39, 4 (2020), 78--1.
[52]
T. Zhou, S. Tulsiani, W. Sun, J. Malik, and A. A. Efros. 2016. View synthesis by appearance flow. In ECCV. 286--301.
[53]
Yurui Zhu, Jie Huang, Xueyang Fu, and et al. 2022. Bijective mapping network for shadow removal. In CVPR. 5627--5636.
Index Terms
- Portrait Shadow Removal via Self-Exemplar Illumination Equalization
Recommendations
Generative Portrait Shadow Removal
We introduce a high-fidelity portrait shadow removal model that can effectively enhance the image of a portrait by predicting its appearance under disturbing shadows and highlights. Portrait shadow removal is a highly ill-posed problem where multiple ...
Portrait shadow manipulation
Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. Aesthetic qualities such as the position and softness of shadows and the lighting ratio between the bright and ...
Unsupervised Portrait Shadow Removal via Generative Priors
MM '21: Proceedings of the 29th ACM International Conference on MultimediaPortrait images often suffer from undesirable shadows cast by casual objects or even the face itself. While existing methods for portrait shadow removal require training on a large-scale synthetic dataset, we propose the first unsupervised method for ...
Comments
Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Guangdong Natural Science Funds for Distinguished Young Scholar
- Guangdong Basic and Applied Basic Research Foundation
- National Research Foundation Singapore under the AI Singapore Programme
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 172Total Downloads
- Downloads (Last 12 months)172
- Downloads (Last 6 weeks)83
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in