research-article

Exploiting Residual and Illumination with GANs for Shadow Detection and Shadow Removal

Authors:

Chengjiang Long,

Xiaolong Zhang,

Chunxia XiaoAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications and Applications, Volume 19, Issue 3

Article No.: 120, Pages 1 - 22

https://doi.org/10.1145/3571745

Published: 25 February 2023 Publication History

Abstract

Residual image and illumination estimation have been proven to be helpful for image enhancement. In this article, we propose a general framework, called RI-GAN, that exploits residual and illumination using generative adversarial networks (GANs). The proposed framework detects and removes shadows in a coarse-to-fine fashion. At the coarse stage, we employ three generators to produce a coarse shadow-removal result, a residual image, and an inverse illumination map. We also incorporate two indirect shadow-removal images via the residual image and the inverse illumination map. With the residual image, the illumination map, and the two indirect shadow-removal images as auxiliary information, the refinement stage estimates a shadow mask to identify shadow regions in the image, and then refines the coarse shadow-removal result to the fine shadow-free image. We introduce a cross-encoding module to the refinement generator, in which the use of feature-crossing can provide additional details to promote the shadow mask and the high-quality shadow-removal result. In addition, we apply data augmentation to the discriminator to reduce the dependence between representations of the discriminator and the quality of the predicted image. Experiments for shadow detection and shadow removal demonstrate that our method outperforms state-of-the-art methods. Furthermore, RI-GAN exhibits good performance in terms of image dehazing, rain removal, and highlight removal, demonstrating the effectiveness and flexibility of the proposed framework.

References

[1]

Y. Akashi and T. Okatani. 2016. Separation of reflection components by sparse non-negative matrix factorization. Computer Vision and Image Understanding 146 (2016), 77–85.

Digital Library

[2]

C. Chen and H. Li. 2021. Robust representation learning with feedback for single image deraining. In Conference on Computer Vision and Pattern Recognition (CVPR’21).

[3]

Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. 2019. Self-supervised GANs via auxiliary rotation loss. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 12146–12155.

[4]

Zipei Chen, Chengjiang Long, Ling Zhang, and Chunxia Xiao. 2021. CANet: A context-aware network for shadow removal. In IEEE International Conference on Computer Vision (ICCV’21). 4743–4752.

[5]

Zhihao Chen, Lei Zhu, Liang Wan, Song Wang, and Pheng Ann Heng. 2020. A multi-task mean teacher for semi-supervised shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’20).

[6]

Rita Cucchiara, Costantino Grana, Massimo Piccardi, Andrea Prati, and Stefano Sirotti. 2002. Improving shadow suppression in moving object detection with HSV color information. In IEEE Intelligent Transportation Systems. 334–339.

[7]

Xiaodong Cun, Chi-Man Pun, and Cheng Shi. 2020. Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN. In AAAI. 10680–10687.

[8]

S. D. Das and S. Dutta. 2020. Fast deep multi-patch hierarchical network for nonhomogeneous image dehazing. In Conference on Computer Vision and Pattern Recognition (CVPR’20).

[9]

Bin Ding, Chengjiang Long, Ling Zhang, and Chunxia Xiao. 2019. ARGAN: Attentive recurrent generative adversarial network for shadow detection and removal. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 10212–10221.

[10]

Liu Feng and Michael Gleicher. 2008. Texture-consistent shadow removal. In European Conference on Computer Vision (ECCV’08). 437–450.

[11]

G. D. Finlayson, S. D. Hordley, C. Lu, and M. S. Drew. 2005. On the removal of shadows from images. T-PAMI (2005).

[12]

Lan Fu, Changqing Zhou, Qing Guo, Felix Juefei-Xu, Hongkai Yu, Wei Feng, Yang Liu, and Song Wang. 2021. Auto-exposure fusion for single-image shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’21).

[13]

Xueyang Fu, Delu Zeng, Yue Huang, Xiao-Ping Zhang, and Xinghao Ding. 2016. A weighted variational model for simultaneous reflectance and illumination estimation. In Conference on Computer Vision and Pattern Recognition (CVPR’16). 2782–2790.

[14]

Fu Gang, Zhang Qing, Zhu Lei, Li Ping, and Chunxia Xiao. 2021. A multi-task network for joint specular highlight detection and removal. In Conference on Computer Vision and Pattern Recognition (CVPR’21).

[15]

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, X. Bing, and Y. Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS’14).

[16]

Maciej Gryka, Michael Terry, and Gabriel J. Brostow. 2015. Learning to Remove Soft Shadows, Vol. 34. 1–15.

Digital Library

[17]

Ruiqi Guo, Qieyun Dai, and D. Hoiem. 2011. Single-image shadow detection and removal using paired regions. In Conference on Computer Vision and Pattern Recognition (CVPR’11). 2033–2040.

Digital Library

[18]

Xiaojie Guo, Yu Li, and Haibin Ling. 2017. LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing 26, 2 (2017), 982–993.

Digital Library

[19]

K. He, J. Sun, and Xiaoou Tang. 2011. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 12 (2011), 2341–2353.

Digital Library

[20]

Zhang He and Vishal M. Patel. 2018. Densely connected pyramid dehazing network. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 3194–3203.

[21]

X. Hu, C. W. Fu, Z. Lei, Q. Jing, and P. A. Heng. 2020. Direction-aware spatial context features for shadow detection and removal. 42, 11 (2020), 2795–2808.

[22]

Gang Hua, Chengjiang Long, Ming Yang, and Yan Gao. 2018. Collaborative active visual recognition from crowds: A distributed ensemble approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 3 (2018), 582–594.

[23]

S. H. Khan, M. Bennamoun, F. Sohel, and R. Togneri. 2016. Automatic shadow detection and removal from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 3 (2016), 431–446.

Digital Library

[24]

Hieu Le, Yago Vicente, F. Tomas, Vu Nguyen, Minh Hoai, and Dimitris Samaras. 2018. A+ D net: Training a shadow detector with adversarial shadow attenuation. In European Conference on Computer Vision (ECCV’18). 680–696.

Digital Library

[25]

B. Li, X. Peng, Z. Wang, J. Xu, and F. Dan. 2017. AOD-Net: All-in-one dehazing network. In IEEE International Conference on Computer Vision (ICCV’17). 4780–4788.

[26]

Feng Liu and Michael Gleicher. 2008. Texture-consistent shadow removal. In European Conference on Computer Vision (ECCV’08). (2008).

[27]

Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. 2019. GridDehazeNet: Attention-based multi-scale network for image dehazing. In International Conference on Computer Vision (ICCV’19). 7313–7322.

[28]

Chengjiang Long and Gang Hua. 2015. Multi-class multi-annotator active learning with robust Gaussian process for visual recognition. In International Conference on Computer Vision (ICCV’15). 2839–2847.

Digital Library

[29]

Chengjiang Long and Gang Hua. 2017. Correlational gaussian processes for cross-domain visual recognition. In Conference on Computer Vision and Pattern Recognition (CVPR”17). 4932–4940.

[30]

Chengjiang Long, Xiaoyu Wang, Gang Hua, Ming Yang, and Yuanqing Lin. 2014. Accurate object detection with location relaxation and regionlets re-localization. In Asia Conference on Computer Vision.

[31]

Wenhan Luo, Peng Sun, Fangwei Zhong, Wei Liu, Tong Zhang, and Yizhou Wang. 2020. End-to-end active object tracking and its real-world deployment via reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 6 (2020), 1317–1332.

[32]

Ivana Mikic, Pamela C. Cosman, Greg Kogut, and Mohan M. Trivedi. 2000. Moving shadow and object detection in traffic scenes. In Conference on International Conference on Pattern Recognition.

[33]

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. International Conference on Learning Representations (2018).

[34]

Vu Nguyen, Tomas F. Yago Vicente, Maozheng Zhao, Minh Hoai, and Dimitris Samaras. 2017. Shadow detection with conditional generative adversarial networks. In International Conference on Computer Vision (ICCV’19).

[35]

Long Peng, Aiwen Jiang, Qiaosi Yi, and Mingwen Wang. 2020. Cumulative rain density sensing network for single image derain. IEEE Signal Processing Letters 27 (2020), 406–410.

[36]

Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson W. H. Lau. 2017. DeshadowNet: A multi-context embedding deep network for shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’17). 2308–2316.

[37]

N. Bharath Raj and N. Venkateswaran. 2018. Single image haze removal using a generative adversarial network. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 37–42.

[38]

Dongwei Ren, Wei Shang, Pengfei Zhu, Qinghua Hu, and Wangmeng Zuo. 2020. Single image deraining using bilateral recurrent network. IEEE Transactions on Image Processing 29, 99 (2020), 6852–6863.

[39]

H. Shen and Z. Zheng. 2013. Real-time highlight removal using intensity ratio. Applied Optics 52, 19 (2013), 4483–4493.

[40]

Yael Shor and Dani Lischinski. 2008. The shadow meets the mask: Pyramid-based shadow removal. Computer Graphics Forum 27, 2 (2008), 577–586.

[41]

Oleksii Sidorov. 2019. Conditional GANs for multi-illuminant color constancy: Revolution or yet another approach? In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’19). 1748–1758.

[42]

Tomas F. Yago Vicente, Le Hou, Chenping Yu, Minh Hoai, and Dimitris Samaras. 2016. Large-scale training of shadow detectors with noisily-annotated shadow examples. In Conference on European Conference on Computer Vision (ECCV’16).

[43]

H. Wang, Q. Xie, Q. Zhao, and D. Meng. 2020. A model-driven deep neural network for single image rain removal. In Conference on Computer Vision and Pattern Recognition (CVPR’20).

[44]

Jifeng Wang, Xiang Li, Le Hui, and Jian Yang. 2018. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 1788–1797.

[45]

Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. 2019. Underexposed photo enhancement using deep illumination estimation. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 6842–6850.

[46]

Tianyu Wang, Xiaowei Hu, Qiong Wang, Pheng Ann Heng, and Chi Wing Fu. 2020. Instance shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’20). 1877–1886.

[47]

Jingjiang Wei, Chengjiang Long, Hua Zhou, and Chunxia Xiao. 2019. Shadow inpainting and removal using generative adversarial networks with slice convolutions. Computer Graphics Forum 38, 7 (2019), 381–392.

[48]

Tai Pang Wu, Chi Keung Tang, Michael S. Brown, and Heung Yeung Shum. 2007. Natural shadow matting. ACM Transactions on Graphics 26, 2 (2007), 8.

Digital Library

[49]

B. Xiao, Z. Zheng, X. Chen, C. Lv, Y. Zhuang, and T. Wang. 2021. Single UHD image dehazing via interpretable pyramid network. In Conference on Computer Vision and Pattern Recognition (CVPR’21).

[50]

Chunxia Xiao, Ruiyun She, Donglin Xiao, and Kwan Liu Ma. 2013. Fast shadow removal using adaptive multi-scale illumination transfer. Computer Graphics Forum 32, 8 (2013), 207–218.

[51]

Chunxia Xiao, Donglin Xiao, Ling Zhang, and Lin Chen. 2013. Efficient shadow removal using subregion matching illumination transfer. Computer Graphics Forum 32, 7 (2013), 421–430.

[52]

T. Yamamoto and A. Nakazawa. 2019. General improvement method of specular component separation using high-emphasis filter and similarity function. ITE Transactions on Media Technology Applications 7, 2 (2019), 92–102.

[53]

Q. Yang, J. Tang, and N. Ahuja. 2015. Efficient and robust specular highlight removal. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 6 (2015), 1304–1311.

Digital Library

[54]

Lin Yun-Hsuan, Chen Wen-Chin, and Chuang Yung-Yu. 2020. BEDSR-net: A deep shadow removal network from a single document image. In Conference on Computer Vision and Pattern Recognition (CVPR’20) (2020), 12902–12911.

[55]

Ling Zhang, Chengjiang Long, Xiaolong Zhang, and Chunxia Xiao. 2020. RIS-GAN: Explore residual and illumination with generative adversarial networks for shadow removal. In Proceedings of the AAAI Conference on Artificial Intelligence (2020), 12829–12836.

[56]

Ling Zhang, Qingan Yan, Yao Zhu, Xiaolong Zhang, and Chunxia Xiao. 2019. Effective shadow removal via multi-scale image decomposition. The Visual Computer 35, 6–8 (2019), 1091–1104.

Digital Library

[57]

Ling Zhang, Qing Zhang, and Chunxia Xiao. 2015. Shadow remover: Image shadow removal based on illumination recovering optimization. IEEE Transactions on Image Processing 24, 11 (2015), 4623–4636.

Digital Library

[58]

Quanlong Zheng, Xiaotian Qiao, Ying Cao, and Rynson W. H. Lau. 2019. Distraction-aware shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 5162–5171.

[59]

Lei Zhu, Zijun Deng, Xiaowei Hu, Chi-Wing Fu, Xuemiao Xu, Jing Qin, and Pheng-Ann Heng. 2018. Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In European Conference on Computer Vision (ECCV’18). 122–137.

Digital Library

[60]

Lei Zhu, Ke Xu, Zhanghan Ke, and Rynson W. H. Lau. 2021. Mitigating intensity bias in shadow detection via feature decomposition and reweighting. In International Conference on Computer Vision (ICCV’21). 4682–4691.

Cited By

Tan MChen QHuang ZWu QLi YZhou J(2025)Auto-3D-house Design from Structured User RequirementsMachine Intelligence Research10.1007/s11633-024-1498-0Online publication date: 7-Jan-2025
https://doi.org/10.1007/s11633-024-1498-0
Wang YLiu SLi LZhou WLi H(2024)SwinShadow: Shifted Window for Ambiguous Adjacent Shadow DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3688803Online publication date: 27-Aug-2024
https://doi.org/10.1145/3688803
Bowen DHaiquan WYuxuan LZhao JMa YRunhe H(2024)Fair and Robust Federated Learning via Decentralized and Adaptive Aggregation based on BlockchainACM Transactions on Sensor Networks10.1145/3673656Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3673656
Show More Cited By

Index Terms

Exploiting Residual and Illumination with GANs for Shadow Detection and Shadow Removal
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

Simple shadow removal using shadow depth map and illumination-invariant feature
Abstract
Shadows included in images provide useful information for visual scene analysis, but are also factors that negatively affect digital image analysis. Therefore, shadow detection and removal must be considered essential in the preprocessing of the ...
Shadow Remover: Image Shadow Removal Based on Illumination Recovering Optimization
In this paper, we present a novel shadow removal system for single natural images as well as color aerial images using an illumination recovering optimization method. We first adaptively decompose the input image into overlapped patches according to the ...
Shadow Removal Using Intensity Surfaces and Texture Anchor Points

Removal of shadows from a single image is a challenging problem. Producing a high-quality shadow-free image which is indistinguishable from a reproduction of a true shadow-free scene is even more difficult. Shadows in images are typically affected by ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 19, Issue 3

May 2023

514 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3582886

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 February 2023

Online AM: 17 November 2022

Accepted: 06 November 2022

Revised: 18 July 2022

Received: 04 December 2021

Published in TOMM Volume 19, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSFC

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
449
Total Downloads

Downloads (Last 12 months)108
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tan MChen QHuang ZWu QLi YZhou J(2025)Auto-3D-house Design from Structured User RequirementsMachine Intelligence Research10.1007/s11633-024-1498-0Online publication date: 7-Jan-2025
https://doi.org/10.1007/s11633-024-1498-0
Wang YLiu SLi LZhou WLi H(2024)SwinShadow: Shifted Window for Ambiguous Adjacent Shadow DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3688803Online publication date: 27-Aug-2024
https://doi.org/10.1145/3688803
Bowen DHaiquan WYuxuan LZhao JMa YRunhe H(2024)Fair and Robust Federated Learning via Decentralized and Adaptive Aggregation based on BlockchainACM Transactions on Sensor Networks10.1145/3673656Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3673656
Ma JZhang FJin BSu CLi SWang ZNi J(2024)Push the Limit of Highly Accurate Ranging on Commercial UWB DevicesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596028:2(1-27)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659602
Chen YKe QLi HWu YZhang Y(2024)xMeta: SSD-HDD-hybrid Optimization for Metadata Maintenance of Cloud-scale Object StorageACM Transactions on Architecture and Code Optimization10.1145/365260621:2(1-20)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3652606
Wu HWang ZLi YLiu XLee T(2024)Suitable and Style-Consistent Multi-Texture Recommendation for Cartoon IllustrationsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365251820:7(1-26)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1145/3652518
Chen LLi WCui XWang ZBerretti SWan S(2024)MS-GDA: Improving Heterogeneous Recipe Representation via Multinomial Sampling Graph Data AugmentationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364862020:7(1-23)Online publication date: 25-Apr-2024
https://dl.acm.org/doi/10.1145/3648620
Ding XHuang PZhang DLiang WLi FYang GLiao XLi Y(2024)MSEConv: A Unified Warping Framework for Video Frame InterpolationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/3648364Online publication date: 14-Feb-2024
https://dl.acm.org/doi/10.1145/3648364
Zhang ZSun WWu HZhou YLi CChen ZMin XZhai GLin W(2024)GMS-3DQA: Projection-Based Grid Mini-patch Sampling for 3D Model Quality AssessmentACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364381720:6(1-19)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3643817
Ma YZhao CHuang BLi XBasu A(2024)RAST: Restorable Arbitrary Style TransferACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363877020:5(1-21)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3638770
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents