Ling Zhang
Skip Abstract Section
Abstract
Residual image and illumination estimation have been proven to be helpful for image enhancement. In this article, we propose a general framework, called RI-GAN, that exploits residual and illumination using generative adversarial networks (GANs). The proposed framework detects and removes shadows in a coarse-to-fine fashion. At the coarse stage, we employ three generators to produce a coarse shadow-removal result, a residual image, and an inverse illumination map. We also incorporate two indirect shadow-removal images via the residual image and the inverse illumination map. With the residual image, the illumination map, and the two indirect shadow-removal images as auxiliary information, the refinement stage estimates a shadow mask to identify shadow regions in the image, and then refines the coarse shadow-removal result to the fine shadow-free image. We introduce a cross-encoding module to the refinement generator, in which the use of feature-crossing can provide additional details to promote the shadow mask and the high-quality shadow-removal result. In addition, we apply data augmentation to the discriminator to reduce the dependence between representations of the discriminator and the quality of the predicted image. Experiments for shadow detection and shadow removal demonstrate that our method outperforms state-of-the-art methods. Furthermore, RI-GAN exhibits good performance in terms of image dehazing, rain removal, and highlight removal, demonstrating the effectiveness and flexibility of the proposed framework.
- [1] Y. Akashi and T. Okatani. 2016. Separation of reflection components by sparse non-negative matrix factorization. Computer Vision and Image Understanding 146 (2016), 77–85.Google Scholar
Digital Library
- [2] . 2021. Robust representation learning with feedback for single image deraining. In Conference on Computer Vision and Pattern Recognition (CVPR’21).Google ScholarCross Ref
- [3] . 2019. Self-supervised GANs via auxiliary rotation loss. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 12146–12155.Google ScholarCross Ref
- [4] . 2021. CANet: A context-aware network for shadow removal. In IEEE International Conference on Computer Vision (ICCV’21). 4743–4752.Google ScholarCross Ref
- [5] . 2020. A multi-task mean teacher for semi-supervised shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’20).Google ScholarCross Ref
- [6] . 2002. Improving shadow suppression in moving object detection with HSV color information. In IEEE Intelligent Transportation Systems. 334–339.Google Scholar
- [7] . 2020. Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting GAN. In AAAI. 10680–10687.Google Scholar
- [8] . 2020. Fast deep multi-patch hierarchical network for nonhomogeneous image dehazing. In Conference on Computer Vision and Pattern Recognition (CVPR’20).Google Scholar
- [9] . 2019. ARGAN: Attentive recurrent generative adversarial network for shadow detection and removal.. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 10212–10221.Google ScholarCross Ref
- [10] . 2008. Texture-consistent shadow removal. In European Conference on Computer Vision (ECCV’08). 437–450.Google Scholar
- [11] . 2005. On the removal of shadows from images. T-PAMI (2005).Google Scholar
- [12] . 2021. Auto-exposure fusion for single-image shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’21).Google ScholarCross Ref
- [13] . 2016. A weighted variational model for simultaneous reflectance and illumination estimation. In Conference on Computer Vision and Pattern Recognition (CVPR’16). 2782–2790.Google ScholarCross Ref
- [14] . 2021. A multi-task network for joint specular highlight detection and removal. In Conference on Computer Vision and Pattern Recognition (CVPR’21).Google Scholar
- [15] . 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS’14).Google Scholar
- [16] . 2015. Learning to Remove Soft Shadows, Vol. 34. 1–15.Google ScholarDigital Library
- [17] . 2011. Single-image shadow detection and removal using paired regions. In Conference on Computer Vision and Pattern Recognition (CVPR’11). 2033–2040.Google ScholarDigital Library
- [18] . 2017. LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing 26, 2 (2017), 982–993.Google ScholarDigital Library
- [19] . 2011. Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 12 (2011), 2341–2353.Google ScholarDigital Library
- [20] . 2018. Densely connected pyramid dehazing network. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 3194–3203.Google Scholar
- [21] . 2020. Direction-aware spatial context features for shadow detection and removal. 42, 11 (2020), 2795–2808.Google Scholar
- [22] . 2018. Collaborative active visual recognition from crowds: A distributed ensemble approach. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 3 (2018), 582–594.Google ScholarCross Ref
- [23] . 2016. Automatic shadow detection and removal from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 3 (2016), 431–446.Google ScholarDigital Library
- [24] . 2018. A+ D net: Training a shadow detector with adversarial shadow attenuation. In European Conference on Computer Vision (ECCV’18). 680–696.Google ScholarDigital Library
- [25] . 2017. AOD-Net: All-in-one dehazing network. In IEEE International Conference on Computer Vision (ICCV’17). 4780–4788.Google ScholarCross Ref
- [26] . 2008. Texture-consistent shadow removal. In European Conference on Computer Vision (ECCV’08). (2008).Google Scholar
- [27] . 2019. GridDehazeNet: Attention-based multi-scale network for image dehazing. In International Conference on Computer Vision (ICCV’19). 7313–7322.Google ScholarCross Ref
- [28] . 2015. Multi-class multi-annotator active learning with robust Gaussian process for visual recognition. In International Conference on Computer Vision (ICCV’15). 2839–2847.Google ScholarDigital Library
- [29] . 2017. Correlational gaussian processes for cross-domain visual recognition. In Conference on Computer Vision and Pattern Recognition (CVPR”17). 4932–4940.Google ScholarCross Ref
- [30] . 2014. Accurate object detection with location relaxation and regionlets re-localization. In Asia Conference on Computer Vision.Google Scholar
- [31] . 2020. End-to-end active object tracking and its real-world deployment via reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 6 (2020), 1317–1332.Google ScholarCross Ref
- [32] . 2000. Moving shadow and object detection in traffic scenes. In Conference on International Conference on Pattern Recognition.Google Scholar
- [33] . 2018. Spectral normalization for generative adversarial networks. International Conference on Learning Representations (2018).Google Scholar
- [34] . 2017. Shadow detection with conditional generative adversarial networks. In International Conference on Computer Vision (ICCV’19).Google ScholarCross Ref
- [35] . 2020. Cumulative rain density sensing network for single image derain. IEEE Signal Processing Letters 27 (2020), 406–410.Google ScholarCross Ref
- [36] . 2017. DeshadowNet: A multi-context embedding deep network for shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’17). 2308–2316.Google ScholarCross Ref
- [37] . 2018. Single image haze removal using a generative adversarial network. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 37–42.Google Scholar
- [38] . 2020. Single image deraining using bilateral recurrent network. IEEE Transactions on Image Processing 29, 99 (2020), 6852–6863.Google ScholarCross Ref
- [39] H. Shen and Z. Zheng. 2013. Real-time highlight removal using intensity ratio. Applied Optics 52, 19 (2013), 4483–4493.Google ScholarCross Ref
- [40] . 2008. The shadow meets the mask: Pyramid-based shadow removal. Computer Graphics Forum 27, 2 (2008), 577–586.Google ScholarCross Ref
- [41] . 2019. Conditional GANs for multi-illuminant color constancy: Revolution or yet another approach? In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’19). 1748–1758.Google ScholarCross Ref
- [42] . 2016. Large-scale training of shadow detectors with noisily-annotated shadow examples. In Conference on European Conference on Computer Vision (ECCV’16).Google Scholar
- [43] . 2020. A model-driven deep neural network for single image rain removal. In Conference on Computer Vision and Pattern Recognition (CVPR’20).Google Scholar
- [44] . 2018. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In Conference on Computer Vision and Pattern Recognition (CVPR’18). 1788–1797.Google ScholarCross Ref
- [45] . 2019. Underexposed photo enhancement using deep illumination estimation. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 6842–6850.Google ScholarCross Ref
- [46] . 2020. Instance shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’20). 1877–1886.Google ScholarCross Ref
- [47] . 2019. Shadow inpainting and removal using generative adversarial networks with slice convolutions. Computer Graphics Forum 38, 7 (2019), 381–392.Google ScholarCross Ref
- [48] . 2007. Natural shadow matting. ACM Transactions on Graphics 26, 2 (2007), 8.Google ScholarDigital Library
- [49] . 2021. Single UHD image dehazing via interpretable pyramid network. In Conference on Computer Vision and Pattern Recognition (CVPR’21).Google Scholar
- [50] . 2013. Fast shadow removal using adaptive multi-scale illumination transfer. Computer Graphics Forum 32, 8 (2013), 207–218.Google ScholarCross Ref
- [51] . 2013. Efficient shadow removal using subregion matching illumination transfer. Computer Graphics Forum 32, 7 (2013), 421–430.Google ScholarCross Ref
- [52] . 2019. General improvement method of specular component separation using high-emphasis filter and similarity function. ITE Transactions on Media Technology Applications 7, 2 (2019), 92–102.Google ScholarCross Ref
- [53] . 2015. Efficient and robust specular highlight removal. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 6 (2015), 1304–1311.Google ScholarDigital Library
- [54] . 2020. BEDSR-net: A deep shadow removal network from a single document image. In Conference on Computer Vision and Pattern Recognition (CVPR’20) (2020), 12902–12911.Google Scholar
- [55] . 2020. RIS-GAN: Explore residual and illumination with generative adversarial networks for shadow removal. In Proceedings of the AAAI Conference on Artificial Intelligence (2020), 12829–12836.Google ScholarCross Ref
- [56] . 2019. Effective shadow removal via multi-scale image decomposition. The Visual Computer 35, 6–8 (2019), 1091–1104.Google ScholarDigital Library
- [57] . 2015. Shadow remover: Image shadow removal based on illumination recovering optimization. IEEE Transactions on Image Processing 24, 11 (2015), 4623–4636.Google ScholarDigital Library
- [58] . 2019. Distraction-aware shadow detection. In Conference on Computer Vision and Pattern Recognition (CVPR’19). 5162–5171.Google ScholarCross Ref
- [59] . 2018. Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In European Conference on Computer Vision (ECCV’18). 122–137.Google ScholarDigital Library
- [60] . 2021. Mitigating intensity bias in shadow detection via feature decomposition and reweighting. In International Conference on Computer Vision (ICCV’21). 4682–4691.Google ScholarCross Ref
Index Terms
- Exploiting Residual and Illumination with GANs for Shadow Detection and Shadow Removal
Recommendations
Simple shadow removal using shadow depth map and illumination-invariant feature
AbstractShadows included in images provide useful information for visual scene analysis, but are also factors that negatively affect digital image analysis. Therefore, shadow detection and removal must be considered essential in the preprocessing of the ...
Shadow Remover: Image Shadow Removal Based on Illumination Recovering Optimization
In this paper, we present a novel shadow removal system for single natural images as well as color aerial images using an illumination recovering optimization method. We first adaptively decompose the input image into overlapped patches according to the ...
Shadow Removal Using Intensity Surfaces and Texture Anchor Points
Removal of shadows from a single image is a challenging problem. Producing a high-quality shadow-free image which is indistinguishable from a reproduction of a true shadow-free scene is even more difficult. Shadows in images are typically affected by ...
Comments