Abstract
Deep learning-based shadow removal methods are frequently hard to obtain a detail-rich and boundary-smoothing shadow removal result. In this work, we propose an illumination-sensitive filter and a multi-task generative adversarial networks architecture to tackle these problems. Firstly, we detect the shadow for the input shadow image and use the illumination-sensitive filter to extract the texture information for generating a coarse image with fewer texture details. Secondly, we conduct illumination estimation for this coarse shadow image to remove the shadow indirectly. Next, we restore the shadow boundary realistically inspired by the idea of image in painting. Finally, we recover the texture details for obtaining the final shadow removal result. Besides, we filter two large benchmark datasets, i.e., SRD and ISTD, to create a Low Error Synthesized Dataset (LESD). The extensive experiments demonstrate that our method can achieve superior performance to state of the arts.










Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Arbel, E., Hel-Or, H.: Shadow removal using intensity surfaces and texture anchor points. IEEE Trans. Pattern Anal. Mach. Intell. 33(6), 1202–1216 (2010)
Raj, N. B., N, V.: Single image haze removal using a generative adversarial network. ArXiv Preprint arXiv:1810.09479 (2018)
Cun, X., Pun, C.M., Shi, C.: Towards ghost-free shadow removal via dual hierarchical aggregation network and shadow matting gan. In: AAAI, pp. 10680–10687 (2020)
Ding, B., Long, C., Zhang, L., Xiao, C.: Argan: attentive recurrent generative adversarial network for shadow detection and removal. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10212–10221 (2019)
Fu, X., Zeng, D., Huang, Y., Zhang, X.P., Ding, X.: A weighted variational model for simultaneous reflectance and illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2782–2790 (2016)
Fu, X., Zeng, D., Huang, Y., Zhang, X.P., Ding, X.: A weighted variational model for simultaneous reflectance and illumination estimation. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2782–2790 (2016)
Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Gong, H., Cosker, D.: Interactive shadow removal and ground truth for variable scene categories. In: BMVC, pp. 1–11 (2014)
Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: CVPR 2011, pp. 2033–2040. IEEE (2011)
Guo, X., Li, Y., Ling, H.: Lime: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2016)
Guo, X., Li, Y., Ling, H.: Lime: low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 26(2), 982–993 (2017)
Hu, X., Fu, C.W., Zhu, L., Qin, J., Heng, P.A.: Direction-aware spatial context features for shadow detection and removal. IEEE Trans. Pattern Anal. Mach. Intell. 42(11), 2795–2808 (2020)
Hu, X., Jiang, Y., Fu, C.W., Heng, P.A.: Mask-shadowgan: Learning to remove shadows from unpaired data. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2472–2481 (2019)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision, pp. 694–711. Springer (2016)
Khan, S.H., Bennamoun, M., Sohel, F., Togneri, R.: Automatic shadow detection and removal from a single image. IEEE Trans. Pattern Anal. Mach. Intell. 38(3), 431–446 (2015)
Le, H., Vicente, T.F.Y., Nguyen, V., Hoai, M., Samaras, D.: A+d net: Training a shadow detector with adversarial shadow attenuation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 680–696 (2018)
Liu, D., Long, C., Zhang, H., Yu, H., Dong, X., Xiao, C.: Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Liu, D., Long, C., Zhang, H., Yu, H., Dong, X., Xiao, C.: Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8139–8148 (2020)
Liu, F., Gleicher, M.: Texture-consistent shadow removal. In: European Conference on Computer Vision, pp. 437–450. Springer (2008)
Liu, Z., Xiang, Q., Tang, J., Wang, Y., Zhao, P.: Robust salient object detection for RGB images. Vis. Comput. 36(9), 1823–1835 (2020)
Mohajerani, S., Saeedi, P.: Cpnet: A context preserver convolutional neural network for detecting shadows in single RGB images. In: 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–5 (2018)
Mohan, A., Tumblin, J., Choudhury, P.: Editing soft shadows in a digital photograph. IEEE Comput. Graph. Appl. 27(2), 23–31 (2007)
Nguyen, V., Yago Vicente, T.F., Zhao, M., Hoai, M., Samaras, D.: Shadow detection with conditional generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4510–4518 (2017)
PNVR, K., Zhou, H., Jacobs, D.: Sharingan: Combining synthetic and real data for unsupervised geometry estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Qu, L., Tian, J., He, S., Tang, Y., Lau, R.W.: Deshadownet: A multi-context embedding deep network for shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4067–4075 (2017)
Sidorov, O.: Conditional gans for multi-illuminant color constancy: Revolution or yet another approach? In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1748–1758 (2019)
Subr, K., Soler, C., Durand, F.: Edge-preserving multiscale image decomposition based on local extrema. ACM Trans. Graph. 28(5), 1–9 (2009)
Tian, J., Qi, X., Qu, L., Tang, Y.: New spectrum ratio properties and features for shadow detection. Pattern Recogn. 51, 85–96 (2016)
Vicente, T.F.Y., Hou, L., Yu, C.P., Hoai, M., Samaras, D.: Large-scale training of shadow detectors with noisily-annotated shadow examples. In: European Conference on Computer Vision, pp. 816–832. Springer (2016)
Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1788–1797 (2018)
Wang, R., Zhang, Q., Fu, C.W., Shen, X., Zheng, W.S., Jia, J.: Underexposed photo enhancement using deep illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6849–6857 (2019)
Wang, T., Hu, X., Wang, Q., Heng, P.A., Fu, C.W.: Instance shadow detection. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1880–1889 (2020)
Yago Vicente, T.F., Hoai, M., Samaras, D.: Leave-one-out kernel optimization for shadow detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3388–3396 (2015)
Yagyu, S., Sakiyama, A., Tanaka, Y.: Edge preserving multiscale image decomposition with customized domain transform filters. In: 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 458–462. IEEE (2015)
Yang, X., Zhu, S., Xia, S., Zhou, D.: A new TLD target tracking method based on improved correlation filter and adaptive scale. Vis. Comput. 36(9), 1783–1795 (2020)
Zhang, L., Long, C., Zhang, X., Xiao, C.: Ris-gan: explore residual and illumination with generative adversarial networks for shadow removal. ArXiv preprint arXiv:1911.09178 (2019)
Zhang, L., Long, C., Zhang, X., Xiao, C.: Ris-gan: explore residual and illumination with generative adversarial networks for shadow removal. AAAI 2020: Thirty-Fourth AAAI Conf. Artif. Intell. 34(7), 12829–12836 (2020)
Zhang, S., Liang, R., Wang, M.: Shadowgan: shadow synthesis for virtual objects with conditional adversarial networks. Comput. Vis. Media 5(1), 105–115 (2019)
Zhang, X.C., Barron, J.T., Tsai, Y.T., Pandey, R., Zhang, X., Ng, R., Jacobs, D.E.: Portrait shadow manipulation. ACM Trans. Graph. 39(4), 78 (2020)
Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.: Distraction-aware shadow detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5167–5176 (2019)
Zhu, L., Deng, Z., Hu, X., Fu, C.W., Xu, X., Qin, J., Heng, P.A.: Bidirectional feature pyramid network with recurrent attention residual modules for shadow detection. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 121–136 (2018)
Acknowledgements
This work was supported by the Natural Science Foundation of Xinjiang Autonomous Region in China (NO. 2020D01A48) and the National Social Science Foundation Western Project (NO. 20XGL029).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wu, W., Wu, X. & Wan, Y. Single-image shadow removal using detail extraction and illumination estimation. Vis Comput 38, 1677–1687 (2022). https://doi.org/10.1007/s00371-021-02096-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-021-02096-4