Abstract
The existence of shadows is difficult to avoid in images. Also, it will affect object recognition and image understanding. But on the other hand, shadow can provide information about the light source and object shape. Therefore, accurate shadow detection and removal can contribute to many computer vision tasks. However, even the same object, its shadow will vary greatly under different lighting conditions. Thus it is quite challenging to detect and remove shadows from images. Recent research always treated these two tasks independently, but they are closely related to each other actually. Therefore, we propose a multi-task adversarial generative networks (mtGAN) that can detect and remove shadows simultaneously. In order to enhance shadow detection and shadow removal mutually, a cross-stitch unit is proposed to learn the optimal ways to fuse and constrain features between multi-tasks. Also, the combination weight of multi-task loss functions are learned according to the uncertainty distribution of each task, which is not set empirically as usual. Based on these multi-task learning strategies, the proposed mtGAN can jointly achieve shadow detection and removal tasks better than existing methods. In experiments, the effectiveness of the proposed mtGAN is shown.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Benedek, C., Szirányi, T.: Bayesian foreground and shadow detection in uncertain frame rate surveillance videos. IEEE Trans. Image Process. 17(4), 608–621 (2008)
Qi, M., Dai, J., Zhang, Q., Kong, J.: Cascaded cast shadow detection method in surveillance scenes. Optik-Int. J. Light Electron Opt. 125(3), 1396–1400 (2014)
Finlayson, G.D., Hordley, S.D., Lu, C., Drew, M.S.: On the removal of shadows from images. IEEE Trans. Patt. Recogn. Mach. Intelli. 28(1), 59–68 (2006)
Vazquez, E., Baldrich, R., De Weijer, J.V., Vanrell, M.: Describing reflectances for color segmentation robust to shadows, highlights, and textures. IEEE Trans. Pattern Recogn. Mach. Intell. 33(5), 917–930 (2011)
Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Detecting ground shadows in outdoor consumer photographs. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 322–335. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15552-9_24
Jiang, X., Schofield, A.J., Wyatt, J.L.: Shadow detection based on colour segmentation and estimated illumination. In: British Machine Vision Conference (BMVC), pp. 1–11 (2011)
Shen, L., Chua, T.W., Leman, K.: Shadow optimization from structured deep edge detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2067–2074. IEEE (2015)
Khan, S.H., Bennamoun, M., Sohel, F., Togneri, R.: Automatic shadow detection and removal from a single image. IEEE Trans. Pattern Recogn. Mach. Intell. 3, 431–446 (2016)
Zheng, Q., Qiao, X., Cao, Y., Lau, R.W.H.: Distraction-aware shadow detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5167–5176 (2019)
Finlayson, G.D., Hordley, S.D., Lu, C., Drew, M.S.: On the removal of shadows from images. IEEE Trans. Pattern Recogn. Mach. Intell. 28(1), 59–68 (2006)
Gryka, M., Terry, M., Brostow, G.J.: Learning to remove soft shadows. ACM Trans. Graph. (TOG) 34(5), 153 (2015)
Qu, L., Tian, J., He, S., Tang, Y., Lau, R.: DeshadowNet: a multi-context embedding deep network for shadow removal. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), no. 2, p. 3 (2017)
Yang, Q., Tan, K.-H., Ahuja, N.: Shadow removal using bilateral filtering. IEEE Trans. Image Process. 21(10), 4361–4368 (2012)
Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Recogn. Mach. Intell. 37(8), 1670–1687 (2015)
Le, H., Samaras, D.: Shadow removal via shadow image decomposition. In: International Conference on Computer Vision (ICCV) (2019)
Fu, L., Zhou, C., Guo, Q., Juefei-Xu, F., Wang, S.: Auto-exposure fusion for single-image shadow removal. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Wang, J., Li, X., Yang, J.: Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1788–1797 (2018)
Ding, B., Long, C., Zhang, L., Xiao, C.: ARGAN: attentive recurrent generative adversarial network for shadow detection and removal. In: International Conference on Computer Vision (ICCV) (2019)
Liu, Z., Yin, H., Wu, X., Wu, Z., Mi, Y., Wang, S.: From shadow generation to shadow removal. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Long, M., Wang, J.: Learning multiple tasks with deep relationship networks. arXiv: Learning (2015)
Ranjan, R., Patel, V.M., Chellappa, R.: Hyperface: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans. Pattern Recogn. Mach. Intell. 41(1), 121–135 (2019)
Cipolla, R., Gal, Y., Kendall, A.: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7482–7491 (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167D (2015)
Misra, I., Shrivastava, A., Gupta, A., Hebert, M.: Cross-stitch networks for multi-task learning. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3994–4003 (2016)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Guo, R., Dai, Q., Hoiem, D.: Single-image shadow detection and removal using paired regions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2033–2040 (2011)
Gong, H., Cosker, D.: Interactive shadow removal and ground truth for variable scene categories. In: British Machine Vision Conference (BMVC) (2014)
Acknowledgement
This work is partly supported by the Key Research and Development Program of Shaanxi (Program Nos. 2020GY-050, 2021ZDLGY15-01, 2021ZDLGY09-04, 2021GY-004), and Shenzhen International Science and Technology Cooperation Project (No. GJHZ20200731095204013).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Jiang, X., Hu, Z., Ni, Y., Li, Y., Feng, X. (2021). Shadow Detection and Removal Based on Multi-task Generative Adversarial Networks. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12890. Springer, Cham. https://doi.org/10.1007/978-3-030-87361-5_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-87361-5_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87360-8
Online ISBN: 978-3-030-87361-5
eBook Packages: Computer ScienceComputer Science (R0)