Abstract
Cloud detection (CD) with deep learning (DL) algorithms has been greatly developed in the applications involving the predictions of extreme weather and climate. In this review, the different conventional CD methods based on threshold, time differentiation, machine learning, and the intelligent algorithms including convolution neural networks (CNN), simple linear iterative clustering (SLIC), and semantic segmentation algorithms (SSAs) are introduced in detail, and, especially, the majority of CD publications employing the advanced and prevalent DL algorithms during the last decade are summarized and analyzed. First, in terms of the detection for different types of clouds, we meticulously compare the labels, scenarios and volumes of three popular CD datasets and put forward further the constructive recommendations about the cloud images selection, multi-bands images preprocessing, and truth labels combination for creating similar datasets. Subsequently, the structures, detection accuracies, and operating speeds of several different CD network models comprising the fully convolutional neural networks (FCNs), U-Net, SegNet, pyramid scene parsing network (PSPNet), as well as the associated derivatives are conducted elaborately to explore the comprehensively optimal performance for CD. In addition, aiming at expanding the applications in the resource-limited space-borne environment, we conclude the mainstream compression strategies of a number of different lightweight networks. Finally, the various limitations constraining the performance of the existing state-of-the-art DL CD methods and the corresponding development tendency are presented, which, expectantly, could be referential for the following researches.
Similar content being viewed by others
References
Zhu, Z., Woodcock, C.E.: Continuous change detection and classification of land cover using all available Landsat data. Remote Sens. Environ. 144, 152–171 (2014)
Lu, C., Bai, Z.: Characteristics and typical applications of GF-1 satellite. In Proceedings of the 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2015), Milan, Italy, 26–31 July; pp. 1246–1249 (2015)
Xu, X., Shi, Z., Pan, B.: l0-based sparse hyperspectral unmixing using spectral information and a multi-objectives formulation. ISPRS J. Photogramm. Remote Sens. 141, 46–58 (2018)
Wei, J., Wei, H., Li, Z.: Estimating 1-km-resolution PM2.5 concentrations across China using the space-time random forest approach. Remote Sensing Environ. 231: 111221 (2019)
Luo, S., Li, H., Shen, H.: Deeply supervised convolutional neural network for shadow detection based on a novel aerial shadow imagery dataset. ISPRS J. Photogramm. Remote. Sens. 167, 443–457 (2020). https://doi.org/10.1016/j.isprsjprs.2020.07.016
Kong, W., Jia, Y., et al.: Hybrid approaches based on deep whole-sky-image learning to photovoltaic generation forecasting. Appl. Energy 280, 115875 (2020). https://doi.org/10.1016/j.apenergy.2020.115875
Yao, F., Wang, J., Wang, C.: Constructing long-term high-frequency time series of global lake and reservoir areas using Landsat imagery. Remote Sens. Environ. 232, 111210 (2019)
Qian, Y., Chen, Q., Zhu, G.: Infrared small target detection based on saliency and gradients difference measure. Opt. Quant. Electron. 52, 151 (2020). https://doi.org/10.1007/s11082-020-2197-x
Wei, Y., Cheng, Z., Zhu, B.: Multiscale hysteresis threshold detection algorithm for a small infrared target in a complex background. Opt. Quant. Electron. 51, 98 (2019). https://doi.org/10.1007/s11082-019-1808-x
Ding, Z.G., Zhang, T.Y., Li, Y.: A Ship ISAR Imaging Algorithm Based on Generalized Radon-Fourier Transform with Low SNR. IEEE Trans. Geosci. Remote Sensing 57(9) (2019)
Zhu, Z., Woodcock, C.E.: Automated cloud, cloud shadow, and snow detection in multitemporal Landsat data: An algorithm designed specifically for monitoring land cover change. Remote Sens. Environ. 152, 217–234 (2014)
Carvalho, O., de Carvalho Júnior, O., Albuquerque, A., Bem, P., et al.: Instance segmentation for large, multi-channel remote sensing imagery using Mask-RCNN and a Mosaicking approach. Remote Sens. 13, 39 (2021). https://doi.org/10.3390/rs13010039
Zhang, Y., Rossow, W.B., Lacis, A.A.: Calculation of radiative fluxes from the surface to top of atmosphere based on ISCCP and other global data sets. Geophys. Res. 109, 1121–1125 (2004)
Liu, J.: Improvement of dynamic threshold value extraction technic in FY-2 cloud detection. Infrared Millim. Waves 29, 288–292 (2010)
Du, J., Min, Q., Zhang, P., Guo, J., Yang, J., Yin, B.: Short-term solar irradiance forecasts using sky images and radiative transfer model. Energies 11:1107 (2018). https://doi.org/10.3390/en11051107
Yang, Y., Zheng, H., Chen, H.: Automated cloud detection algorithm for multi-spectral high spatial resolution images using Landsat-8 OLI. In Advances in Image and Graphics Technologies. Springer: Berlin/Heidelberg, Germany, pp. 396–407 (2015)
Long, C.N., Sabburg, J.M., Calbó, J.D.: Retrieving cloud characteristics from ground-based daytime color all-sky images. Atmos. Ocean. Tech. 23, 633–652 (2006)
Kreuter, A., Zangerl, M., Schwarzmann, M.: All-sky imaging: a simple, versatile system for atmospheric research. Appl. Optics 48, 1091–1097 (2009)
Heinle, A., Macke, A., Srivastav, A.: Automatic cloud classification of whole sky images. Atmos. Meas. Tech. 3, 557–567 (2010). https://doi.org/10.5194/amt-3-557-2010
Frantz, D., Haß, E., Uhl, A.: Improvement of the Fmask algorithm for Sentinel-2 images: separating clouds from bright surfaces based on parallax effects. Remote Sens. Environ. 215, 471–481 (2018)
Huang, C., Thomas, N., Goward, S.N.: Automated masking of cloud and cloud shadow for forest change analysis using Landsat images. Int. J. Remote Sens. 31, 5449–5464 (2010). https://doi.org/10.1080/01431160903369642
Zhu, Z., Woodcock, C.E.: Object-based cloud and cloud shadow detection in Landsat imagery. Remote Sens. Environ. 118, 83–94 (2012). https://doi.org/10.1016/j.rse.2011.10.028
Louis, J., Debaecker, V., Pflug, B.: Sentinel-2 Sen2Cor: L2A processor for users. Proceedings Living Planet Symposium 2016, Spacebooks Online, pp. 1–8 (2016)
Rossow, W.B., Garder, L.C.: Cloud detection using satellite measurements of infrared and visible radiances for ISCCP. Climate 6, 2341–2369 (1993)
Saunders, R.W., Kriebel, K.T.: An improved method for detecting clear sky and cloudy radiances from AVHRR data. Remote Sens. 9, 123–150 (1988)
Yang, Y., Di, G.: Selection of the automated thresholding algorithm for Multi-angle Imaging SpectroRadiometer Radiometric Camera-by-Camera Cloud Mask over land. Remote Sens. Environ. 107, 159–171 (2007). https://doi.org/10.1016/j.rse.2006.05.020
Ackerman, S., Strabala, K., Menzel, P.: Discriminating clear-sky from cloud with MODIS algorithm theoretical basis document (MOD35). Cooperative Institute for Meteorological Satellite Studies, University of Wisconsin, MODIS Cloud Mask Team (2010)
Chen, Y., Fan, R., Wang, J.: Cloud detection of ZY-3 satellite remote sensing images based on deep learning. Acta Optica Sinica 38(1), 1–6 (2018)
Sui, Y., He, B., Fu, T.: Energy-based cloud detection in multispectral images based on the SVM technique. Int. J. Remote Sens. 40, 5530–5543 (2019). https://doi.org/10.1080/01431161.2019.1580788
Guo, Y., Cao, X., Liu, B., Gao, M.: Cloud detection for satellite imagery using attention-based U-Net convolutional neural network. Symmetry. 12(6), 1056 (2020). https://doi.org/10.3390/sym12061056
Hughes, M.J., Hayes, D.J.: Automated detection of cloud and cloud shadow in single-date Landsat imagery using neural networks and spatial post-processing. Remote Sens. 6, 4907–4926 (2014). https://doi.org/10.3390/rs6064907
Zi, Y., Xie, F.: A cloud detection method for Landsat 8 images based on PCANet. Remote Sens. 10(6), 877 (2018)
Hollstein, A., Segl, K., Guanter, L., Brell, M., Enesco, M.: Ready-to-use methods for the detection of clouds, cirrus, snow, shadow, water and clear sky pixels in Sentinel-2 MSI images. Remote Sens. 8(8), 666 (2016). https://doi.org/10.3390/rs8080666
Ghasemian, N., Akhoondzadeh, M.: Introducing two Random Forest based methods for cloud detection in remote sensing images. Adv. Space Res. 62(2), 288–303 (2018). https://doi.org/10.1016/j.asr.2018.04.03
Wieland, M., Li, Y., Martinis, S.: Multi-sensor cloud and cloud shadow segmentation with a convolutional neural network. Remote Sens. Environ. 230, 111203 (2019). https://doi.org/10.1016/j.rse.2019.05.022
Souzaecher, M.P., Pereira, E.B., Bins, L.S., Andrade, M.A.R.: A simple method for the assessment of the cloud cover state in high-latitude regions by a ground-based digital camera. Atmos. Ocean. Tech. 23, 437–447 (2004). https://doi.org/10.1175/jtech1833.1
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324. https://doi.org/10.1109/5.726791
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40, 834–848 (2018)
Shao, Z.F., Pan, Y., Diao, C.Y., Cai, J.J.: Cloud detection in remote sensing images based on multiscale features-convolutional neural network. IEEE Trans. Geosci. Remote Sens. 57, 4062–4076 (2019)
An, J., Ma, J.W.: Automatic cloud segmentation based on the fully convolutional neural networks. J. Signal Process., 1003–0530 (2019)
Alistair, F., Panagiotis, S., Jan-Peter, M.: Cloud FCN: accurate and robust cloud detection for satellite imagery with deep learning. Remote Sens. 11(19), 2312 (2019)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, vol 7–12, 1–9 (2015)
Lu, J.Y., Wang, Y.Z., Zhu, Y.Q.: P_Segnet and NP_Segnet: New neural network architectures for cloud detection of remote sensing images. IEEE Access Digital Object Identifier. (2019). https://doi.org/10.1109/ACCESS.2019.2925565
Xie, F.Y., Shi, M.Y., Shi, Z.W., Yin, J.H., Zhao, D.P.: Multilevel cloud detection in remote sensing images based on deep learning. IEEE-INST Electrical Electronics Engineers INC. 2017, pp 3631–3640 (2017). https://doi.org/10.1109/JSTARS.2017.2686488
Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: "Pyramid Scene Parsing Network," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, pp. 6230–6239 (2017). https://doi.org/10.1109/CVPR.2017.660.
Yang, J., Guo, J., Yue, H., Liu, Z., Hu, H., Li, K.: CDnet: CNN-based cloud detection for remote sensing imagery. IEEE Trans. Geosci. Remote Sensing, 57(8), 6195–6211 (2019). https://doi.org/10.1109/TGRS.2019.2904868.
Ji, S., Dai, P., Lu, M., Zhang, Y.: Simultaneous cloud detection and removal from bitemporal remote sensing images using cascade convolutional neural networks. IEEE Trans. Geosci. Remote Sens. 59(1), 732–748 (2021). https://doi.org/10.1109/TGRS.2020.2994349
Scaramuzza, P.L., Bouchard, M.A., Dwyer, J.L.: Development of the Landsat data continuity mission cloud-cover assessment algorithms. IEEE Trans. Geosci. Remote Sens. 50(4), 1140–1154 (2012). https://doi.org/10.1109/TGRS.2011.2164087
Foga, S., Scaramuzza, P.L., Guo, S., Zhu, Z., Dilley, R.D., Beckmann, T., Schmidt, G.L., Dwyer, J.L., Hughes, M.J., Laue, B.: Cloud detection algorithm comparison and validation for operational landsat data products. Remote Sens. Environ. 194, 379–390 (2017)
Xie, W., Liu, D., Yang, M., Chen, S., Wang, B., Wang, Z., Xia, Y., Liu, Y., Wang, Y., Zhang, C.: SegCloud: a novel cloud image segmentation model using a deep convolutional neural network for ground-based all-sky-view camera observation. Atmos. Meas. Tech. 13, 1953–1961 (2020). https://doi.org/10.5194/amt-13-1953-2020
Dev, S., Lee, Y.H., Winkler, S.: Color-based segmentation of sky/cloud images from ground-based cameras. IEEE J. Sel. Top. Appl. 10, 231–242 (2017)
Li, Q., Lu, W., Yang, J.: A hybrid thresholding algorithm for cloud detection on ground-based color images. J. Atmos. Ocean. Tech. 28, 1286–1296 (2011)
Zhou, Y., Yang, Y., Gao, M., Zhai, P.W.: Cloud detection over snow and ice with oxygen A- and B-band observations from the Earth Polychromatic Imaging Camera (EPIC). Atmos. Meas. Tech. 13, 1575–1591 (2020). https://doi.org/10.5194/amt-13-1575-2020
Ma, N., Sun, L., Wang, Q., Yu, Z.J., Liu, S.C.: Improved cloud detection for Landsat 8 images using a combined neural network model. Remote Sensing Lett.11, 274–282 (2019). https://doi.org/10.1080/2150704X.2019.1708503
Sejnowski, T.J.: The deep learning revolution. Cambridge, MA: MIT Press. (2018). https://doi.org/10.7551/mitpress/11474.001.0001
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei, F.L.: ImageNet: a large-scale hierarchical image database. In IEEE Conf. Computer Vision and Pattern Detection, Miami, FL, 20–25 June (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2012)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image detection. In Proc. IEEE Conf. Computer Vision and Pattern Detection, Las Vegas, NV, 26 June–1 July, pp. 770–778. IEEE. (2016)
Mateo, G.G., Laparra, V., G, C.L.: Transferring deep learning models for cloud detection between Landsat-8 and Proba-V. ISPRS JOURNAL OF PHOTOGRAMMETRY AND REMOTE SENSING. 160(1–17) (2019). https://doi.org/10.1016/j.isprsjprs.2019.11.024
Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. IEEE Transactions on pattern analysis and machine intelligence, vol 39, NO. 4, APRIL (2017)
Pei, L., Liu, Y., Tan, H.: Cloud detection of ZY-3 satellite remote sensing images based on improved fully convolutional neural networks. Laser Optoelectronics Progress 56(05), 226–233 (2019)
Rune, J., Thomas, F.: A cloud detection algorithm for satellite imagery based on deep learning. Remote Sens. Environ. 229, 247–259 (2019)
Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), pp 2481–2495 (2017)
Chai, D.F., Newsam, S., Zhang, H.K.K., Qiu, Y.F., Huang, J.F.: Cloud and cloud shadow detection in Landsat imagery based on deep convolutional neural networks. Remote Sens. Environ. 225, 307–316 (2019). https://doi.org/10.1016/j.rse.2019.03.007
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image detection, Proc. Int. Conf. Learn. Representat. pp 1–14 (2015)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In Proc. Int. Conf. Med. Image Computing and Computer-Assisted Intervention, pp. 234–241 (2015)
Yang, J., Guo, J., Yue, H., Liu, Z., Hu, H., Li, K.: CDnet: CNN-Based Cloud Detection for Remote Sensing Imagery. IEEE Trans. Geosci. Remote Sensing 57(8), pp. 6195–6211 (2019). https://doi.org/10.1109/TGRS.2019.2904868
Yu, J.C., Li, Y.C., Zheng, X.X.: An effective cloud detection method for Gaofen-5 Images via deep learning. Remote Sens. 12(13), 2106 (2020). https://doi.org/10.3390/rs12132106
Landola, F.N., Han, S., Moskewicz, M.W.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size. Computer Vision and Pattern Recognition. v4 (2016)
Howard, A.G., Zhu, M., Chen, B.: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Computer Vision and Pattern Recognition. (2017)
Zhang, X., Zhou, X., Lin, M.: ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. Computer Vision and Pattern Recognition. (2017)
Sandler, M., Howard, A., Zhu, M.: Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification. Detection and Segmentation, Computer Vision and Pattern Recognition (2018)
Ma, N., Zhang, X., Zheng, H.T.: ShuffleNet V2:Practical Guidelines for Efficient CNN Architecture Design. European Conference on Computer Vision. (2018)
Wang, M., Zhang, Z.Q., Dong, Z.P.: Stream-computing based high accuracy on-board real-time cloud detection for high resolution optical satellite imagery. Acta Geodaetica et Cartographica Sinica 47(06), 760–769 (2018)
Zhang, Y., Yang, T.: An efficient object detection algorithm for remote sensing image based on deep learning. Information Technol. 43(09), 163–167 (2019)
Yang, Z., Guo, H., Jiao, H.: An Ultra-lightweight Real-time Detection Method. China Academic Journal Electronic Publishing House, pp 122–128 (2019)
Zhang, Z.X., Xu, G.D., Song, J.N.: CubeSat cloud detection based on JPEG2000 compression and deep learning. Adv. Mech. Eng. 10(10) (2018)
Zhang, Z.X., Iwasaki, A., Xu, G.D.: Cloud detection on small satellites based on lightweight U-net and image compression. Appl. Remote Sens 13(2), 026502 (2019)
Zhang, J., Li, X., Li, L.: Lightweight U-Net for cloud detection of visible and thermal infrared remote sensing images. Opt Quant Electron 52, 397 (2020). https://doi.org/10.1007/s11082-020-02500-8
Kanungo, T., Mount, D.M., Netanyahu, N.S., Piatko, C.D., Silverman, R., Wu, A.Y.: An efficient k-means clustering algorithm: Analysis and implementation. IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 881–892 (2002)
Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5) 603–619 (2002)
Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)
Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the Mumford and Shah model. Int. J. Comput. Vision 50(3), 271–293 (2002)
Zhang, Q., Xiao, C.: Cloud detection of RGB color aerial photographs by progressive refinement scheme. IEEE Trans. Geosci. Remote Sens. 52(11) 7264–7275 (2014)
An, Z., Shi, Z.: Scene learning for cloud detection on remote-sensing images. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 8(8), 4206–4222 (2015)
Zhao, R., Ouyang, W., Li, H., Wang, X.: Saliency detection by multi-context deep learning. Proc. IEEE Conf. Comput. Vision Pattern Recognit. 1265–1274 (2015)
Chai, Y., Fu, K., Sun, X.: Compact cloud detection with bidirectional self-attention knowledge distillation. Remote Sens. 12(17), 2770 (2020). https://doi.org/10.3390/rs12172770
Acknowledgements
This work was supported by the National Natural Science Foundation of China (61975222).
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Li, L., Li, X., Jiang, L. et al. A review on deep learning techniques for cloud detection methodologies and challenges. SIViP 15, 1527–1535 (2021). https://doi.org/10.1007/s11760-021-01885-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-021-01885-7