Abstract
Image classification - or semantic segmentation - from input multiresolution imagery is a demanding task. In particular, when dealing with images of the same scene collected at the same time by very different acquisition systems, for example multispectral sensors onboard satellites and unmanned aerial vehicles (UAVs), the difference between the involved spatial resolutions can be very large and multiresolution information fusion is particularly challenging. This work proposes two novel multiresolution fusion approaches, based on deep convolutional networks, Bayesian modeling, and probabilistic graphical models, addressing the challenging case of input imagery with very diverse spatial resolutions. The first method aims to fuse the multimodal multiresolution imagery via a posterior probability decision fusion framework, after computing posteriors on the multiresolution data separately through deep neural networks or decision tree ensembles. The optimization of the parameters of the model is fully automated by also developing an approximate formulation of the expectation maximization (EM) algorithm. The second method aims to perform the fusion of the multimodal multiresolution information through a pyramidal tree structure, where the imagery can be inserted, modeled, and analyzed at its native resolutions. The application is to the semantic segmentation of areas affected by wildfires for burnt area mapping and management. The experimental validation is conducted with UAV and satellite data of the area of Marseille, France. The code is available at https://github.com/Ayana-Inria/BAS_UAV_satellite_fusion.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
In Sect. 3.2, the pixel lattice of the input coarser-resolution image was indicated S. Here, it is denoted \(S^0\) to distinguish it from the other lattices in the tree.
References
Alvarez-Vanhard, E., Corpetti, T., Houet, T.: UAV & satellite synergies for optical remote sensing applications: a literature review. Sci. Remote Sens. 3, 100019 (2021)
Benediktsson, J., Kanellopoulos, I.: Classification of multisource and hyperspectral data based on decision fusion. IEEE Trans. Geosci. Remote Sens. 37(3), 1367–1377 (1999)
Bouman, C., Liu, B.: Multiple resolution segmentation of textured images. IEEE Trans. Pattern Anal. Mach. Intell. 13(2), 99–113 (1991)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Briem, G., Benediktsson, J., Sveinsson, J.: Multiple classifiers applied to multisource remote sensing data. IEEE Trans. Geosci. Remote Sens. 40(10), 2291–2299 (2002)
Cole-Rhodes, A., Johnson, K., LeMoigne, J., Zavorin, I.: Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient. IEEE Trans. Image Process. 12, 1495–1511 (2003)
Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Series B (Stat. Methodol.) 39(1), 1–38 (1977)
Esch, T., et al.: Breaking new ground in mapping human settlements from space - the global urban footprint. ISPRS J. Photogramm. Remote. Sens. 134, 30–42 (2017)
Fang, W., Fu, Y., Sheng, V.S.: Dual backbone interaction network for burned area segmentation in optical remote sensing images. IEEE Geosci. Remote Sens. Lett. 21, 1–5 (2024)
Gómez-Chova, L., Tuia, D., Moser, G., Camps-Valls, G.: Multimodal classification of remote sensing images: a review and future directions. Proc. IEEE 103(9), 1560–1584 (2015)
Hyyppä, J., Hyyppä, H., Leckie, D., Gougeon, F., Yu, X., Maltamo, M.: Review of methods of small-footprint airborne laser scanning for extracting forest inventory data in boreal forests. Int. J. Remote Sens. 29(5), 1339–1366 (2008)
Kato, Z., Zerubia, J.: Markov random fields in image segmentation. Found. Trends Signal Process. 5(1–2), 1–155 (2012)
Kim, J.B., Kim, H.J.: Multiresolution-based watersheds for efficient image segmentation. Pattern Recognit. Lett. 24(1), 473–488 (2003)
Kuhn, E., Matias, C., Rebafka, T.: Properties of the stochastic approximation EM algorithm with mini-batch sampling. Stat. Comput. 30(6), 1725–1739 (2020). https://doi.org/10.1007/s11222-020-09968-0
Laferté, J.M., Heitz, F., Perez, P.: A multiresolution EM algorithm for unsupervised image classification. In: International Conference on Pattern Recognition (ICPR). vol. 2, pp. 849–853 (1996)
Laferté, J.M., Pérez, P., Heitz, F.: Discrete Markov image modeling and inference on the quadtree. IEEE Trans. Image Process. 9(3), 390–404 (2000)
Laine, A., Fan, J.: Texture classification by wavelet packet signatures. IEEE Trans. Pattern Anal. Mach. Intell. 15(11), 1186–1191 (1993)
Li, S.Z.: Markov random field modeling in image analysis. Springer, 3rd edn. (2009)
Liu, H., et al.: Clusterformer for pine tree disease identification based on UAV remote sensing image segmentation. IEEE Trans. Geosci. Remote Sens. 62, 1–15 (2024)
Liu, S., Cai, T., Tang, X., Wang, C.: MRL-Net: multi-scale representation learning network for COVID-19 lung CT image segmentation. IEEE J. Biomed. Health Inform. 27(9), 4317–4328 (2023)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440 (2015)
Luan, W., Zhang, X., Xiao, P., Wang, H., Chen, S.: Binary and fractional MODIS snow cover mapping boosted by machine learning and big Landsat data. IEEE Trans. Geosci. Remote Sens. 60, 1–14 (2022)
Luotamo, M., Metsämäki, S., Klami, A.: Multiscale cloud detection in remote sensing images using a dual convolutional neural network. IEEE Trans. Geosci. Remote Sens. 59(6), 4972–4983 (2021)
Luppino, L.T., et al.: Code-aligned autoencoders for unsupervised change detection in multimodal remote sensing images. IEEE Trans. Neural Netw. Learn. Syst. 35(1), 60–72 (2024)
Mallat, S.: A wavelet tour of signal processing – The sparse way. Academic Press, 3rd edn. (2009)
Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(7), 3523–3542 (2022)
Moon, T.: The expectation-maximization algorithm. IEEE Signal Process. Mag. 13(6), 47–60 (1996)
Moser, G., Serpico, S.B.: Unsupervised change detection from multichannel SAR data by Markovian data fusion. IEEE Trans. Geosci. Remote Sens. 47(7), 2114–2128 (2009)
Pastorino, M., et al.: Multisensor and multiresolution remote sensing image classification through a causal hierarchical Markov framework and decision tree ensembles. Remote Sens. 13(5), 849 (2021)
Pastorino, M., Moser, G., Serpico, S.B., Zerubia, J.: Semantic segmentation of remote-sensing images through fully convolutional neural networks and hierarchical probabilistic graphical models. IEEE Trans. Geosci. Remote Sens. 60(5407116), 1–16 (2022)
Pyun, K., Lim, J., Won, C.S., Gray, R.M.: Image segmentation using hidden Markov Gauss mixture models. IEEE Trans. Image Process. 16(7), 1902–1911 (2007)
Rezaee, M., van der Zwet, P., Lelieveldt, B., van der Geest, R., Reiber, J.: A multiresolution image segmentation technique based on pyramidal segmentation and fuzzy clustering. IEEE Trans. Image Process. 9(7), 1238–1248 (2000)
Richards, J.A.: Remote sensing digital image analysis: An introduction. Springer, 5th edn. (2013)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Song, P., Li, J., An, Z., Fan, H., Fan, L.: CTMFNet: CNN and transformer multiscale fusion network of remote sensing urban scene imagery. IEEE Trans. Geosci. Remote Sens. 61, 1–14 (2023)
Thoonen, G., Mahmood, Z., Peeters, S., Scheunders, P.: Multisource classification of color and hyperspectral images using color attribute profiles and composite decision fusion. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 5(2), 510–521 (2012)
van Rijthoven, M., Balkenhol, M., Silina, K., van der Laak, J., Ciompi, F.: HookNet: multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images. Med. Image Anal. 68, 101890 (2021)
Wang, L., Liu, J.: Texture classification using multiresolution Markov random field models. Pattern Recognit. Lett. 20(2), 171–182 (1999)
Wang, L., Zhang, C., Li, R., Duan, C., Meng, X., Atkinson, P.M.: Scale-aware neural network for semantic segmentation of multi-resolution remote sensing images. Remote Sens. 13(24), 5015 (2021)
Waske, B., Benediktsson, J.A.: Fusion of support vector machines for classification of multisensor data. IEEE Trans. Geosci. Remote Sens. 45(12), 3858–3866 (2007)
Wu, C.F.J.: On the convergence properties of the EM algorithm. Annal. Stat. 11(1), 95–103 (1983)
Zavorin, I., Moigne, J.: Use of multiresolution wavelet feature pyramids for automatic registration of multisensor imagery. IEEE Trans. Image Process. 14, 770–82 (2005)
Zheng, G., Jiang, Z., Zhang, H., Yao, X.: Deep semantic segmentation of unmanned aerial vehicle remote sensing images based on fully convolutional neural network. Front. Earth Sci. 11, 1115805 (2023)
Zhou, F., et al.: Efficient inference for dynamic flexible interactions of neural populations. J. Mach. Learn. Res. 23(211), 1–49 (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pastorino, M., Moser, G., Guerra, F., Serpico, S.B., Zerubia, J. (2025). Probabilistic Fusion Framework Combining CNNs and Graphical Models for Multiresolution Satellite and UAV Image Classification. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15302. Springer, Cham. https://doi.org/10.1007/978-3-031-78166-7_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-78166-7_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-78165-0
Online ISBN: 978-3-031-78166-7
eBook Packages: Computer ScienceComputer Science (R0)