Skip to main content

Probabilistic Fusion Framework Combining CNNs and Graphical Models for Multiresolution Satellite and UAV Image Classification

  • Conference paper
  • First Online:
Pattern Recognition (ICPR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15302))

Included in the following conference series:

  • 81 Accesses

Abstract

Image classification - or semantic segmentation - from input multiresolution imagery is a demanding task. In particular, when dealing with images of the same scene collected at the same time by very different acquisition systems, for example multispectral sensors onboard satellites and unmanned aerial vehicles (UAVs), the difference between the involved spatial resolutions can be very large and multiresolution information fusion is particularly challenging. This work proposes two novel multiresolution fusion approaches, based on deep convolutional networks, Bayesian modeling, and probabilistic graphical models, addressing the challenging case of input imagery with very diverse spatial resolutions. The first method aims to fuse the multimodal multiresolution imagery via a posterior probability decision fusion framework, after computing posteriors on the multiresolution data separately through deep neural networks or decision tree ensembles. The optimization of the parameters of the model is fully automated by also developing an approximate formulation of the expectation maximization (EM) algorithm. The second method aims to perform the fusion of the multimodal multiresolution information through a pyramidal tree structure, where the imagery can be inserted, modeled, and analyzed at its native resolutions. The application is to the semantic segmentation of areas affected by wildfires for burnt area mapping and management. The experimental validation is conducted with UAV and satellite data of the area of Marseille, France. The code is available at https://github.com/Ayana-Inria/BAS_UAV_satellite_fusion.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    In Sect. 3.2, the pixel lattice of the input coarser-resolution image was indicated S. Here, it is denoted \(S^0\) to distinguish it from the other lattices in the tree.

References

  1. Alvarez-Vanhard, E., Corpetti, T., Houet, T.: UAV & satellite synergies for optical remote sensing applications: a literature review. Sci. Remote Sens. 3, 100019 (2021)

    Article  Google Scholar 

  2. Benediktsson, J., Kanellopoulos, I.: Classification of multisource and hyperspectral data based on decision fusion. IEEE Trans. Geosci. Remote Sens. 37(3), 1367–1377 (1999)

    Article  Google Scholar 

  3. Bouman, C., Liu, B.: Multiple resolution segmentation of textured images. IEEE Trans. Pattern Anal. Mach. Intell. 13(2), 99–113 (1991)

    Article  Google Scholar 

  4. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  5. Briem, G., Benediktsson, J., Sveinsson, J.: Multiple classifiers applied to multisource remote sensing data. IEEE Trans. Geosci. Remote Sens. 40(10), 2291–2299 (2002)

    Article  Google Scholar 

  6. Cole-Rhodes, A., Johnson, K., LeMoigne, J., Zavorin, I.: Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient. IEEE Trans. Image Process. 12, 1495–1511 (2003)

    Article  MathSciNet  Google Scholar 

  7. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Series B (Stat. Methodol.) 39(1), 1–38 (1977)

    Google Scholar 

  8. Esch, T., et al.: Breaking new ground in mapping human settlements from space - the global urban footprint. ISPRS J. Photogramm. Remote. Sens. 134, 30–42 (2017)

    Article  Google Scholar 

  9. Fang, W., Fu, Y., Sheng, V.S.: Dual backbone interaction network for burned area segmentation in optical remote sensing images. IEEE Geosci. Remote Sens. Lett. 21, 1–5 (2024)

    Google Scholar 

  10. Gómez-Chova, L., Tuia, D., Moser, G., Camps-Valls, G.: Multimodal classification of remote sensing images: a review and future directions. Proc. IEEE 103(9), 1560–1584 (2015)

    Article  Google Scholar 

  11. Hyyppä, J., Hyyppä, H., Leckie, D., Gougeon, F., Yu, X., Maltamo, M.: Review of methods of small-footprint airborne laser scanning for extracting forest inventory data in boreal forests. Int. J. Remote Sens. 29(5), 1339–1366 (2008)

    Article  Google Scholar 

  12. Kato, Z., Zerubia, J.: Markov random fields in image segmentation. Found. Trends Signal Process. 5(1–2), 1–155 (2012)

    Google Scholar 

  13. Kim, J.B., Kim, H.J.: Multiresolution-based watersheds for efficient image segmentation. Pattern Recognit. Lett. 24(1), 473–488 (2003)

    Article  Google Scholar 

  14. Kuhn, E., Matias, C., Rebafka, T.: Properties of the stochastic approximation EM algorithm with mini-batch sampling. Stat. Comput. 30(6), 1725–1739 (2020). https://doi.org/10.1007/s11222-020-09968-0

    Article  MathSciNet  Google Scholar 

  15. Laferté, J.M., Heitz, F., Perez, P.: A multiresolution EM algorithm for unsupervised image classification. In: International Conference on Pattern Recognition (ICPR). vol. 2, pp. 849–853 (1996)

    Google Scholar 

  16. Laferté, J.M., Pérez, P., Heitz, F.: Discrete Markov image modeling and inference on the quadtree. IEEE Trans. Image Process. 9(3), 390–404 (2000)

    Article  MathSciNet  Google Scholar 

  17. Laine, A., Fan, J.: Texture classification by wavelet packet signatures. IEEE Trans. Pattern Anal. Mach. Intell. 15(11), 1186–1191 (1993)

    Article  Google Scholar 

  18. Li, S.Z.: Markov random field modeling in image analysis. Springer, 3rd edn. (2009)

    Google Scholar 

  19. Liu, H., et al.: Clusterformer for pine tree disease identification based on UAV remote sensing image segmentation. IEEE Trans. Geosci. Remote Sens. 62, 1–15 (2024)

    Article  Google Scholar 

  20. Liu, S., Cai, T., Tang, X., Wang, C.: MRL-Net: multi-scale representation learning network for COVID-19 lung CT image segmentation. IEEE J. Biomed. Health Inform. 27(9), 4317–4328 (2023)

    Article  Google Scholar 

  21. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440 (2015)

    Google Scholar 

  22. Luan, W., Zhang, X., Xiao, P., Wang, H., Chen, S.: Binary and fractional MODIS snow cover mapping boosted by machine learning and big Landsat data. IEEE Trans. Geosci. Remote Sens. 60, 1–14 (2022)

    Article  Google Scholar 

  23. Luotamo, M., Metsämäki, S., Klami, A.: Multiscale cloud detection in remote sensing images using a dual convolutional neural network. IEEE Trans. Geosci. Remote Sens. 59(6), 4972–4983 (2021)

    Article  Google Scholar 

  24. Luppino, L.T., et al.: Code-aligned autoencoders for unsupervised change detection in multimodal remote sensing images. IEEE Trans. Neural Netw. Learn. Syst. 35(1), 60–72 (2024)

    Article  Google Scholar 

  25. Mallat, S.: A wavelet tour of signal processing – The sparse way. Academic Press, 3rd edn. (2009)

    Google Scholar 

  26. Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(7), 3523–3542 (2022)

    Google Scholar 

  27. Moon, T.: The expectation-maximization algorithm. IEEE Signal Process. Mag. 13(6), 47–60 (1996)

    Article  Google Scholar 

  28. Moser, G., Serpico, S.B.: Unsupervised change detection from multichannel SAR data by Markovian data fusion. IEEE Trans. Geosci. Remote Sens. 47(7), 2114–2128 (2009)

    Article  Google Scholar 

  29. Pastorino, M., et al.: Multisensor and multiresolution remote sensing image classification through a causal hierarchical Markov framework and decision tree ensembles. Remote Sens. 13(5), 849 (2021)

    Article  Google Scholar 

  30. Pastorino, M., Moser, G., Serpico, S.B., Zerubia, J.: Semantic segmentation of remote-sensing images through fully convolutional neural networks and hierarchical probabilistic graphical models. IEEE Trans. Geosci. Remote Sens. 60(5407116), 1–16 (2022)

    Article  Google Scholar 

  31. Pyun, K., Lim, J., Won, C.S., Gray, R.M.: Image segmentation using hidden Markov Gauss mixture models. IEEE Trans. Image Process. 16(7), 1902–1911 (2007)

    Article  MathSciNet  Google Scholar 

  32. Rezaee, M., van der Zwet, P., Lelieveldt, B., van der Geest, R., Reiber, J.: A multiresolution image segmentation technique based on pyramidal segmentation and fuzzy clustering. IEEE Trans. Image Process. 9(7), 1238–1248 (2000)

    Article  Google Scholar 

  33. Richards, J.A.: Remote sensing digital image analysis: An introduction. Springer, 5th edn. (2013)

    Google Scholar 

  34. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  35. Song, P., Li, J., An, Z., Fan, H., Fan, L.: CTMFNet: CNN and transformer multiscale fusion network of remote sensing urban scene imagery. IEEE Trans. Geosci. Remote Sens. 61, 1–14 (2023)

    Google Scholar 

  36. Thoonen, G., Mahmood, Z., Peeters, S., Scheunders, P.: Multisource classification of color and hyperspectral images using color attribute profiles and composite decision fusion. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 5(2), 510–521 (2012)

    Google Scholar 

  37. van Rijthoven, M., Balkenhol, M., Silina, K., van der Laak, J., Ciompi, F.: HookNet: multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images. Med. Image Anal. 68, 101890 (2021)

    Google Scholar 

  38. Wang, L., Liu, J.: Texture classification using multiresolution Markov random field models. Pattern Recognit. Lett. 20(2), 171–182 (1999)

    Article  Google Scholar 

  39. Wang, L., Zhang, C., Li, R., Duan, C., Meng, X., Atkinson, P.M.: Scale-aware neural network for semantic segmentation of multi-resolution remote sensing images. Remote Sens. 13(24), 5015 (2021)

    Google Scholar 

  40. Waske, B., Benediktsson, J.A.: Fusion of support vector machines for classification of multisensor data. IEEE Trans. Geosci. Remote Sens. 45(12), 3858–3866 (2007)

    Article  Google Scholar 

  41. Wu, C.F.J.: On the convergence properties of the EM algorithm. Annal. Stat. 11(1), 95–103 (1983)

    Article  MathSciNet  Google Scholar 

  42. Zavorin, I., Moigne, J.: Use of multiresolution wavelet feature pyramids for automatic registration of multisensor imagery. IEEE Trans. Image Process. 14, 770–82 (2005)

    Article  Google Scholar 

  43. Zheng, G., Jiang, Z., Zhang, H., Yao, X.: Deep semantic segmentation of unmanned aerial vehicle remote sensing images based on fully convolutional neural network. Front. Earth Sci. 11, 1115805 (2023)

    Google Scholar 

  44. Zhou, F., et al.: Efficient inference for dynamic flexible interactions of neural populations. J. Mach. Learn. Res. 23(211), 1–49 (2022)

    MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Martina Pastorino .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pastorino, M., Moser, G., Guerra, F., Serpico, S.B., Zerubia, J. (2025). Probabilistic Fusion Framework Combining CNNs and Graphical Models for Multiresolution Satellite and UAV Image Classification. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15302. Springer, Cham. https://doi.org/10.1007/978-3-031-78166-7_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-78166-7_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-78165-0

  • Online ISBN: 978-3-031-78166-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics