Abstract
Modern neural networks achieve state-of-the-art results on land cover classification from satellite imagery, as is the case for almost all vision tasks. One of the main challenges in this context is dealing with geographic variability in both image and label distributions. To tackle this problem, we study the effectiveness of incorporating bioclimatic information into neural network training and prediction. Such auxiliary data can easily be extracted from freely available rasters at satellite images’ georeferenced locations. We compare two methods of incorporation, learned embeddings and conditional batch normalization, to a bioclimate-agnostic baseline ResNet18. In our experiments on the EuroSAT and BigEarthNet datasets, we find that especially the use of conditional batch normalization improves the network’s overall accuracy, generalizability, as well as training efficiency, in both a supervised and a self-supervised learning setup. Code and data are publicly available at https://t.ly/NDQFF.
This work was partly funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - SFB 1502/1-2022 - Projektnummer: 450058266 and partly funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2070 – 390732324.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alemohammad, H., Booth, K.: LandCoverNet: a global benchmark land cover classification training dataset. In: AI for Earth Sciences Workshop at NeurIPS (2020)
Audebert, N., Le Saux, B., Lefèvre, S.: Beyond RGB: very high resolution urban remote sensing with multimodal deep networks. ISPRS J. Photogramm. Remote. Sens. 140, 20–32 (2018)
Ayush, K., et al.: Geography-aware self-supervised learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10181–10190 (2021)
Cao, Y., Steffey, S., He, J., Xiao, D., Tao, C., Chen, P., Müller, H.: Medical image retrieval: a multimodal approach. Cancer Inform. 13, CIN–S14053 (2014)
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)
De Vries, H., Strub, F., Mary, J., Larochelle, H., Pietquin, O., Courville, A.C.: Modulating early visual processing by language. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Demir, I., et al.: DeepGlobe 2018: a challenge to parse the earth through satellite images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 172–181 (2018)
Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. In: International Conference on Learning Representations (2016)
Eddin, M.H.S., Roscher, R., Gall, J.: Location-aware adaptive normalization: a deep learning approach for wildfire danger forecasting. IEEE Trans. Geosci. Remote Sens. 61, 1–18 (2023)
EEA: Biogeographical regions Europe 2016 (2016). https://www.eea.europa.eu/ds_resolveuid/9b7911cc33ad4a9c940847a7ff653a40
Farr, T.G., et al.: The shuttle radar topography mission. Rev. Geophys. 45(2) (2007)
Fick, S.E., Hijmans, R.J.: WorldClim 2: new 1-km spatial resolution climate surfaces for global land areas. Int. J. Climatol. 37(12), 4302–4315 (2017). https://worldclim.org/data/worldclim21.html
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Heidler, K., et al.: Self-supervised audiovisual representation learning for remote sensing data. Int. J. Appl. Earth Obs. Geoinf. 116, 103130 (2023)
Helber, P., Bischke, B., Dengel, A., Borth, D.: EuroSAT: a novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 12(7), 2217–2226 (2019)
Hu, L., Robinson, C., Dilkina, B.: Model generalization in deep learning applications for land cover mapping. arXiv preprint arXiv:2008.10351 (2020)
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1501–1510 (2017)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)
Jain, A., Singh, A., Koppula, H.S., Soh, S., Saxena, A.: Recurrent neural networks for driver activity anticipation via sensory-fusion architecture. In: IEEE International Conference on Robotics and Automation, pp. 3118–3125 (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference for Learning Representations (2014)
Kramer, M.A.: Nonlinear principal component analysis using autoassociative neural networks. AIChE J. 37(2), 233–243 (1991)
Lenz, I., Lee, H., Saxena, A.: Deep learning for detecting robotic grasps. Int. J. Robot. Res. 34(4–5), 705–724 (2015)
Li, W., Chen, K., Chen, H., Shi, Z.: Geographical knowledge-driven representation learning for remote sensing images. IEEE Trans. Geosci. Remote Sens. 60, 1–16 (2021)
Li, Y., Wang, N., Shi, J., Hou, X., Liu, J.: Adaptive batch normalization for practical domain adaptation. Pattern Recogn. 80, 109–117 (2018)
Liang, M., Li, Z., Chen, T., Zeng, J.: Integrative data analysis of multi-platform cancer data with a multimodal deep learning approach. IEEE/ACM Trans. Comput. Biol. Bioinf. 12(4), 928–937 (2014)
Lu, X., Gong, T., Zheng, X.: Multisource compensation network for remote sensing cross-domain scene classification. IEEE Trans. Geosci. Remote Sens. 58(4), 2504–2515 (2019)
Marín, J., Escalera, S.: SSSGAN: satellite style and structure generative adversarial networks. Remote Sens. 13(19), 3984 (2021)
Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Proceedings of the 28th International Conference on Machine Learning, pp. 689–696 (2011)
Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2337–2346 (2019)
Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: FiLM: visual reasoning with a general conditioning layer. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Ramachandram, D., Taylor, G.W.: Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process. Mag. 34(6), 96–108 (2017)
Scheibenreif, L., Hanna, J., Mommert, M., Borth, D.: Self-supervised vision transformers for land-cover segmentation and classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1422–1431 (2022)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Sheth, I., Rahman, A.A., Havaei, M., Kahou, S.E.: Pitfalls of conditional batch normalization for contextual multi-modal learning. In: I Can’t Believe It’s Not Better Workshop at NeurIPS (2022)
Srivastava, N., Salakhutdinov, R.R.: Multimodal learning with deep Boltzmann machines. In: Advances in Neural Information Processing Systems, vol. 25 (2012)
Suel, E., Bhatt, S., Brauer, M., Flaxman, S., Ezzati, M.: Multimodal deep learning from satellite and street-level imagery for measuring income, overcrowding, and environmental deprivation in urban areas. Remote Sens. Environ. 257, 112339 (2021)
Sumbul, G., Charfuelan, M., Demir, B., Markl, V.: BigEarthNet: a large-scale benchmark archive for remote sensing image understanding. In: IEEE International Geoscience and Remote Sensing Symposium, pp. 5901–5904. IEEE (2019)
Sumbul, G., et al.: BigEarthNet-MM: a large-scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval [software and data sets]. IEEE Geosci. Remote Sens. Mag. 9(3), 174–180 (2021)
Sun, Y., Hua, Y., Mou, L., Zhu, X.X.: CG-Net: conditional GIS-aware network for individual building segmentation in VHR SAR images. IEEE Trans. Geosci. Remote Sens. 60, 1–15 (2021)
Tang, K., Paluri, M., Fei-Fei, L., Fergus, R., Bourdev, L.: Improving image classification with location context. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1008–1016 (2015)
Townshend, J.G.: Land cover. Int. J. Remote Sens. 13(6–7), 1319–1328 (1992)
Tuia, D., Persello, C., Bruzzone, L.: Domain adaptation for the classification of remote sensing data: an overview of recent advances. IEEE Geosci. Remote Sens. Mag. 4(2), 41–57 (2016)
Turner, M.G., Gardner, R.H.: Landscape Ecology in Theory and Practice. Springer, New York (2015). https://doi.org/10.1007/978-1-4939-2794-4
Wang, J., Zheng, Z., Ma, A., Lu, X., Zhong, Y.: LoveDA: a remote sensing land-cover dataset for domain adaptive semantic segmentation. In: NeurIPS Datasets and Benchmarks Track (2021)
Wang, X., Yu, K., Dong, C., Loy, C.C.: Recovering realistic texture in image super-resolution by deep spatial feature transform. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 606–615 (2018)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Wickramasinghe, C.S., Marino, D.L., Manic, M.: ResNet autoencoders for unsupervised feature learning from high-dimensional data: deep models resistant to performance degradation. IEEE Access 9, 40511–40520 (2021)
Wulder, M.A., Coops, N.C., Roy, D.P., White, J.C., Hermosilla, T.: Land cover 2.0. Int. J. Remote Sens. 39(12), 4254–4284 (2018)
Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2016)
Zhu, P., Abdal, R., Qin, Y., Wonka, P.: SEAN: image synthesis with semantic region-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5104–5113 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Leonhardt, J., Drees, L., Gall, J., Roscher, R. (2024). Leveraging Bioclimatic Context for Supervised and Self-supervised Land Cover Classification. In: Köthe, U., Rother, C. (eds) Pattern Recognition. DAGM GCPR 2023. Lecture Notes in Computer Science, vol 14264. Springer, Cham. https://doi.org/10.1007/978-3-031-54605-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-54605-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54604-4
Online ISBN: 978-3-031-54605-1
eBook Packages: Computer ScienceComputer Science (R0)