Abstract
Discovering potential concepts and events by analyzing Earth Observation (EO) data may be supported by fusing other distributed data sources such as non-EO data, for instance, in-situ citizen observations from social media. The retrieval of relevant information based on a target query or event is critical for operational purposes, for example, to monitor flood events in urban areas, and crop monitoring for food security scenarios. To that end, we propose an early-fusion (low-level features) and late-fusion (high-level concepts) mechanism that combines the results of two EU-funded projects for information retrieval in Sentinel imagery and social media data sources. In the early fusion part, the model is based on active learning that effectively merges Sentinel-1 and Sentinel-2 bands, and assists users to extract patterns. On the other hand, the late fusion mechanism exploits the context of other geo-referenced data such as social media retrieval, to further enrich the list of retrieved Sentinel image patches. Quantitative and qualitative results show the effectiveness of our proposed approach.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
References
Andreadis, S., Bakratsas, M., Giannakeris, P., et al.: Multimedia analysis techniques for flood detection using images, articles and satellite imagery. In: Working Notes Proceedings of the MediaEval 2019 Workshop, Sophia Antipolis, France, 27–30 October 2019. CEUR Workshop Proceedings, vol. 2670. CEUR-WS.org (2019)
Atrey, P.K., Hossain, M.A., El Saddik, A., Kankanhalli, M.S.: Multimodal fusion for multimedia analysis: a survey. Multimed. Syst. 16(6), 345–379 (2010)
Blanchart, P., Ferecatu, M., Cui, S., Datcu, M., et al.: Pattern retrieval in large image databases using multiscale coarse-to-fine cascaded active learning. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 7(4), 1127–1141 (2014)
Chaabouni-Chouayakh, H., Datcu, M.: Backscattering and statistical information fusion for urban area mapping using TerraSAR-X data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 3(4), 718–730 (2010)
Cui, S., Dumitru, C.O., Datcu, M.: Ratio-detector-based feature extraction for very high resolution SAR image patch indexing. IEEE Geosci. Remote Sens. Lett. 10(5), 1175–1179 (2013)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)
Dumitru, C.O., Schwarz, G., Datcu, M.: Monitoring of coastal environments using data mining. In: Knowledge Extraction and Semantic Annotation (KESA 2018), pp. 34–39, April 2018
Gialampoukidis, I., Moumtzidou, A., Bakratsas, M., Vrochidis, S., Kompatsiaris, I.: A multimodal tensor-based late fusion approach for satellite image search in sentinel 2 images. In: Lokoč, J., et al. (eds.) MMM 2021. LNCS, vol. 12573, pp. 294–306. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67835-7_25
Gialampoukidis, I., Moumtzidou, A., Liparas, D., et al.: Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression. Multimed. Tools Appl. 76(21), 22383–22403 (2017)
Jegou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 117–128 (2010)
Kitanovski, I., Strezoski, G., Dimitrovski, I., Madjarov, G., Loskovska, S.: Multimodal medical image retrieval system. Multimed. Tools Appl. 76(2), 2955–2978 (2016). https://doi.org/10.1007/s11042-016-3261-1
Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNS-CRF. arXiv preprint arXiv:1603.01354 (2016)
Mantsis, D.F., Bakratsas, M., Andreadis, S., et al.: Multimodal fusion of sentinel 1 images and social media data for snow depth estimation. IEEE Geosci. Remote Sens. Lett. (2020)
Andreadis, S., et al.: VERGE in VBS 2019. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, W.-H., Vrochidis, S. (eds.) MMM 2019. LNCS, vol. 11296, pp. 602–608. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05716-9_53
Palubinsks, G., Datcu, M.: Information fusion approach for the data classification: an example for ERS-1/2 InSAR data. Int. J. Remote Sens. 29(16), 4689–4703 (2008)
Pittaras, N., Markatopoulou, F., Mezaris, V., Patras, I.: Comparison of fine-tuning and extension strategies for deep convolutional neural networks. In: Amsaleg, L., Guðmundsson, G., Gurrin, C., Jónsson, B., Satoh, S. (eds.) MMM 2017. LNCS, vol. 10132, pp. 102–114. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-51811-4_9
Yao, W., Dumitru, C.O., Datcu, M.: D2.8 data fusion v2, deliverable of the candela project. https://www.candela-h2020.eu/content/data-fusion-v2
Yao, W., Dumitru, C.O., Lorenzo, J., Datcu, M.: Data fusion on the candela cloud platform. In: European Geosciences Union (EGU) General Assembly - Big Data and Machine Learning in Geosciences, May 2020
Younessian, E., Mitamura, T., Hauptmann, A.: Multimodal knowledge-based analysis in multimedia event detection. In: Proceedings of the 2nd ACM International Conference on Multimedia Retrieval, pp. 1–8 (2012)
Acknowledgements
This work has been supported by the EC-funded projects CANDELA (H2020-776193) and EOPEN (H2020-776019), and partly by the ASD HGF project. The content of this paper (DLR part) is mainly based on the results presented in the CANDELA Deliverable D2.8 [17].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Yao, W. et al. (2021). Early and Late Fusion of Multiple Modalities in Sentinel Imagery and Social Media Retrieval. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12667. Springer, Cham. https://doi.org/10.1007/978-3-030-68787-8_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-68787-8_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68786-1
Online ISBN: 978-3-030-68787-8
eBook Packages: Computer ScienceComputer Science (R0)