skip to main content
10.1145/3605098.3636155acmconferencesArticle/Chapter ViewAbstractPublication PagessacConference Proceedingsconference-collections
poster

Multimodal Fusion of Heterogeneous Representations for Anomaly Classification in Satellite Imagery

Published: 21 May 2024 Publication History

Abstract

This study introduces a multimodal approach designed to detect anomaly in satellite imagery. The approach is comprised of three main components: the image processing models (ResNet and Regulated Network), the text processing models (BERT and a graph model), and a combination model that acts as a classifier, seamlessly integrating features extracted from both image and text models. Our work utilizes the SpaceNet8 Challenge dataset, focusing on regions in Louisiana USA, damaged by Hurricane Ida in 2021, and the Ahrweiler district in Germany, impacted by flooding in Western Europe during the same year. The experimental results demonstrate significant performance improvements when integrating image processing models and text processing models, surpassing the baseline CNN models. This study offers insights for future work in anomaly detection and semantic segmentation, emphasizing the effectiveness of a multimodal approach with heterogeneous data.

References

[1]
Rui Ba, Chen Chen, Jing Yuan, Weiguo Song, and Siuming Lo. 2019. SmokeNet: Satellite Smoke Scene Detection Using Convolutional Neural Network with Spatial and Channel-Wise Attention. Remote Sensing 11, 14 (2019).
[2]
Yongqiang Cao, Mengran Wang, Jiaqi Yao, Fan Mo, Hong Zhu, Liuru Hu, and Haoran Zhai. 2023. Stereoscopic Monitoring Methods for Flood Disasters Based on ICESat-2 and Sentinel-2 Data. Remote Sensing 15, 12 (2023).
[3]
Jiahui Chen, Yi Yang, Ling Peng, Luanjie Chen, and Xingtong Ge. 2022. Knowledge Graph Representation Learning-Based Forest Fire Prediction. Remote Sensing 14, 17 (2022).
[4]
Mihai Coca and Mihai Datcu. 2023. FPGA Accelerator for Meta-Recognition Anomaly Detection: Case of Burned Area Detection. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 16 (2023), 5247--5259.
[5]
Wei Cui, Yuanjie Hao, Xing Xu, Zhanyun Feng, Huilin Zhao, Cong Xia, and Jin Wang. 2022. Remote Sensing Scene Graph and Knowledge Graph Matching with Parallel Walking Algorithm. Remote Sensing 14, 19 (2022).
[6]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations. https://openreview.net/forum?id=YicbFdNTTy
[7]
Xuejie Hao, Zheng Ji, Xiuhong Li, Lizeyan Yin, Lu Liu, Meiying Sun, Qiang Liu, and Rongjin Yang. 2021. Construction and Application of a Knowledge Graph. Remote Sensing 13, 13 (2021).
[8]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770--778.
[9]
Ronny Hänsch, Jacob Arndt, Dalton Lunga, Matthew Gibb, Tyler Pedelose, Arnold Boedihardjo, Desiree Petrie, and Todd M. Bacastow. 2022. SpaceNet 8 - The Detection of Flooded Roads and Buildings. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1471--1479.
[10]
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, Los Alamitos, CA, USA, 9992--10002.
[11]
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022. A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11966--11976.
[12]
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. 2020. Designing Network Design Spaces. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10425--10433.
[13]
Bing Xu, Chunju Zhang, Wencong Liu, Jianwei Huang, Yujiao Su, Yucheng Yang, Weijie Jiang, and Wenhao Sun. 2023. Landslide Identification Method Based on the FKGRNet Model for Remote Sensing Images. Remote Sensing 15, 13 (2023).

Index Terms

  1. Multimodal Fusion of Heterogeneous Representations for Anomaly Classification in Satellite Imagery

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SAC '24: Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing
    April 2024
    1898 pages
    ISBN:9798400702433
    DOI:10.1145/3605098
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 21 May 2024

    Check for updates

    Author Tags

    1. multimodal
    2. anomaly classification
    3. knowledge graph
    4. satellite imagery

    Qualifiers

    • Poster

    Conference

    SAC '24
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

    Upcoming Conference

    SAC '25
    The 40th ACM/SIGAPP Symposium on Applied Computing
    March 31 - April 4, 2025
    Catania , Italy

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 28
      Total Downloads
    • Downloads (Last 12 months)28
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 20 Jan 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media