A Transformer Based Multimodal Fine-Fusion Model for False Information Detection

Abstract

With the development and the popularity of internet, WWW is becoming the main platform of exchanging information and accessing information for people. For the openness and free, the information published on it may be not true. The spread of false information has a serious impact on social stability and detecting false information has become an urgent task. Text and image are two main information modals and some work proposed to combine them to enhance the detection accuracy. However, in almost these work, the granularity of information fusion is not refined enough to get the complete representation from multimodal information at the same time. Inspired by this, we propose a Multimodal, fine-grained Fusion false information detection model based on Transformer,namely TMF. First, a feature extraction module is used to extract the representation of each word in the text and the representation of each region in the image, respectively, to obtain a multi-modal, fine-grained representation. Then, a Transformer based multimodal fusion module, Transformer-MF, is designed to learn the interaction between words and words, words and regions, and regions and regions simultaneously from both global and local levels, and obtain global and local multimodal representations. Finally, raw representations of text and images are combined with global and local multi-modal representations for false information detection. Experimental results show that our model performs better than the baseline models in terms of false information detection accuracy.