A novel few-shot learning based multi-modality fusion model for COVID-19 rumor detection from online social media

Bi-GRUs feature extractions

Recently, it is the mainstream to extract features from texts with deep neural networks. Representative RNNs-based models such as LSTMs and GRUs have shown effectiveness in the rumor detection task (Liu, Jin & Shen, 2019; Wang & Guo, 2020). In this paper, we apply bidirectional GRUs (Bi-GRUs) to extract features of source posts and corresponding comments. We take the post m as an example. After applying pretrained BERT embeddings, the input post m turns to the embeddings matrix $B^m=\left[b_1^m,b_2^m,\dots ,b_n^m\right]$, where n is the same definition of token numbers. The BiGRUs are applied upon the embedding matrix to further decode post m to textual hidden features denoted as $H^m=\left[h_1^m,h_2^m,\dots ,h_n^m\right]$. The general structure of Bi-GRUs is as Fig. 5 shows.

For the jth input embeddings $b_j^m$, the decoded features $h_j^m$ of Bi-GRUs’ outcome in one direction can be denoted as $h_j^m=GRUs\left(b_j^m,h_{j-1}^m\right)$. In Eqs. (1)–(4) we show its complete form in detail: (1)${\displaystyle r_j^m=\sigma \left(W_r{b}_j^m+{\beta }_r+W_{hr}{h}_{j-1}^m+{\beta }_{hr}\right)}$ (2)${\displaystyle z_j^m=\sigma \left(W_z{b}_j^m+{\beta }_z+W_{hz}{h}_{j-1}^m+{\beta }_{hz}\right)}$ (3)${\displaystyle h_j^{m^{\prime }}=tanh\left(W_h{b}_j^m+{\beta }_h+r_j^m\ast \left(W_{hh}{h}_{j-1}^m+{\beta }_{hh}\right)\right)}$ (4)${\displaystyle h_j^m=\left(1-z_j^m\right)\ast h_j^{m^{\prime }}+z_j^m\ast h_{j-1}^m}$ The hidden states of the forward input sequence with n tokens can be represented as $\overrightarrow{H^m}=GRUs\left(\overrightarrow{B^m},h_0\right)$, where h₀ is the initial hidden state. Similarly, the hidden states of the backward input sequence are represented as $\stackrel{⃖}{H^m}=GRUs\left(\stackrel{⃖}{B^m},h_0\right)$. The final hidden states of both directions are calculated as Eq. (5) shows, which are also regarded as the features for further rumor detection. (5)${\displaystyle H^m=\frac{\left(\overrightarrow{H^m}+\stackrel{⃖}{H^m}\right)}{2}}$ For the i-th microblog, the extracted feature of the post m_i is denoted as H^m_i, similarly, the extracted features of comments $\left[c_{i1},c_{i2},\dots ,c_{il}\right]$ are denoted as $H^{c_i}=\left[H_1^{c_i},\dots ,H_l^{c_i}\right]$.

Feature fusion layer

Taking the ith instance from Sina Weibo, for example, we consider both source post m_i and corresponding comments [c_i1, …, c_il] as two modalities. Because each instance may have more than one comment, the fusion layer contains two main steps. The first step is to fuse the features $\left[H_1^{c_i},\dots ,H_l^{c_i}\right]$ of all comments, denoted as H^c_i. The second step is to fuse the features of the source post and comments.

In the first fusion step, the features of all the l comments in the same instance are extracted via the same Bi-GRUs. As the comments of each microblog are embedded into the same feature space, it is natural to fuse them with the weighted sum of their features. We regard the contribution of each comment to be equal for the rumor detection task, which is defined in Eq. (6). (6)${\displaystyle H^{c_i}=\sum _{j=1}^l{H}_j^{c_i}}$ In the second fusion step, the features of the source post and comments in the same instance are extracted via two individual Bi-GRUs. Following the common practice, we fuse these multi-modal features together with concatenation to build the features of the microblogs. For the ith instance, the fused feature is defined as Hⁱ = [H^m_i; H^c_i].

Few-shot learning

Usually, rumors from online social media are usually produced according to certain events. Rumors of emergent events could be very distinct from events collected in the past, so that rumor detection models could barely generalize on new events. However, breaking events like COVID-19 are unprecedented so that very rare instances are available. This may result in the failure or overfitting to directly build a rumor detection model based on supervised learning with the lack of labeled training data for emergent events.

To tackle this challenge, we propose a few-shot learning paradigm by learning a generic model with labeled data from past observed events and adapting to unseen events with only a few labeled instances. We propose a meta-learning based strategy in learning the few-shot rumor detection tasks. The core idea is to sample a large number of task combinations in training instances so that the model can learn the transferable knowledge for unseen categories. State-of-the-art methods are optimization-based with the idea of training a good initialized model which could adapt to unseen categories with only a few gradient steps (Finn, Abbeel & Levine, 2017; Sun et al., 2019a; Sun et al., 2019b).

The learning target of few-shot rumor detection with meta-learning methods is to minimize the adaptation loss on unseen tasks during training. Given a batch of few-shot tasks 𝔅 = T₁, …, T_|𝔅|, the total loss 𝔏 is calculated as Eq. (7) shows. (7)${\displaystyle w^{\ast }=mi{n}_{w_m,w_c}\mathfrak{L}\left(w_m,w_c\right),s.t.\mathfrak{L}\left(w_m,w_c\right)=\frac{1}{\left|\mathfrak{B}\right|}\sum _{T\in \mathfrak{B}}{L}_T\left(w_m,w_c\right)}$ where w_m refers to the parameters of the defined Bi-GRUs for dealing with the modality of source posts and w_c refers to the parameters of the defined Bi-GRUs for the modality of comments. L_T is the loss of task T and w^∗ is the optimized model which can fast adapt to unseen events. This optimization problem can be solved iteratively with the steps as shown in Fig. 6, in order to train the models to adapt to sampled new tasks well. We will demonstrate each of the meta-learning steps in detail.

The COVID-19 rumor detection is defined as the N-way M-event K-shot Q-query few-shot learning task.

Step 1. Sampling: This step aims to sample a few-shot task T from E_p (events happened in the past). Each event has both rumor and non-rumor instances, which means the sampling times from E_p equal to N × M. For an N-way M-event K-shot Q-query few-shot learning task T, K rumor instances and K non-rumor instances are sampled from M events respectively to compose of a support set, denoted as $T^{\left(s\right)}.Q$ rumor and non-rumor instances are also sampled from the same M events respectively to compose of a query set, denoted as $T^{\left(q\right)}$. This step would sample N × M × (K + Q) instances for task T.

Step 2. Adaptation: This step aims to learn latent semantics in unseen categories by adapting the current model to the sampled task T in step 1. This step updates the model parameters w_m and w_c with the few-shot labeled data in $T^{\left(s\right)}$ by performing Stochastic Gradient Descent (SGD), as Eqs. (8) and (9) shows. (8)${\displaystyle w_m^{{}^{\prime }}=w_m-\alpha {\nabla }_{w_m}{L}_{T^{\left(s\right)}}\left(w_m\right)}$ (9)${\displaystyle w_c^{{}^{\prime }}=w_c-\alpha {\nabla }_{w_c}{L}_{T^{\left(s\right)}}\left(w_c\right)}$ where α is the step size of adaption, $w_m^{{}^{\prime }}$ and $w_c^{{}^{\prime }}$ are parameters of the adapted models, which can extract features of source posts and comments in the query set $T^{\left(q\right)}$ for further rumor detection.

Step 3. Optimization: This step aims to evaluate $w_m^{{}^{\prime }}$ and $w_c^{{}^{\prime }}$ with more samples in the query set $T^{\left(q\right)}$. The empirical loss functions are as Eqs. (10) and (11) show. (10)${\displaystyle L_T\left(w_m\right)=L_{T^{\left(q\right)}}\left(w_m^{{}^{\prime }}\right)=L_{T^{\left(q\right)}}\left(w_m-\alpha {\nabla }_{w_m}{L}_{T^{\left(s\right)}}\left(w_m\right)\right),}$ (11)${\displaystyle L_T\left(w_c\right)=L_{T^{\left(q\right)}}\left(w_c^{{}^{\prime }}\right)=L_{T^{\left(q\right)}}\left(w_c-\alpha {\nabla }_{w_c}{L}_{T^{\left(s\right)}}\left(w_c\right)\right).}$ To search for the optimal w_m and w_c defined in Eq. (7), we need to compute the Hessian. However, considering the tradeoff between the computational costs and performance, we solve this problem with just one gradient descent to approximate the parameter updates (Finn, Abbeel & Levine, 2017; Sung et al., 2018). (12)${\displaystyle w_m\leftarrow w_m-\gamma {\nabla }_{w_m}{L}_T\left(w_m\right),}$ (13)${\displaystyle w_c\leftarrow w_c-\gamma {\nabla }_{w_c}{L}_T\left(w_c\right),}$ where γ is the learning rate.

To detect rumors about suddenly happened events E_s, we can apply the parameters of the adapted models $w_m^{{}^{\prime }}$ and $w_c^{{}^{\prime }}$ to calculate the probability of the instances with the Sigmoid function.

Datasets for experiments

We carry on extensive empirical studies on two real-world datasets with user comments that have been classified as rumors or non-rumors. The first dataset is collected from Sina Weibo, which is used for detecting rumors about COVID-19. The other dataset we use is PHEME (Zubiaga, Liakata & Procter, 2016), which is publicly available and widely used in most rumor detection researches. Details of both datasets are as follows.

Weibo dataset

We collect microblogs that are written in Chinese from Sina Weibo—the largest online social media in China. In this dataset, there are 14 events with 3,840 instances in total. For each event, both rumors and non-rumors are included. Each event is a hot topic such as MH370, COVID-19 expert Zhong Nanshan, etc., which are widely discussed online. Each instance is recorded with a source post along with its comments. To evaluate the performance of few-shot learning on COVID-19 rumor detection, 11 COVID-19 irrelevant events are selected as the training and validation set, and three COVID-19 relevant events are used for testing (listed in Table 1).

PHEME dataset

This is a publicly available dataset (https://figshare.com/articles/PHEME_dataset_of_rumours_and_non-rumours/4010619) with tweets from Twitter in English, which is widely used for the evaluation of rumor detection tasks (Zubiaga, Liakata & Procter, 2017; Ma, Gao & Wong, 2019). It is collected according to five breaking events discussed on Twitter (Zubiaga, Liakata & Procter, 2016). Each instance is recorded with a source tweet along with its reply reactions. To evaluate the performance under the settings of few-shot learning, three breaking events that happened earlier are selected as the training and validation set (#Ferguson unrest, #Ottawa shooting, #Sydney siege), and the remaining two events that happened most recently are used for testing (#Charlie Hebdo shooting, #Germanwings plane crash). The pre-processing of the PHEME dataset follows the practice in previous work (Ma, Gao & Wong, 2019).

For the Weibo dataset, we crawl the comments of microblogs directly as they are readily available on the same webpage with the source posts. For the PHEME dataset, we regard the replies in the given dataset as comments. For the sake of generality, we randomly divide the dataset into training and validation sets according to distinct events, and repeat three times to form three different splits for robust cross-validation. We choose the number of splits as three for the following reasons. In few-shot learning, the number of splits depends on the number of new events in the test set. We take the 2-way 3-event 5-shot 9-query Weibo dataset for example. It has three COVID-19 relevant events to be detected with only a few labeled data. The number of the event in the definition is determined by the number of new events in the test dataset, so it is 3-event. The number of ways indicates the number of labels, which are rumor and non-rumor. With this definition, during the few-shot learning training process, every training epoch will sample three different events in the training set, for each event, five rumor instances and five non-rumor instances will be sampled for training. According to the few-shot learning setting, we guarantee that all events in the training sets should NOT appeared in the testing sets, and vice versa, to avoid the leakage of event information and guarantee that we are testing on complete novel events. We also assume that the number of events in the training set should be no less than the number of events in the test set to ensure the model capacity for adapting to new events. According to our assumption and task settings, we split our Weibo dataset to three events (COVID-19 relevant) for testing, and 11 events (COVID-19 irrelevant) for training. We fix the three events (COVID-19 relevant) for testing, and construct three folds for “cross-validation” over 11 training events (COVID-19 irrelevant) to guarantee that each fold has more than three events in the Weibo dataset. Table 2 displays the statistics of the data for experiments.