A transformer-encoder-based multimodal multi-attention fusion network for sentiment analysis

Abstract

Feature fusion for multimodal sentiment analysis is a challenging but worthwhile research topic. With the extension of the time dimension, there are interactions between multimodal signals and the lack of control over the target modal representations during the fusion process leads to erroneous shifts of vectors in the feature space. Moreover, ignoring the representation of target modal features under different fusion orders may lead to insufficient fusion of complementary information. To address the above issues, this paper proposes a transformer-encoder-based multimodal multi-attention fusion network model. The model constructs a multi-attention fusion transformer-encoder to learn inter-modal consistent features and enhance the representation of target modal features. Meanwhile, for each target modality, we construct multi-attention fusion transformer-encoder with different fusion orders in the model to capture the complementary features among the sequences with different fusion orders. Then, the three target modal representations containing consistent features and complementary features are fused with initial features through residual connections to guide the final sentiment analysis. We conduct extensive experiments on three public multimodal datasets. The results show that our approach outperforms the compared multimodal sentiment analysis methods on most metrics and can explain the contributions of inter- and intra-modal interactions in multiple modalities.

Appendix A: Positional embedding

Compared with the temporally related models such as CNN, RNN, etc., transformer completely employs the mechanism of multi-head attention and ignores the temporal factors in the sequence. Therefore, for the different orders of the same temporal sequence, transformer produces the same result. To address this issue, consistent with [17], we similarly embed the position information in the temporal sequence. Specifically, for the temporal sequence ${\textbf {X}}\in \mathbb {R}^{T\times {d}}$, we use the sin and cos functions to encode the position information, and the frequency of the functions is determined by the feature index. The encoding process is shown in (A1) and (A2):

$$\begin{aligned} PE[pos,2j]= & {} sin(\frac{pos}{10000^{\frac{2j}{d}}}) \end{aligned}$$

(A1)

$$\begin{aligned} PE[pos,2j+1]= & {} cos(\frac{pos}{10000^{\frac{2j}{d}}}) \end{aligned}$$

(A2)

where $pos=1,2,....,T$ is the position in the temporal sequence and $j=0,\lfloor {\frac{d}{2}}\rfloor$ is the dimension corresponding to d. Therefore, the positional encoding generated by positional embedding shows a sinusoidal character. During the intra-modal feature extraction and interaction and inter-modal multi-attention interaction stages, positional embedding information can be added to the sequence by summing the encoded features with the temporal sequence via (9) and (12).