Fusion Network Based on Motion Learning and Image Feature Representation for Micro-Expression Recognition

Abstract

Micro-expression is usually generated by subtle movements of facial muscles when people control their true emotions. Micro-expression recognition aims to help people understand micro-expression by feature extraction and representation. However, most existing methods only use motion information or RGB image information to extract features, which ignore the feature fusion between these two types of information. To address this issue, we propose a novel fusion network based on motion learning and image feature representation with a hierarchical structure, including three components: a motion learning stream, an image feature representation stream and a heterogeneous information fusion mechanism. Firstly, the motion learning stream extracts multi-scale facial muscle motion features associated with micro-expression from inter-frame motion information. Secondly, the image feature representation stream divides the apex image into four parts based on the intensity of muscle movement and gradually extracts the local and global facial detail and semantic features. Finally, the heterogeneous information fusion mechanism is proposed to perform information interaction between motion features and image features and to integrate categorical information. Specifically, the cross-fusion module is designed to fuse features extracted from the two streams to produce a more comprehensive, richer representation where image features and motion features influence and enhance each other. The result integration strategy considers the results of the two streams as the classification result of the model, further integrating different feature information. Extensive experiments conducted on spontaneous micro-expression datasets show that the proposed model achieves the best performance compared with the existing state-of-the-art methods.