Wenjuan Gong
Skip Abstract Section
Abstract
Despite its wide applications in criminal investigations and clinical communications with patients suffering from autism, automatic micro-expression recognition remains a challenging problem because of the lack of training data and imbalanced classes problems. In this study, we proposed a meta-learning-based multi-model fusion network (Meta-MMFNet) to solve the existing problems. The proposed method is based on the metric-based meta-learning pipeline, which is specifically designed for few-shot learning and is suitable for model-level fusion. The frame difference and optical flow features were fused, deep features were extracted from the fused feature, and finally in the meta-learning-based framework, weighted sum model fusion method was applied for micro-expression classification. Meta-MMFNet achieved better results than state-of-the-art methods on four datasets. The code is available at https://github.com/wenjgong/meta-fusion-based-method.
- [1] . 2016. Low data drug discovery with one-shot learning. Retrieved from http://arxiv.org/abs/1611.03199.Google Scholar
- [2] . 2017. Multi-view 3D object detection network for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). IEEE Computer Society, 6526–6534. Google ScholarCross Ref
- [3] . 2020. A new meta-baseline for few-shot learning. Retrieved from https://arxiv.org/abs/2003.04390.Google Scholar
- [4] . 1995. Active shape models-their training and application. Comput. Vision Image Understand. 61, 1 (1995), 38–59.Google ScholarDigital Library
- [5] . Lie Catching and Microexpressions. The Philosophy of Deception. 118–133.Google Scholar
- [6] . 2017. Model-Agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML’17) (Proceedings of Machine Learning Research), and (Eds.), Vol. 70. PMLR, 1126–1135. Retrieved from http://proceedings.mlr.press/v70/finn17a.html.Google Scholar
- [7] . 2017. Model-Agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML’17). JMLR.org, 1126–1135.Google ScholarDigital Library
- [8] . 2019. OFF-ApexNet on micro-expression recognition system. Signal Process. Image Commun. 74 (2019), 129–139. Google ScholarDigital Library
- [9] . 2021. Transfer spatio-temporal knowledge from emotion-related tasks for facial expression spotting. In Proceedings of the 1st Workshop on Facial Micro-Expression: Advanced Techniques for Facial Expressions Generation and Spotting (FME’21), , , and (Eds.). ACM, 19–24. Google ScholarDigital Library
- [10] . 2022. Hyperspectral image reconstruction using multi-scale fusion learning. ACM Trans. Multim. Comput. Commun. Appl. 18, 1 (2022), 16:1–16:21. Google ScholarDigital Library
- [11] . 2019. Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect. Comput. 10, 3 (2019), 394–406. Google ScholarDigital Library
- [12] . 2016. Densely connected convolutional networks. Retrieved from http://arxiv.org/abs/1608.06993.Google Scholar
- [13] . 2019. Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans. Affect. Comput. 10, 1 (2019), 32–47. Google ScholarDigital Library
- [14] . 2016. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175 (2016), 564–578. Google ScholarDigital Library
- [15] . 2018. Enriched long-term recurrent convolutional network for facial micro-expression recognition. In Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG’18). IEEE Computer Society, 667–674. Google ScholarDigital Library
- [16] . 2016. Micro-Expression recognition with expression-state constrained spatio-temporal feature representations. In Proceedings of the ACM Conference on Multimedia Conference (MM’16), , , , , , , , and (Eds.). ACM, 382–386. Google ScholarDigital Library
- [17] . 2019. Micro-expression recognition based on 3D flow convolutional neural network. Pattern Anal. Appl. 22, 4 (2019), 1331–1339. Google ScholarDigital Library
- [18] . 2019. Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow. Multim. Tools Appl. 78, 20 (2019), 29307–29322. Google ScholarDigital Library
- [19] . 2013. A Spontaneous Micro-expression Database: Inducement, collection and baseline. In Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG’13). IEEE Computer Society, 1–6. Google ScholarCross Ref
- [20] . 2021. Joint local and global information learning with single Apex frame detection for micro-expression recognition. IEEE Trans. Image Process. 30 (2021), 249–263. Google ScholarDigital Library
- [21] . 2018. Micro-expression recognition based on spatiotemporal gabor filters. In Proceedings of the 8th International Conference on Information Science and Technology (ICIST’18). IEEE, 487–491.Google ScholarCross Ref
- [22] . 2016. Less is more: Micro-expression recognition from video using Apex frame. Retrieved from http://arxiv.org/abs/1606.01721.Google Scholar
- [23] . 2018. Less is more: Micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62 (2018), 82–92. Google ScholarCross Ref
- [24] . 2019. DARTS: Differentiable architecture search. In Proceedings of the 7th International Conference on Learning Representations (ICLR’19). Retrieved from https://openreview.net/forum?id=S1eYHoC5FX.Google Scholar
- [25] . 2019. A neural micro-expression recognizer. In Proceedings of the 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG’19). IEEE, 1–4. Google ScholarDigital Library
- [26] . 2018. Scene classification based on two-stage deep feature fusion. IEEE Geosci. Remote. Sens. Lett. 15, 2 (2018), 183–186. Google ScholarCross Ref
- [27] . 2018. Motion descriptors for micro-expression recognition. Signal Process. Image Commun. 67 (2018), 108–117. Google ScholarCross Ref
- [28] . 2014. A Delaunay-based temporal coding model for micro-expression recognition. In Proceedings of the Asian Conference on Computer Vision (ACCV’14) (Lecture Notes in Computer Science), and (Eds.), Vol. 9009. Springer, 698–711. Google ScholarCross Ref
- [29] . 2022. Structure-aware meta-fusion for image super-resolution. ACM Trans. Multim. Comput. Commun. Appl. 18, 2 (2022), 60:1–60:25. Google ScholarDigital Library
- [30] . 2019. Meta-learning update rules for unsupervised representation learning. In Proceedings of the 7th International Conference on Learning Representations (ICLR’19). Retrieved from https://openreview.net/forum?id=HkNDsiC9KQ.Google Scholar
- [31] . 2019. A deep learning system for recognizing facial expression in real-time. ACM Trans. Multim. Comput. Commun. Appl. 15, 2 (2019), 33:1–33:20. Google ScholarDigital Library
- [32] . 2000. How emotions work: The social functions of emotional expression in negotiations. Res. Organiz. Behav. 22 (2000), 1–50. Google ScholarCross Ref
- [33] . 2021. Micro-expression recognition based on the fusion between optical flow and dynamic image. In Proceedings of the 5th International Conference on Machine Learning and Soft Computing (ICMLSC’21). ACM, 115–120. Google ScholarDigital Library
- [34] . 2018. Micro-expression recognition based on local two-order gradient pattern. In Proceedings of the 1st Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia’18). 1–6. Google ScholarCross Ref
- [35] . 1994. Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. In Proceedings of the 12th IAPR International Conference on Pattern Recognition, Conference A: Computer Vision and Image Processing (ICPR’94). IEEE, 582–585. Google ScholarCross Ref
- [36] . 2021. Spatio-temporal convolutional attention network for spotting macro- and micro-expression intervals. In Proceedings of the 1st Workshop on Facial Micro-Expression: Advanced Techniques for Facial Expressions Generation and Spotting (FME’21), , , and (Eds.). ACM, 25–30. Google ScholarDigital Library
- [37] . 2019. A novel apex-time network for cross-dataset micro-expression recognition. In Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction (ACII’19). IEEE, 1–6. Google ScholarCross Ref
- [38] . 2018. From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG’18). IEEE Computer Society, 657–661. Google ScholarDigital Library
- [39] . 2019. CapsuleNet for micro-expression recognition. In Proceedings of the 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG’19). IEEE, 1–7. Google ScholarDigital Library
- [40] . 2019. Regularized evolution for image classifier architecture search. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI’19), the 31st Innovative Applications of Artificial Intelligence Conference (IAAI’19), and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI’19). AAAI Press, 4780–4789. Google ScholarDigital Library
- [41] . 2016. Meta-learning with memory-augmented neural networks. In Proceedings of the 33rd International Conference on Machine Learning (ICML’16) (JMLR Workshop and Conference Proceedings), and (Eds.), Vol. 48. JMLR.org, 1842–1850. Retrieved from http://proceedings.mlr.press/v48/santoro16.html.Google Scholar
- [42] . 2017. Prototypical networks for few-shot learning. In Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, , , , , , , and (Eds.). 4077–4087. Retrieved from https://proceedings.neurips.cc/paper/2017/hash/cb8da6767461f2812ae4290eac7cbc42-Abstract.html.Google Scholar
- [43] . 2019. Recognizing spontaneous micro-expression using a three-stream convolutional neural network. IEEE Access 7 (2019), 184537–184551. Google ScholarCross Ref
- [44] . 2014. A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vis. 106, 2 (2014), 115–137. Google ScholarDigital Library
- [45] . 2017. Image-based facial micro-expression recognition using deep learning on small datasets. In Proceedings of the International Conference on Digital Image Computing: Techniques and Applications (DICTA’17). IEEE, 1–7. Google ScholarCross Ref
- [46] . 1998. Learning to learn: introduction and overview. In Learning to Learn, and (Eds.). Springer, 3–17. Google ScholarCross Ref
- [47] . 2020. Posed and spontaneous expression distinction using latent regression Bayesian networks. ACM Trans. Multim. Comput. Commun. Appl. 16, 3 (2020), 80:1–80:18. Google ScholarDigital Library
- [48] . 2014. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition. In Proceedings of the 12th Asian Conference on Computer Vision (ACCV’14) (Lecture Notes in Computer Science), , , , and (Eds.), Vol. 9003. Springer, 525–537. Google ScholarCross Ref
- [49] . 2022. Multi-feature fusion VoteNet for 3D object detection. ACM Trans. Multim. Comput. Commun. Appl. 18, 1 (2022), 6:1–6:17. Google ScholarDigital Library
- [50] . 2014. The faces of engagement: Automatic recognition of student engagementfrom facial expressions. IEEE Trans. Affect. Comput. 5, 1 (2014), 86–98. Google ScholarCross Ref
- [51] . 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31, 4 (2012), 65:1–65:8. Google ScholarDigital Library
- [52] . 2018. Spontaneous facial micro-expression recognition via deep convolutional network. In Proceedings of the 8th International Conference on Image Processing Theory, Tools and Applications (IPTA’18). IEEE, 1–6. Google ScholarCross Ref
- [53] . 2020. Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans. Multim. 22, 3 (2020), 626–640. Google ScholarCross Ref
- [54] . 2017. Microexpression identification and categorization using a facial dynamics map. IEEE Trans. Affect. Comput. 8, 2 (2017), 254–267. Google ScholarDigital Library
- [55] . 2013. CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG’13). IEEE Computer Society, 1–7. Google ScholarCross Ref
- [56] . 2014. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9, 1 (2014), e86041.Google ScholarCross Ref
- [57] . 2007. A duality-based approach for realtime TV-L\({}^{\mbox{1}}\) optical flow. In Proceedings of the 29th DAGM Symposium on Pattern Recognition (Lecture Notes in Computer Science), , , and (Eds.), Vol. 4713. Springer, 214–223. Google ScholarCross Ref
- [58] . 2022. Rectified meta-learning from noisy labels for robust image-based plant disease classification. ACM Trans. Multim. Comput. Commun. Appl. 18, 1s (2022), 30:1–30:17. Google ScholarDigital Library
- [59] . 2007. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29, 6 (2007), 915–928. Google ScholarDigital Library
- [60] . 2021. Feature refinement: An expression-specific feature learning and fusion method for micro-expression recognition. Retrieved from https://arxiv.org/abs/2101.04838.Google Scholar
- [61] . 2018. Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Trans. Multim. 20, 11 (2018), 3160–3172. Google ScholarDigital Library
Index Terms
- Meta-MMFNet: Meta-learning-based Multi-model Fusion Network for Micro-expression Recognition
Recommendations
Learning two groups of discriminative features for micro-expression recognition
Highlights- This work is the first one to learn discriminative features from two groups of features, and two groups of features have some differences, such as ...
AbstractAs a branch of affective computing and machine learning, recognizing micro-expressions is more difficult than recognizing macro-expressions because micro-expression has a small motion and short duration. A large number of features and ...
Micro-expression Recognition based on Bimodal Contrastive Learning
ICIGP '22: Proceedings of the 2022 5th International Conference on Image and Graphics ProcessingMicro-expressions are brief, subtle and involuntary facial gestures, which usually hide real emotions of people and are difficult to capture. Considering that micro-expressions data samples contain distinctive features of specific categories and common ...
2D facial expression recognition via 3D reconstruction and feature fusion
This paper proposed a method for facial expression recognition.In proposed method, facial depth has been added to facial texture for feature extraction.We demonstrated that adding the facial depth to feature extraction is effective.The 3DH-LLBP is ...
Comments