Skip to main content
SpringerLink
Log in

Unsupervised cross-database micro-expression recognition based on distribution adaptation

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Different from the traditional macro-expressions, micro-expressions are unconscious, quick and trustworthy facial expressions, which can reveal real emotion. Micro-expressions can provide information that is important and crucial in applications such as lie detection, criminal investigation, pain or mood assessment, etc. However, it is worth noting that most current micro-expression recognition methods rely only on a single micro-expression database. If the training and test samples belong to different domains, for example, different micro-expression databases, the accuracy of existing micro-expression recognition methods will decrease dramatically. To solve this problem, we propose an unsupervised cross-database micro-expression recognition method based on distribution adaptation. Compared with most advanced unsupervised cross-database recognition methods, the proposed method has better performance on micro-expression recognition tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Freitas-Magalhães, A.: The Psychology of Emotions—The Allure of Human Face. Leya, Lisbon (2020)

    Google Scholar 

  2. Corneanu, C.A., Simón, M.O., Cohn, J.F., et al.: Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans. Pattern Anal. Mach. Intell. 38(8), 1548–1568 (2016)

    Article  Google Scholar 

  3. Ekman, P.: Emotions revealed: recognizing faces and feelings to improve communication and emotional life. Holt Paperb. 128(8), 140–140 (2003)

    Google Scholar 

  4. Yan, W.J., Wu, Q., Liang, J., et al.: How fast are the leaked facial expressions: the duration of micro-expressions. J. Verbal Behav. 37(4), 217–230 (2013)

    Google Scholar 

  5. Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry Interpers. Biol. Process. 32(1), 88–106 (1969)

    Article  Google Scholar 

  6. Bhushan, B.: Study of Facial Micro-Expressions in Psychology. Springer, Berlin (2015)

    Book  Google Scholar 

  7. Zhao, S., Yao, H., Gao, Y., et al.: Predicting personalized image emotion perceptions in social networks. IEEE Trans. Affect. Comput. 9(4), 526–540 (2016)

    Article  Google Scholar 

  8. Zhao, S., Gao, Y., Ding, G., et al.: Real-time multimedia social event detection in microblog. IEEE Trans. Cybern. 48(11), 3218–3231 (2017)

    Article  Google Scholar 

  9. Li, M., Chen, L., Wei, W., Ben, X., Wang, D.: An improved generative adversarial network for micro-expressions based on multi-label learning from action units. In: The 4th International Conference on Image and Graphics Processing, pp. 59–64. https://doi.org/10.1145/3447587.3447596 (2021)

  10. Ekman, P.: Telling lies: Clues to deceit in the marketplace, politics, and marriage, revised WW Norton & Company, New York (2009)

    Google Scholar 

  11. Ben, X., Ren, Y., Zhang, J., et al.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://doi.org/10.1109/TPAMI.2021.3067464

    Article  Google Scholar 

  12. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)

    Article  Google Scholar 

  13. Ben, X., Zhang, P., Yan, R., et al.: Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput. Appl. 27(8), 2629–2646 (2016)

    Article  Google Scholar 

  14. Huang, T., Chen, L., Feng, Y., et al.: A multiview representation framework for micro-expression recognition. IEEE Access 7, 120670–120680 (2019)

    Article  Google Scholar 

  15. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29, 915–928 (2007)

    Article  Google Scholar 

  16. Wang, Y., See, J., Phan, R.C.W., et al.: LBP with six intersection points: reducing redundant information in LBP-top for micro-expression recognition. In: Asian Conference on Computer Vision, pp. 525–537 (2014)

  17. Ben, X., Jia, X., Yan, R., et al.: Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recogn. Lett. 107, 50–58 (2017)

    Article  Google Scholar 

  18. Zhang, P., Ben, X., Yan, R., et al.: Micro-expression recognition system. Optik 127(3), 1395–1400 (2016)

    Article  Google Scholar 

  19. Liu, Y.J., Zhang, J.K., Yan, W.J., et al.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2015)

    Article  Google Scholar 

  20. Xie, H.X., Lo, L., Shuai, H.H, et al.: AU-assisted graph attention convolutional network for micro-expression recognition. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2871–2880 (2020)

  21. Peng, M., Wang, C., Chen, T., et al.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745–1756 (2017)

    Article  Google Scholar 

  22. Zhu, X., Ben, X., Liu, S., et al.: Coupled source domain targetized with updating tag vectors for micro-expression recognition. Multimed. Tools Appl. 77(3), 3105–3124 (2018)

    Article  Google Scholar 

  23. Jia, X., Ben, X., Yuan, H., et al.: Macro-to-micro transformation model for micro-expression recognition. J. Comput. Sci. 25, 289–297 (2018)

    Article  Google Scholar 

  24. Zong Y, Huang X, Zheng W, et al. (2017) Learning a target sample re-generator for cross-database micro-expression recognition. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 872–880

  25. Zong, Y., Zheng, W., Cui, Z., et al.: Toward bridging micro expressions from different domains. IEEE Trans. Cybern. 50(12), 5047–5060 (2019)

    Article  Google Scholar 

  26. Peng, M., Wu, Z., Zhang, Z., et al.: From macro to micro expression recognition: deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 657–661 (2018)

  27. Deng, J., Dong, W., Socher, R., et al.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

  28. Pan, S.J., Tsang, I.W., Kwok, J.T., et al.: Domain adaptation via transfer component analysis. IEEE Trans. Neural Netw. 22(2), 199–210 (2010)

    Article  Google Scholar 

  29. Long, M., Wang, J., Ding, G., et al.: Transfer feature learning with joint distribution adaptation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2200–2207 (2013)

  30. Wang, J., Feng, W., Chen, Y., et al.: Visual domain adaptation with manifold embedded distribution alignment. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 402–410 (2018)

  31. Gong, B., Grauman, K., Sha, F.: Connecting the dots with landmarks: discriminatively learning domain-invariant features for unsupervised domain adaptation. In: International Conference on Machine Learning, pp. 222–230 (2013)

  32. Zhang, L., Chen, C., Bu, J., et al.: Active learning based on locally linear reconstruction. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 2026–2038 (2011)

  33. Wang, J., Chen, Y., Hao, S., et al.: Balanced distribution adaptation for transfer learning. In: 2017 IEEE International Conference on Data Mining, pp. 1129–1134 (2017)

  34. Wang, K., He, R., Wang, L., et al.: Joint feature selection and subspace learning for cross-modal retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 2010–2023 (2016)

    Article  Google Scholar 

  35. Lucey, P., Cohn, J.F., Kanade. T., et al.: The extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Computer Vision and Pattern Recognition Workshops pp. 94–101 (2010)

  36. Yan, W.J., Li, X., Wang, S.J., et al.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)

    Article  Google Scholar 

  37. Davison, A.K., Lansley, C., Costen, N., et al.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)

    Article  Google Scholar 

  38. Li, X., Pfister, T., Huang, X., et al.: A Spontaneous Micro-expression Database: inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6 (2013)

  39. Qu, F., Wang, S.J., Yan, W.J., et al.: CAS (ME)2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans. Affect. Comput. 9(4), 424–436 (2017)

    Article  Google Scholar 

  40. Chen, D., Ren, S., Wei, Y., et al.: Joint cascade face detection and alignment. In: European Conference on Computer Vision, pp. 109–122 (2014)

  41. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  42. Li, X., Fang, M., Wang, H., et al.: Supervised transfer kernel sparse coding for image classification. Pattern Recogn. Lett. 68, 27–33 (2015)

    Article  Google Scholar 

  43. Long, M., Wang, J., Ding, G., et al.: Transfer joint matching for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1410–1417 (2014)

  44. Long, M., Wang, J., Sun, J., et al.: Domain invariant transfer kernel learning. IEEE Trans. Knowl. Data Eng. 27(6), 1519–1532 (2014)

    Article  Google Scholar 

  45. Wang, S., Zhang, L., Zuo, W., et al.: Class-specific reconstruction transfer learning for visual recognition across domains. IEEE Trans. Image Process. 29, 2424–2438 (2019)

    Article  Google Scholar 

  46. Zhang, L., Wang, S., Huang, G.B., et al.: Manifold criterion guided transfer learning via intermediate domain generation. IEEE Trans. Neural Netw. Learn. Syst. 30(12), 3759–3773 (2019)

    Article  MathSciNet  Google Scholar 

  47. Zhou, T., Fu, H., Gong, C., et al.: Multi-mutual consistency induced transfer subspace learning for human motion segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10277–10286

  48. Huang, Z., Xue, C., Han, B., et al.: Universal semi-supervised learning. In: Annual Conference on Neural Information Processing Systems (2021)

Download references

Acknowledgements

This work was supported in part by the Natural Science Foundation of China under Grant 61571275, 61971468, and in part by the Shandong Provincial Key Research and Development Program (Major Scientific and Technological Innovation Project) under Grant 2019JZZY010119.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Shandong University, Qingdao, People’s Republic of China

    Bing Li, Ying Zhou, Ruixue Xiao, Jianchao Wang, Xianye Ben & Hongchao Zhou

  2. IETR CNRS UMR 6164, Institut National Des Sciences Appliquées de Rennes, 35708, Rennes, France

    Kidiyo Kpalma

Authors
  1. Bing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ruixue Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianchao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xianye Ben
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kidiyo Kpalma
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hongchao Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xianye Ben or Hongchao Zhou.

Additional information

Communicated by X. Yang

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, B., Zhou, Y., Xiao, R. et al. Unsupervised cross-database micro-expression recognition based on distribution adaptation. Multimedia Systems 28, 1099–1116 (2022). https://doi.org/10.1007/s00530-022-00896-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00896-9

Keywords

Search

Navigation