Skip to main content
SpringerLink
Log in

Micro-expression Recognition Using Motion Magnification and Spatiotemporal Texture Map

  • Conference paper
  • First Online:
Proceedings of the 13th International Conference on Ubiquitous Information Management and Communication (IMCOM) 2019 (IMCOM 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 935))

Abstract

Micro-expressions are short-lived, rapid facial expressions exhibited by individuals when they are in high stakes situations. Studying these micro-expressions is important as these cannot be controlled by an individual and hence offer a peek into what the individual is actually feeling and thinking as opposed to what he/she is trying to portray. The spotting and recognition of micro-expressions has vital applications in many fields including criminal investigation, psychotherapy, and education. However, due to their short-lived, rapid nature, spotting, recognizing and classifying micro-expressions is a major challenge. In this paper, a highly effective hybrid approach is proposed, which utilizes both Eulerian Video Magnification for motion magnification and Spatiotemporal Texture Map for feature extraction. The experiments are carried out on CASME II, the only spontaneous micro-expression dataset currently available. The proposed approach uses Support Vector Machines (SVM) classifier with kernels, and achieves an accuracy of 80%, viz. an increase by 5% comparing with the best existing method using SVM on CASME II. This work is a significant step for recognizing micro-expressions; the proposed method would be widely applicable to facial sentiment recognition in both images and videos.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Dhall, A.: Collecting large, richly annotated facial-expression databases from movies (2012)

    Article  Google Scholar 

  2. Ekman, P.: Darwin, deception, and facial expression. In: Anonymous 2003. https://doi.org/10.1196/annals.1280.010

    Article  Google Scholar 

  3. Ekman, P., O’Sullivan, M., Frank, M.G.: A few can catch a liar. Psychol. Sci. 10(3), 263–266 (1999). https://doi.org/10.1111/1467-9280.00147

    Article  Google Scholar 

  4. Förstner, W., Gülch, E.: A fast operator for detection and precise location of distinct points, corners and centres of circular features. In: Proceedings of ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data (1987)

    Google Scholar 

  5. Frank, M., et al.: I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: The Annual Meeting of the International Communication Association. Sheraton New York, New York City (2009)

    Google Scholar 

  6. Harris, C., Stephens, M.: A combined corner and edge detector. In: Alvey Vision Conference (1988)

    Google Scholar 

  7. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (2000)

    Google Scholar 

  8. Kienzle, W., et al.: Face detection - efficient and rank deficient, vol. 17 (2005)

    Google Scholar 

  9. Liu, C.: Beyond pixels: exploring new representations and applications for motion analysis (2009)

    Google Scholar 

  10. Lucey, P., et al.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2010)

    Google Scholar 

  11. Lyons, M.J., et al.: The Japanese Female Facial Expression (JAFFE) database. In: Proceedings of Third International Conference on Automatic Face and Gesture Recognition (1998)

    Google Scholar 

  12. Lyons, M., et al.: Coding facial expressions with Gabor wavelets. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition (1998). https://doi.org/10.1109/afgr.1998.670949

  13. Matsumoto, D., Hwang, H.S.: Evidence for training the ability to read microexpressions of emotion. Motiv. Emot. 35(2), 181–191 (2011). https://doi.org/10.1007/s11031-011-9212-2

    Article  Google Scholar 

  14. Mehrabian, A.: Communication without words. In: Communication Theory, pp. 193–200 (2008)

    Chapter  Google Scholar 

  15. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)

    Article  Google Scholar 

  16. Pantic, M., et al.: Web-based database for facial expression analysis. In: IEEE International Conference on Multimedia and Expo, ICME 2005 (2005)

    Google Scholar 

  17. Russell, T.A., Chu, E., Phillips, M.L.: A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool. Br. J. Clin. Psychol. 45(4), 579–583 (2006)

    Article  Google Scholar 

  18. Kamarol, S.K.A., et al.: Spatiotemporal feature extraction for facial expression recognition. IET Image Proc. 10(7), 534–541 (2016). https://doi.org/10.1049/iet-ipr.2015.0519

    Article  Google Scholar 

  19. Sandbach, G., et al.: Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis. Comput. 30(10), 683–697 (2012). https://doi.org/10.1016/j.imavis.2012.06.005. http://www.sciencedirect.com/science/article/pii/S0262885612000935

    Article  Google Scholar 

  20. Schopler, E., Mesibov, G.B., Kunce, L.J.: Asperger syndrome or high-functioning autism? (1998)

    Google Scholar 

  21. Tomasi, C., Kanade, T.: Detection and tracking of point features (1991)

    Google Scholar 

  22. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vision 57(2), 137–154 (2004)

    Article  Google Scholar 

  23. Wang, Y., et al.: Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PLoS ONE 10(5), e0124674 (2015)

    Article  Google Scholar 

  24. Wang, Y., et al.: Effective recognition of facial micro-expressions with video motion magnification. Multimed. Tools Appl. 76(20), 21665–21690 (2017). https://doi.org/10.1007/s11042-016-4079-6

    Article  Google Scholar 

  25. Wu, H., et al.: Eulerian Video Magnification for Revealing Subtle Changes in the World. ACM Trans. Graph. 31(4), 65:1–65:8 (2012). https://doi.org/10.1145/2185520.2185561. http://doi.acm.org/10.1145/2185520.2185561

    Article  Google Scholar 

  26. Wu, Q., Shen, X., Fu, X.: The machine knows what you are hiding: an automatic micro-expression recognition system. In: Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction - Volume Part II (2011). http://dl.acm.org/citation.cfm?id=2062850.2062867

    Chapter  Google Scholar 

  27. Li, X., et al.: A spontaneous micro-expression database: Inducement, collection and baseline. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (2013). https://doi.org/10.1109/fg.2013.6553717

  28. Li, X., et al.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. PP(99), 1 (2017). https://doi.org/10.1109/taffc.2017.2667642

    Article  Google Scholar 

  29. Yan, W., et al.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)

    Article  Google Scholar 

  30. Yan, W., et al.: How fast are the leaked facial expressions: the duration of micro-expressions. J. Nonverbal Behav. 37(4), 217–230 (2013). https://doi.org/10.1007/s10919-013-0159-8

    Article  Google Scholar 

  31. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  32. Matsumoto, D., Skinner and, L.G., Hwang, H.: Reading people: behavioral anomalies and investigative interviewing. The FBI Law Enforcement Bulletin (2015)

    Google Scholar 

  33. Handeyside, H.: Be careful with your face at airports (opinion) – CNN. CNN. https://www.cnn.com/2015/03/19/opinions/handeyside-tsa-spot-program/index.html. Accessed 5 May 2018

  34. Hammal, Z., Couvreur, L., Caplier, A., Rombaut, M.: Facial expression classification: an approach based on the fusion of facial deformations using the transferable belief model. Int. J. Approximate Reasoning 46(3), 542–567 (2007)

    Article  Google Scholar 

  35. Eleftheriadis, S., Rudovic, O., Pantic, M.: Multi-conditional latent variable model for joint facial action unit detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3792–3800 (2015)

    Google Scholar 

  36. Liu, M., Li, S., Shan, S., Chen, X.: AU-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126–136 (2015)

    Article  Google Scholar 

  37. Bartlett, P.L., Collins, M., Taskar, B., McAllester, D.A.: Exponentiated gradient algorithms for large-margin structured classification. In: Proceedings of the Neural Information Processing Systems (NIPS) (2004)

    Google Scholar 

Download references

Acknowledgements

Melody Moh and Teng Moh are supported in part by SJSU RSCA awards.

Author information

Authors and Affiliations

  1. Department of Computer Science, San José State University, San José, CA, USA

    Shashank Shivaji Pawar, Melody Moh & Teng-Sheng Moh

Authors
  1. Shashank Shivaji Pawar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Melody Moh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Teng-Sheng Moh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Teng-Sheng Moh .

Editor information

Editors and Affiliations

  1. Intelligent Systems Research Institute, Sungkyunkwan University, Suwon, Korea (Republic of)

    Sukhan Lee

  2. Software Engineering, Universiti Kuala Lumpur, Kuala Lumpur, Malaysia

    Roslan Ismail

  3. College of Software, Sungkyunkwan University, Suwon, Korea (Republic of)

    Hyunseung Choo

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pawar, S.S., Moh, M., Moh, TS. (2019). Micro-expression Recognition Using Motion Magnification and Spatiotemporal Texture Map. In: Lee, S., Ismail, R., Choo, H. (eds) Proceedings of the 13th International Conference on Ubiquitous Information Management and Communication (IMCOM) 2019. IMCOM 2019. Advances in Intelligent Systems and Computing, vol 935. Springer, Cham. https://doi.org/10.1007/978-3-030-19063-7_29

Download citation

Publish with us

Policies and ethics

Search

Navigation