Skip to main content
SpringerLink
Log in
Book cover

AVI Workshop on Big Data Applications

Italian Workshop on Visualization and Visual Analytics

AVI-BDA 2020, ITAVIS 2020: Advanced Visual Interfaces. Supporting Artificial Intelligence and Big Data Applications pp 109–124Cite as

Recognition and Visualization of Facial Expression and Emotion in Healthcare

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12585))

Abstract

To make the SenseCare KM-EP system more useful and smart, we integrated emotion recognition from facial expression. People with dementia have capricious feelings; the target of this paper is measuring and predicting these facial expressions. Analysis of data from emotional monitoring of dementia patients at home or during medical treatment will help healthcare professionals to judge the behavior of people with dementia in an improved and more informed way. In relation to the research project, SenseCare, this paper describes methods of video analysis focusing on facial expression and visualization of emotions, in order to implement an “Emotional Monitoring” web tool, which facilitates recognition and visualization of facial expression, in order to raise the quality of therapy. In this study, we detail the conceptual design of each process of the proposed system, and we describe our methods chosen for the implementation of the prototype using face-api.js and tensorflow.js for detection and recognition of facial expression and the PAD space model for 3D visualization of emotions.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Engel, F., et al.: Sensecare: towards an experimental platform for home-based, visualisation of emotional states of people with dementia. In: Bornschlegl, M.X., Engel, F.C., Bond, R., Hemmje, M.L. (eds.) AVI-BDA 2016. LNCS, vol. 10084, pp. 63–74. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-50070-6_5

    Chapter  Google Scholar 

  2. Bornschlegl, M.X., et al.: IVIS4BigData: a reference model for advanced visual interfaces supporting big data analysis in virtual research environments. In: Bornschlegl, M.X., Engel, F.C., Bond, R., Hemmje, M.L. (eds.) AVI-BDA 2016. LNCS, vol. 10084, pp. 1–18. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-50070-6_1

    Chapter  Google Scholar 

  3. Goleman, D.: Emotional Intelligence. Bantam Books, Inc, New York (1995)

    Google Scholar 

  4. Bond, R.R., et al.: SenseCare: using affective computing to manage and care for the emotional wellbeing of older people. In: Giokas, K., Bokor, L., Hopfgartner, F. (eds.) eHealth 360°. LNICST, vol. 181, pp. 352–356. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-49655-9_42

    Chapter  Google Scholar 

  5. Machine Intelligence and Signal Processing, Ebook. In: Proceedings of International Conference, Springer, Singapore, MISP (2019). ISBN 978-981-13-0923-6

    Google Scholar 

  6. Minhas, R.A., Javed, A., Irtaza, A., Mahmood, M.T., Joo, Y.B.: Shot classification of field sports videos using alexnet convolutional neural network. Appl. Sci. 9(3), 483 (2019)

    Article  Google Scholar 

  7. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)

    Google Scholar 

  8. Brownlee, J.: Deep Learning for Computer Vision: Image Classification, Object Detection, and Face Recognition in Python. Machine Learning Mastery, Vermont, Australia (2019)

    Google Scholar 

  9. Lim, Y.K., Liao, Z., Petridis, S., Pantic, M.: Transfer learning for action unit recognition. CoRR abs/1807.07556 (2018)

    Google Scholar 

  10. He., K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  11. Kim, Byoungjun, Lee, Joonwhoan: A deep-learning based model for emotional evaluation of video clips. Int. J. Fuzzy Logic Intell. Syst. 18(4), 245–253 (2018)

    Article  MathSciNet  Google Scholar 

  12. Turabzadeh, S., Meng, H., Swash, R.M., Pleva, M., Juhar, J.: Facial expression emotion detection for real-time embedded systems. Technologies 6, 17 (2018)

    Article  Google Scholar 

  13. Bahreini, K., van der Vegt, W., Westera, W.: A fuzzy logic approach to reliable real-time recognition of facial emotions. Multi. Tools Appl. 78, 18943–18966 (2019)

    Article  Google Scholar 

  14. Guérin-Dugué, A., Roy, R.N., Kristensen, E., Rivet, B., Vercueil, L., Tcherkassof, A.: Temporal dynamics of natural static emotional facial expressions decoding: a study using event- and eye fixation-related. Potentials. Front. Psychol. 9, 1190 (2018). https://doi.org/10.3389/fpsyg.2018.01190

    Article  Google Scholar 

  15. Long short-term memory. Neural Comput. 9 (8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735. PMID 9377276

  16. Ding, H., Zhou, S.K., Chellappa, R.: FaceNet2ExpNet: regularizing a deep face recognition net for expression recognition. In: 12th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 118–126 (2017)

    Google Scholar 

  17. Vasudevan, C.: Concepts and Programming in PyTorch: A way to dive into the technicality, BPB Publications (2018). ISBN 9388176057, 9789388176057

    Google Scholar 

  18. OpenCV (Open Source Computer Vision Library), link: https://opencv.org/. Accessed 23 June 2020

  19. Rosebrock, A.: Live video streaming over network with OpenCV and ImageZMQ. https://www.pyimagesearch.com/2019/04/15/live-video-streaming-over-network-with-opencv-and-imagezmq/. Accessed 23 June 2020

  20. Rao, K.S., Koolagudi, G.: Emotion Recognition using Speech Features, Springer New York (2013). https://doi.org/10.1007/978-1-4614-5143-3

  21. Research Centre on Scientific and Technical Information, link: http://www.cerist.dz. Accessed 23 June 2020)

  22. Face-api.js, JavaScript API for face detection and face recognition in the browser and nodejs with tensorflow.js, link: https://github.com/justadudewhohacks/face-api.js/. Accessed 23 June 2020

  23. TensorFlow.js, JavaScript library for machine learning, link: https://www.tensorflow.org/js. Accessed 23 June 2020

  24. Smilkov, D., et al.: Tensorflow.js: Machine learning for the web and beyond. arXiv preprint arXiv:1901.05350 (2019)

  25. Tiny YOLO v2 object detection with tensorflow.js, Link: https://github.com/justadudewhohacks/tfjs-tiny-yolov2. Accessed 23 June 2020

  26. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig. Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  27. Dlib C ++ Library, link: http://dlib.net/. Accessed 23 June 2020

  28. Realtime Face Recognition in the Browser, link: https://morioh.com/p/ddbc538212df. Accessed 23 June 2020

  29. Ding, H.S., Zhou, K., Chellappa, R.: FaceNet2ExpNet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, pp. 118–126 (2017). https://doi.org/10.1109/fg.2017.23

  30. Russell, J.: A circumplex model of affect. J. Pers. Soc. Psychol. 39(6), 1161–1178 (1980). https://doi.org/10.1037/h0077714

    Article  Google Scholar 

  31. Mehrabian, A.: Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Curr. Psychol. 14, 261–292 (1996)

    Article  MathSciNet  Google Scholar 

  32. Tavara, D.D.L.A.: Visualization of Affect in Faces Based on Context Appraisal. Doctoral Thesis, University of Balearic Islands, Spain (2012)

    Google Scholar 

  33. Hadjar, H., Meziane, A., Gherbi, R., Setitra, I., Aouaa, N.: WebVR based interactive visualization of open healthdata. In: International conference on Web Studies (WS.2 2018), October 3–5, 2018, Paris, France. ACM, New York, NY, USA, p. 8 (2018)

    Google Scholar 

  34. Reis, T.M.X., Bornschlegl, M.L.H.: Towards a reference model for artificial intelligence supporting big data analysis. In: Proceedings of the 2020 International Conference on Data Science (ICDATA 2020) (2020)

    Google Scholar 

  35. OECD, Artificial Intelligence in Society (2019)

    Google Scholar 

  36. Keras implementation of residual networks, link: https://gist.github.com/mjdietzx/0cb95922aac14d446a6530f87b3a04ce. Accessed 23 June 2020

Download references

Acknowledgements

This research has been developed in the context of the SenseCare project. SenseCare has received funding from the European Union’s H2020 Programme under grant agreement No 690862. However, this paper reflects only the authors’ views and the European Commission is not responsible for any use that may be made of the information it contains.

Author information

Authors and Affiliations

  1. University of Hagen, Faculty of Mathematics and Computer Science, 58097, Hagen, Germany

    Hayette Hadjar, Thoralf Reis & Marco X. Bornschlegl

  2. Research Institute for Telecommunication and Cooperation, FTK, Dortmund, Germany

    Felix C. Engel & Matthias L. Hemmje

  3. Ulster University, Derry/Londonderry, Northern Ireland

    Paul Mc Kevitt

Authors
  1. Hayette Hadjar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thoralf Reis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marco X. Bornschlegl
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Felix C. Engel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Paul Mc Kevitt
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Matthias L. Hemmje
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hayette Hadjar .

Editor information

Editors and Affiliations

  1. University of Hagen, Hagen, Germany

    Thoralf Reis

  2. University of Hagen, Hagen, Germany

    Marco X. Bornschlegl

  3. Sapienza University of Rome, Rome, Italy

    Marco Angelini

  4. University of Hagen, Hagen, Germany

    Matthias L. Hemmje

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hadjar, H., Reis, T., Bornschlegl, M.X., Engel, F.C., Mc Kevitt, P., Hemmje, M.L. (2021). Recognition and Visualization of Facial Expression and Emotion in Healthcare. In: Reis, T., Bornschlegl, M.X., Angelini, M., Hemmje, M.L. (eds) Advanced Visual Interfaces. Supporting Artificial Intelligence and Big Data Applications. AVI-BDA ITAVIS 2020 2020. Lecture Notes in Computer Science(), vol 12585. Springer, Cham. https://doi.org/10.1007/978-3-030-68007-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68007-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68006-0

  • Online ISBN: 978-3-030-68007-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation