Skip to main content
SpringerLink
Log in

Classification and Synthesis of Emotion in Sign Languages Using Neutral Expression Deviation Factor and 4D Trajectories

  • Conference paper
  • First Online:
Enterprise Information Systems (ICEIS 2020)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 417))

Included in the following conference series:

Abstract

3D Avatars are an efficient solution to complement the representation of sign languages in computational environments. A challenge, however, is to generate facial expressions realistically, without high computational cost in the synthesis process. This work synthesizes facial expressions with precise control through spatio-temporal parameters automatically. With parameters compatible with the gesture synthesis models for 3D avatars, it is possible to build complex expressions and interpolations of emotions through the model presented. The built method uses independent regions that allow the optimization of the animation synthesis process, reducing the computational cost and allowing independent control of the main facial regions. This work contributes to the definition of non-manual markers for 3D Avatar facia expression and its synthesis process. Also, a dataset with the base expressions was built where 4D information of the geometric control points of the avatar built for the experiments presented is found. The results of the generated outputs are validated in comparison with other expression classification approaches using Spatio-temporal data and machine learning, presenting superior accuracy for the base expressions. The rating is reinforced by evaluations conducted with the deaf community showing a positive acceptance of the facial expressions and synthesized emotions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Lombardo, V., Battaglino, C., Damiano, R., Nunnari, F.: An avatar-based interface for the Italian sign language. In: 2011 International Conference on Complex, Intelligent and Software Intensive Systems (CISIS), pp. 589–594 (2011)

    Google Scholar 

  2. Punchimudiyanse, M., Meegama, R.: 3D signing avatar for Sinhala Sign language. In: 2015 IEEE 10th International Conference on Industrial and Information Systems (ICIIS), pp. 290–295 (2015)

    Google Scholar 

  3. Kacorri, H., Huenerfauth, M., Ebling, S., Patel, K., Willard, M.: Demographic and experiential factors influencing acceptance of sign language animation by deaf users. In: Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 147–154. ACM (2015)

    Google Scholar 

  4. Sofiato, L.: Brazilian sign language dictionaries: comparative iconographical and lexical study. Educação e Pesquisa 40(1), 109–126 (2014). https://doi.org/10.1590/S1517-97022014000100008

    Article  Google Scholar 

  5. Bento, J., Claudio, A., Urbano, P.: Avatars on Portuguese sign language. In: 2014 9th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–7 (2014)

    Google Scholar 

  6. Adhan, S., Pintavirooj, C.: Thai sign language recognition by using geometric invariant feature and ANN classification. In: 2016 9th Biomedical Engineering International Conference (BMEiCON), pp. 1–4 (2016)

    Google Scholar 

  7. Ratan, R., Hasler, B.: Playing well with virtual classmates: relating avatar design to group satisfaction. In: Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work; Social Computing, pp. 564–573. ACM (2014)

    Google Scholar 

  8. Elons, A., Ahmed, M., Shedid, H.: Facial expressions recognition for Arabic sign language translation. In: 2014 9th International Conference on Computer Engineering Systems (ICCES), pp. 330–335 (2014)

    Google Scholar 

  9. Hyde, J., Carter, E., Kiesler, S., Hodgins, J.: Evaluating animated characters: facial motion magnitude influences personality perceptions. ACM Trans. Appl. Percept. 13(2), 8:1–8:17 (2016)

    Google Scholar 

  10. Neidle, C., Bahan, B., MacLaughlin, D., Lee, R., Kegl, J.: Realizations of syntactic agreement in American sign language: Similarities between the clause and the noun phrase. Studia Linguistica 52(3), 191–226 (1998)

    Article  Google Scholar 

  11. Iatskiu, C., Garcia, L., Antunes, D.: Automatic SignWriting generation of libras signs from CORE-SL. In: Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems, pp. 55:1–55:4. ACM (2017)

    Google Scholar 

  12. Griffin, H., Aung, M., Romera-Paredes, B., McLoughlin, C., McKeown, G., Curran, W., Bianchi-Berthouze, N.: Perception and automatic recognition of laughter from whole-body motion: continuous and categorical perspectives. IEEE Trans. Affective Comput. 6(2), 165–178 (2015)

    Article  Google Scholar 

  13. Grif, M., Manueva, Y.: Semantic analyses of text to translate to Russian sign language. In: 2016 11th International Forum on Strategic Technology (IFOST), pp. 286–289 (2016)

    Google Scholar 

  14. Wiegand, K.: Intelligent assistive communication and the web as a social medium. In: Proceedings of the 11th Web for All Conference, pp. 27:1–27:2. ACM (2014)

    Google Scholar 

  15. Basawapatna, A., Repenning, A., Savignano, M., Manera, J., Escherle, N., Repenning, L.: Is drawing video game characters in an hour of code activity a waste of time? In: Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, pp. 93–98. ACM (2018)

    Google Scholar 

  16. Feng, R., Prabhakaran, B.: On the "Face of Things". In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, pp. 3–4. ACM (2016)

    Google Scholar 

  17. Ahire, A., Evans, A., and Blat, J.: Animation on the web: a survey. In: Proceedings of the 20th International Conference on 3D Web Technology, pp. 249–257. ACM (2015)

    Google Scholar 

  18. Kacorri, H.: TR-2015001: A Survey and Critique of Facial Expression Synthesis in Sign Language Animation. CUNY Academic Works (2015)

    Google Scholar 

  19. Kaur, S., Singh, M.: Indian Sign Language animation generation system. In: 2015 1st International Conference on Next Generation Computing Technologies (NGCT), pp. 909–914 (2015)

    Google Scholar 

  20. Huenerfauth, M., Lu, P., Rosenberg, A.: Evaluating importance of facial expression in American sign language and pidgin signed english animations. In: The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 99–106. ACM (2011)

    Google Scholar 

  21. Alkawaz, M., Basori, A.: The effect of emotional colour on creating realistic expression of avatar. In: Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, pp. 143–152. ACM (2012)

    Google Scholar 

  22. Szwoch, W.: Model of emotions for game players. In: 2015 8th International Conference on Human System Interactions (HSI), pp. 285–290 (2015)

    Google Scholar 

  23. Happy, S., Routray, A.: Automatic Facial Expression Recognition Using Features of Salient Facial PatchesAffective Computing. IEEE Transactions on 6(1), 1–12 (2015)

    Google Scholar 

  24. Bouzid, Y., El Ghoul, O., Jemni, M.: Synthesizing facial expressions for signing avatars using MPEG4 feature points. In: 2013 Fourth International Conference on Information and Communication Technology and Accessibility (ICTA), pp. 1–6 (2013)

    Google Scholar 

  25. Sikdar, B.: Spatio-temporal correlations in cyber-physical systems: a defense against data availability attacks. In: Proceedings of the 3rd ACM Workshop on Cyber-Physical System Security, pp. 103–110. ACM (2017)

    Google Scholar 

  26. Yu, S., Poger, S.: Using a temporal weighted data model to maximize influence in mobile messaging apps for computer science education. J. Comput. Sci. Coll. 32(6), 210–211 (2017)

    Google Scholar 

  27. Erwig, M., Güting, R., Schneider, M., Vazirgiannis, M.: Abstract and discrete modeling of spatio-temporal data types. In: Proceedings of the 6th ACM International Symposium on Advances in Geographic Information Systems, pp. 131–136. ACM (1998)

    Google Scholar 

  28. Lee, J., Han, B., Choi, S.: Interactive motion effects design for a moving object in 4D films. In: Proceedings of the 22Nd ACM Conference on Virtual Reality Software and Technology, pp. 219–228. ACM (2016)

    Google Scholar 

  29. Mahmoud, M., Baltrusaitis, T., Robinson, P.: Automatic detection of naturalistic hand-over-face gesture descriptors. In: Proceedings of the 16th International Conference on Multimodal Interaction, pp. 319–326. ACM (2014)

    Google Scholar 

  30. Suheryadi, A., Nugroho, H.: Spatio-temporal analysis for moving object detection under complex environment. In: 2016 International Conference on Advanced Computer Science and Information Systems (ICACSIS), pp. 498–505 (2016)

    Google Scholar 

  31. Oliveira, M., Chatbri, H., Little, S., O’Connor, N.E., Sutherland, A.: A comparison between end-to-end approaches and feature extraction based approaches for Sign Language recognition. In: 2017 International Conference on Image and Vision Computing New Zealand (IVCNZ), pp. 1–6 (2017)

    Google Scholar 

  32. Lemaire, P., Ben Amor, B., Ardabilian, M., Chen, L., Daoudi, M.: Fully Automatic 3D Facial Expression Recognition Using a Region-based Approach. In Proceedings of the 2011 Joint ACM Workshop on Human Gesture and Behavior Understanding, pp. 53–58. ACM (2011)

    Google Scholar 

  33. Kacorri, H., Huenerfauth, M.: Implementation and evaluation of animation controls sufficient for conveying ASL facial expressions. In: Proceedings of the 16th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 261–262. ACM (2014)

    Google Scholar 

  34. Obaid, M., Mukundan, R., Billinghurst, M., Pelachaud, C.: Expressive MPEG-4 facial animation using quadratic deformation models. In: 2010 Seventh International Conference on Computer Graphics, Imaging and Visualization (CGIV), pp. 9–14 (2010)

    Google Scholar 

  35. Lv, S., Da, F., Deng, X.: A 3D face recognition method using region-based extended local binary pattern. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 3635–3639 (2015)

    Google Scholar 

  36. Dailey, M.N., et al.: Evidence and a computational explanation of cultural differences in facial expression recognition. Emotion, vol. 10(6) (2010)

    Google Scholar 

  37. Lyons, M.J., Akemastu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with gabor wavelets. In: 3rd IEEE International Conference on Automatic Face and Gesture Recognition (1998)

    Google Scholar 

  38. Lundqvist, J.: The averaged karolinska directed emotional faces - AKDEF. In: CD ROM from Department of Clinical Neuroscience, Psychology section (1998)

    Google Scholar 

  39. Aldrich, J.: Doing Least Squares: Perspectives from Gauss and Yule (1998)

    Google Scholar 

  40. Erwig, M., Güting, R.H.: Temporal objects for spatio-temporal data models and a comparison of their representations. In: International Workshop on Advances in Database Technologies, pp. 454–465 (1998)

    Google Scholar 

  41. Le, V., Tang, H., Huang, T.: Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. In: 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011), pp. 414–421 (2011)

    Google Scholar 

  42. Gloderer, M., Hertle, A.: Spline-based Trajectory Optimization for Autonomous Vehicles with Ackerman drive (2010)

    Google Scholar 

  43. T. Jusko, E. Scalable Trajectory Optimization Based on Bézier Curves (2016)

    Google Scholar 

  44. Li, H., Kulik, L., Ramamohanarao, K.: Spatio-temporal trajectory simplification for inferring travel paths. In: Proceedings of the 22Nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pp. 63–72. ACM (2014)

    Google Scholar 

  45. Song, N., Yang, H., and P. Wu (2018). A Gesture-to-Emotional Speech Conversion by Combining Gesture Recognition and Facial Expression Recognition

    Google Scholar 

  46. Adil, B., Nadjib, K.M., Yacine, L.: A novel approach for facial expression recognition propagation for core extraction (2019)

    Google Scholar 

  47. Kakarla, M., Reddy, G.R.M.: A real time facial emotion recognition using depth sensor and interfacing with Second Life based Virtual 3D avatar. In: International Conference on Recent Advances and Innovations in Engineering (ICRAIE-2014), Jaipur, 2014, pp. 1–7 (2014). https://doi.org/10.1109/ICRAIE.2014.6909153

  48. Wan, Y., Chiu, C., Liang, K., Chang, P.: Midoriko chatbot: LSTM-based emotional 3D avatar. In: 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE), Osaka, Japan, 2019, pp. 937–940. https://doi.org/10.1109/GCCE46687.2019.9015303

  49. Pandita, S.: Affective embodiment: the effect of avatar appearance and posture representation on emotions in VR. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Atlanta, GA, USA, 2020, pp. 539–540 (2020). https://doi.org/10.1109/VRW50115.2020.00121

  50. Shang, Z., Joshi, J., Hoey, J.: Continuous facial expression recognition for affective interaction with virtual avatar. In: 2017 IEEE International Conference on Image Processing (ICIP), Beijing, pp. 1995–1999 (2017). https://doi.org/10.1109/ICIP.2017.8296631

  51. Angga, P.A., Fachri, W.E., Elevanita, A., Suryadi, Agushinta, R.D.: Design of chatbot with 3D avatar, voice interface, and facial expression. In:2015 International Conference on Science in Information Technology (ICSITech), Yogyakarta, pp. 326–330 (2015). https://doi.org/10.1109/ICSITech.2015.7407826

  52. Silva, V., Soares, F., Esteves, J.S.: Mirroring emotion system - on-line synthesizing facial expressions on a robot face. In: 8th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), Lisbon 2016, pp. 213–218 (2016). https://doi.org/10.1109/ICUMT.2016.7765359

  53. Cho, T., Choi, J.-H., Kim, H.-J., Choi, S.-M.: Vision-based animation of 3D facial avatars. In: 2014 International Conference on Big Data and Smart Computing (BIGCOMP), Bangkok, 2014, pp. 128–132 (2014). https://doi.org/10.1109/BIGCOMP.2014.6741422

  54. Zhao, Y., Jiang, D., Sahli, H.: 3D emotional facial animation synthesis with factored conditional Restricted Boltzmann Machines. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Xi’an, 2015, pp. 797–803 (2015). https://doi.org/10.1109/ACII.2015.7344664

  55. Choy, C., Gwak, J., Savarese, S.: 4D spatio-temporal ConvNets: Minkowski convolutional neural networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 3070–3079 (2019). https://doi.org/10.1109/CVPR.2019.00319

  56. Ivson, P., Nascimento, D., Celes, W., Barbosa, S.D.: CasCADe: a novel 4D visualization system for virtual construction planning. IEEE Trans. Visualization Comput. Graph. 24(1), 687–697 (2018). https://doi.org/10.1109/TVCG.2017.2745105

    Article  Google Scholar 

  57. Zhang, H., Parker, L.E.: CoDe4D: color-depth local spatio-temporal features for human activity recognition from RGB-D videos. IEEE Trans. Circuits Syst. Video Technol. 26(3), 541–555 (2016). https://doi.org/10.1109/TCSVT.2014.2376139

    Article  Google Scholar 

  58. Zhao, S., Yao, H., Sun, X., Xu, P., Liu, X., Ji, R.: Video indexing and recommendation based on affective analysis of viewers. In: Proceedings of the 19th ACM International Conference on Multimedia (MM 2011), pp. 1473–1476. Association for Computing Machinery, New York (2011)

    Google Scholar 

  59. Lu, C., et al.: Multiple spatio-temporal feature learning for video-based emotion recognition in the wild. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI 2018), pp. 646–652. Association for Computing Machinery, New York (2018)

    Google Scholar 

  60. Kim, D.H., Baddar, W.J., Ro, Y.M.: Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM International Conference on Multimedia (MM 2016), pp. 382–386. Association for Computing Machinery, New York (2016)

    Google Scholar 

  61. Michel, P., El Kaliouby, R.: Real time facial expression recognition in video using support vector machines. In: Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI ’03), pp. 258–264. Association for Computing Machinery, New York (2003)

    Google Scholar 

  62. Yang, P., Liu, Q., Metaxas, D.N.: Exploring facial expressions with compositional features. In: CVPR (2010)

    Google Scholar 

  63. Gonçalves, D., Baranauskas, M., Reis, J., Todt, E.: Facial expressions animation in sign language based on spatio-temporal centroid. In: Proceedings of the 22nd International Conference on Enterprise Information Systems - Volume 2: ICEIS, pp. 463–475 (2020). https://doi.org/10.5220/0009344404630475.ISBN 978-989-758-423-7

  64. Liu, Z., Zhang, C.: Spatio-temporal analysis for infrared facial expression recognition from videos. In: Proceedings of the International Conference on Video and Image Processing (ICVIP 2017), pp. 63–67. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3177404.3177408

Download references

Acknowledgments

This work was financially supported by the São Paulo Research Foundation (FAPESP) (grants #2015/16528-0, #2015/24300-9 and Number 2019/12225-3), and CNPq (grant #306272/2017-2). We thank the University of Campinas (UNICAMP) and Universidade Federal do Paraná (UFPR) for making this research possible.

Author information

Authors and Affiliations

  1. Department of Informatics, Universidade Federal do Paraná, Curitiba, Brazil

    Diego Addan Gonçalves & Eduardo Todt

  2. Institute of Computing, University of Campinas, São Paulo, Brazil

    Maria Cecília Calani Baranauskas

  3. Post-graduation Program in Informatics, Federal University of Paraná (UFPR-PPGInf), Curitiba, Brazil

    Maria Cecília Calani Baranauskas & Eduardo Todt

Authors
  1. Diego Addan Gonçalves
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Cecília Calani Baranauskas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eduardo Todt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diego Addan Gonçalves .

Editor information

Editors and Affiliations

  1. Polytechnic Institute of Setúbal/INSTICC, Setúbal, Portugal

    Joaquim Filipe

  2. Warsaw University of Technology, Warsaw, Poland

    Michał Śmiałek

  3. George Mason University, Fairfax, VA, USA

    Alexander Brodsky

  4. MODESTE/ESEO, Angers, France

    Slimane Hammoudi

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gonçalves, D.A., Baranauskas, M.C.C., Todt, E. (2021). Classification and Synthesis of Emotion in Sign Languages Using Neutral Expression Deviation Factor and 4D Trajectories. In: Filipe, J., Śmiałek, M., Brodsky, A., Hammoudi, S. (eds) Enterprise Information Systems. ICEIS 2020. Lecture Notes in Business Information Processing, vol 417. Springer, Cham. https://doi.org/10.1007/978-3-030-75418-1_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-75418-1_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-75417-4

  • Online ISBN: 978-3-030-75418-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation