Skip to main content
Springer Nature Link
Log in

Emotional Intelligence in Multimodal Object Oriented User Interfaces

  • Chapter
New Directions in Intelligent Interactive Multimedia Systems and Services - 2

Part of the book series: Studies in Computational Intelligence ((SCI,volume 226))

  • 657 Accesses

Abstract

In this paper, we describe a multimodal emotion recognition system that is structured according to the object oriented method. The system uses the OO approach that combines evidence frommultiplemodalities of interaction and data from emotion stereotypes and classifies them into well structured objects with their own properties and methods. The system’s architecture can be adopted in future emotion recognition systems with multiple modalities and improved emotion detection algorithms, while the resulting emotion detection server is capable of using and handling transmitted information from different sources of data during human-computer interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Pantic, M., Rothkrantz, L.J.M.: Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE 91, 1370–1390 (2003)

    Article  Google Scholar 

  2. Jascanu, N., Jascanu, V., Bumbaru, S.: Toward emotional E-commerce: The customer agent. In: Lovrek, I., Howlett, R.J., Jain, L.C. (eds.) KES 2008, Part I. LNCS (LNAI), vol. 5177, pp. 202–209. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  3. Bernhardt, D., Robinson, P.: Interactive control of music using emotional body expressions. In: Conference on Human Factors in Computing Systems - Proceedings, pp. 3117–3122 (2008)

    Google Scholar 

  4. Picard, R.W.: Affective Computing: Challenges. Int. Journal of Human-Computer Studies 59(1-2), 55–64 (2003)

    Article  Google Scholar 

  5. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.: Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine 18(1), 32–80 (2001)

    Article  Google Scholar 

  6. Cacioppo, J.T., Berntson, G.G., Larsen, J.T., Poehlmann, K.M., Ito, T.A.: The Psycho-physiology of Emotion. In: Lewis, M., Haviland-Jones, J.M. (eds.) Handbook of Emotions, pp. 173–191 (2000)

    Google Scholar 

  7. Chen, L.S., Huang, T.S., Miyasato, T., Nakatsu, R.: Multimodal Human Emotion/Expression Recognition. In: Proceedings of the 3rd International Conference on Face & Gesture Recognition. IEEE Computer Society, Los Alamitos (1998)

    Google Scholar 

  8. De Silva, L., Miyasato, T., Nakatsu, R.: Facial emotion recognition using multimodal information. In: IEEE Int. Conf. on Information, Communications and Signal Processing (ICICS 1997), pp. 397–401 (1997)

    Google Scholar 

  9. Huang, T.S., Chen, L.S., Tao, H.: Bimodal emotion recognition by man and machine. In: ATR Workshop on Virtual Communication Environments, Kyoto, Japan (1998)

    Google Scholar 

  10. Oviatt, S.: User-modeling and evaluation of multimodal interfaces. Proceedings of the IEEE, 1457–1468 (2003)

    Google Scholar 

  11. Zeng, Z., Tu, J., Liu, M., Huang, T., Pianfetti, B., Roth, D., Levinson, S.: Audio-Visual Affect Recognition. IEEE Transactions on Multimedia 9, 424–428 (2007)

    Article  Google Scholar 

  12. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C., Kazemzadeh, A., Lee, S., Neumann, U., Narayanan, S.: Analysis of emotion recognition using facial expressions, speech and multimodal information. In: Proceedings of the 6th international conference on Multimodal interfaces State College. ACM, PA (2004)

    Google Scholar 

  13. Liao, W., Zhang, W., Zhu, Z., Ji, Q., Gray, W.D.: Toward a decision-theoretic framework for affect recognition and user assistance. Int. J. Hum.-Comput. Stud. 64, 847–873 (2006)

    Article  Google Scholar 

  14. Hwang, C.L., Yoon, K.: Multiple Attribute Decision Making: Methods and Applications, vol. 186. Springer, Heidelberg (1981)

    MATH  Google Scholar 

  15. Wikipedia. The free encyclopedia, http://en.wikipedia.org/wiki/Object_oriented

  16. Rumbaugh, J., Blaha, M., Premerlani, W., Eddy, F., Lorensen, W.: Object Oriented Modeling and Design. Prentice Hall, Englewood Cliffs (1991)

    Google Scholar 

  17. Jacobson, I., Christerson, M., Jonsson, P., Overgaard, G.: Object-oriented software engineering. A use case driven approach. ACM Press/Addison-Wesley (1992)

    Google Scholar 

  18. Pastor, O., Gómez, J., Insfrán, E., Pelechano, V.: The OO-Method approach for information systems modeling: From object-oriented conceptual modeling to automated programming. Information Systems 26(7), 507–534 (2001)

    Article  MATH  Google Scholar 

  19. Benz, U.C., Hofmann, P., Willhauck, G., Lingenfelder, I., Heynen, M.: Multi-resolution, object-oriented fuzzy analysis of remote sensing data for GIS-ready information. ISPRS Journal of Photogrammetry and Remote Sensing 58(3-4), 239–258 (2004)

    Article  Google Scholar 

  20. Booch, G.: The unified modeling language. Performance Computing/Unix Review 14(13), 41–48 (1996)

    Google Scholar 

  21. Rational Software Corporation, Rational Rose User’s Manual (1997)

    Google Scholar 

  22. Platinum Technology, Inc., Paradigm Plus: Round-Trip Engineering for JAVA, White Paper (1997)

    Google Scholar 

  23. Armstrong, D.J.: The Quarks of Object-Oriented Development. Communications of the ACM 49(2), 123–128 (2006)

    Article  Google Scholar 

  24. Alepis, E., Virvou, M., Kabassi, K.: Requirements Analysis and Design of an Affective Bi-Modal Intelligent Tutoring System: The Case of Keyboard and Microphone. Studies in Computational Intelligence, Intelligent Interactive Systems in Knowledge-Based Environments, vol. 104, pp. 9–24 (2008)

    Google Scholar 

  25. Tsihrintzis, G., Virvou, M., Stathopoulou, I.O., Alepis, E.: On improving visual-facial emotion recognition with audio-lingual and keyboard stroke pattern information. In: Web Intelligence and Intelligent Agent Technology, WI-IAT 2008, vol. 1, pp. 810–816 (2008)

    Google Scholar 

  26. Alepis, E., Virvou, M.: Emotional Intelligence: Constructing user stereotypes for affective bi- modal interaction. In: Gabrys, B., Howlett, R.J., Jain, L.C. (eds.) KES 2006, Part I. LNCS (LNAI), vol. 4251, pp. 435–442. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  27. Virvou, M., Tsihrintzis, G., Alepis, E., Stathopoulou, I.O., Kabassi, K.: Combining Empirical Studies of Audio-Lingual and Visual-Facial Modalities for Emotion Recognition. In: Apolloni, B., Howlett, R.J., Jain, L. (eds.) KES 2007, Part II. LNCS (LNAI), vol. 4693, pp. 1130–1137. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  28. Moriyama, T., Ozawa, S.: Measurement of Human Vocal Emotion Using Fuzzy Control. Systems and Computers in Japan 32(4) (2001)

    Google Scholar 

  29. Kay, J.: Stereotypes, student models and scrutability. In: Gauthier, G., VanLehn, K., Frasson, C. (eds.) ITS 2000. LNCS, vol. 1839, pp. 19–30. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  30. Rich, E.: Users are individuals: individualizing user models. International Journal of Man-Machine Studies 18, 199–214 (1983)

    Article  Google Scholar 

  31. Fishburn, P.C.: Additive Utilities with Incomplete Product Set: Applications to Priorities and Assignments, Operations Research (1967)

    Google Scholar 

  32. Hwang, C.L., Yoon, K.: Multiple Attribute Decision Making: Methods and Applications. Lecture Notes in Economics and Mathematical Systems, vol. 186. Springer, Heidelberg (1981)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, University of Piraeus, 80 Karaoli & Dimitriou St., 18534, Piraeus, Greece

    Efthymios Alepis & Maria Virvou

Authors
  1. Efthymios Alepis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Virvou
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Milan, Italy

    Ernesto Damiani

  2. Hanyang University, Seoul, Korea

    Jechang Jeong

  3. University of Brighton, UK

    Robert J. Howlett

  4. University of South Australia, Australia

    Lakhmi C. Jain

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Alepis, E., Virvou, M. (2009). Emotional Intelligence in Multimodal Object Oriented User Interfaces. In: Damiani, E., Jeong, J., Howlett, R.J., Jain, L.C. (eds) New Directions in Intelligent Interactive Multimedia Systems and Services - 2. Studies in Computational Intelligence, vol 226. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02937-0_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-02937-0_32

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-02936-3

  • Online ISBN: 978-3-642-02937-0

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics