Skip to main content
SpringerLink
Log in

Hybrid affective computing—keyboard, mouse and touch screen: from review to experiment

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Emotions play an important role in human interactions. They can be integrated into the computer system to make human–computer interaction (HCI) more effective. Affective computing is an innovative computational modeling and detecting user’s emotions to optimize system responses in HCI. However, there is a trade-off between recognition accuracy and real-time performance in some of the methods such as processing the facial expressions, human voice and body gestures. Other methods lack efficiency and usability in real-world applications such as natural language processing and electroencephalography signals. To accomplish a reliable, usable and high-performance system, this paper proposes an intelligent hybrid approach to recognize users’ emotions by using easily accessible and low computational cost input devices including keyboard, mouse (touch pad: single touch) and touch screen display (single touch). Using the proposed approach, the system is developed and trained in a supervised mode by artificial neural network and support vector machine (SVM) techniques. The result shows an increase in accuracy of 6 % (93.20 %) by SVM in comparison with the currently existing methods. It is a significant contribution to show new directions of future research in emotion recognition, user modeling and emotional intelligence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Leon E, Clarke G, Callaghan V, Sepulveda F (2007) A user-independent real-time emotion recognition system for software agents in domestic environments. Eng Appl Artif Intell 20(3):337–345. doi:10.1016/j.engappai.2006.06.001

    Article  Google Scholar 

  2. Harris M, Ishii H, Chung C, Dodsworth C, Buxton B (1999) Natural and invisible human interfaces. In: International conference on computer graphics and interactive techniques (SIGGRAPH). doi:10.1145/311625.311922

  3. Palm G, Glodek M (2013) Towards emotion recognition in human computer interaction. In: Neural nets and surroundings, vol 19. Springer, pp 323–336. doi:10.1007/978-3-642-35467-0_32

  4. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human–computer interaction. Sig Process Mag IEEE 18(1):32–80

    Article  Google Scholar 

  5. Picard RW (1998) Affective computing. MIT Press, Cambridge

    Google Scholar 

  6. Strongman KT (2003) The psychology of emotion. Department of Psychology, University of Canterbury, 5th edn. Wiley, Christchurch

    Google Scholar 

  7. Lewis M, Haviland-Jones JM, Barrett LF (2010) The cultural psychology of the emotions—ancient and renewed. In: Richard A, Shweder JH, Horton R, Joseph C (eds) Handbook of emotions, 3rd edn. The Guilford Press, New York, p 19

    Google Scholar 

  8. Fredrickson BL (2001) The role of positive emotions in positive psychology—the broaden-and-build theory of positive emotions. Am Psychol 56(3):9. doi:10.1037/0003-066X.56.3.218

    Article  Google Scholar 

  9. SRamaraj V, Ravindran A, Thirumurugan A (2013) Emotion recognition from human eye expression. IJRCCT 2(4):158–164

    Google Scholar 

  10. Schmidt K, Cohn J (2002) Human facial expressions as adaptations: evolutionary questions in facial expression. Am J Phys Anthropol 33:3–24

    Google Scholar 

  11. Baker C, Cokely D (1980) American sign language: a teacher’s resource text on grammar and culture. T.J. Publishers, Silver Spring

    Google Scholar 

  12. Jamshidnejad A (2009) Facial emotion recognition for human computer interaction using a fuzzy model in the e-business. In: Conference on innovative technologies in intelligent systems and industrial applications (CITISIA), pp 202–204

  13. Tsihrintzis GA, Virvou M, Alepis E, Stathopoulou IO (2008) Towards improving visual-facial emotion recognition through use of complementary keyboard-stroke pattern information. In: Fifth international conference on information technology: new generations, pp 32–37

  14. Valstar M, Patras I, Pantic M (2004) Facial action unit recognition using temporal templates. In: IEEE international workshop on robot and human interactive communication, pp 253–258

  15. Gu W, Xiang C, Venkatesh Y, Huang D, Lin H (2012) Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recogn 45(1):80–91

    Article  Google Scholar 

  16. Konar A, Chakraborty A, Halder A, Mandal R, Janarthanan R (2012) Interval type-2 fuzzy model for emotion recognition from facial expression. In: Kundu MK, Mitra S, Mazumdar D, Pal SK (eds) Perception and machine intelligence. Lecture notes in computer science, vol 7143. Springer, Heidelberg, pp 114–121. doi:10.1007/978-3-642-27387-2_15

  17. Kao CY, Fahn CS (2012) A design of face detection and facial expression recognition techniques based on boosting schema. Appl Mech Mater 121:617–621

    Google Scholar 

  18. Ilbeygi M, Shah-Hosseini H (2012) A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng Appl Artif Intell 25(1):130–146. doi:10.1016/j.engappai.2011.07.004

    Article  Google Scholar 

  19. Rahulamathavan Y, Phan RC-W, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4(1):83–92. doi:10.1109/T-AFFC.2012.33

    Article  Google Scholar 

  20. Beaudrya O, Roy-Charlanda A, Perrona M, Cormierb I, Tappa R (2014) Featural processing in recognition of emotional facial expressions. Cogn Emot 28(3):416–432. doi:10.1080/02699931.2013.833500

    Article  Google Scholar 

  21. Kleinsmith A, Bianchi-Berthouze N (2013) Affective body expression perception and recognition: a survey. IEEE Trans Affect Comput 4(1):15–33. doi:10.1109/T-AFFC.2012.16

    Article  Google Scholar 

  22. Lee JS, Shin D-H (2013) A study on the interaction between human and smart devices based on emotion recognition. In: HCI international 2013-posters’ extended abstracts. Springer, pp 352–356

  23. Bulwer J (1644) Chirologia: or the natural language of the hand. London. doi:10.1037/11828-001

  24. de Jorio A (1832) Gesture in naples and gesture in classical antiquity. Indiana University Press, Bloomington

    Google Scholar 

  25. McNeill D (2005) Gesture and thought. Chicago University Press, Chicago

    Book  Google Scholar 

  26. Goldin-Meadow S (2003) Hearing gesture: how our hands help us think. Harvard University Press, Cambridge

    Google Scholar 

  27. Gunes H, Piccardi M (2005) Fusing face and body gesture for machine recognition of emotions. In: IEEE international workshop on robots and human interactive communication, pp 306–311

  28. Glowinski D, Camurri A, Volpe G, Dael N, Scherer K (2008) Technique for automatic emotion recognition by body gesture analysis. In: IEEE computer society conference on computer vision and pattern recognition workshops

  29. Chena S, Tiana Y, Liub Q, Metaxasc DN (2013) Recognizing expressions from face and body gesture by temporal normalized motion and appearance features. Image Vis Comput 31(2):175–185. doi:10.1016/j.imavis.2012.06.014

    Article  Google Scholar 

  30. Steunebrink BR, Dastani M, Meyer JJC (2009) The OCC model revisited. Utrecht University, The Netherlands

    Google Scholar 

  31. Li H, Pang N, Guo S, Wang H (2008) Research on textual emotion recognition incorporating personality factor. In: IEEE international conference on robotics and biomimetics, pp 2222–2227

  32. Gil GB, Jesús ABd, Lopéz JMM (2013) combining machine learning techniques and natural language processing to infer emotions using Spanish twitter corpus. In: International workshops of PAAMS 365:149–167. doi:10.1007/978-3-642-38061-7_15

  33. Calvo RA, Kim SM (2013) Emotions in text: dimensional and categorical models. Comput Intell 29(3):527–543. doi:10.1111/j.1467-8640.2012.00456.x

    Article  MathSciNet  Google Scholar 

  34. Shahin I (2013) Speaker identification in emotional talking environments based on CSPHMM2s. Eng Appl Artif Intell 26(7):1652–1659. doi:10.1016/j.engappai.2013.03.013

    Article  Google Scholar 

  35. Klecková J (2009) Important nonverbal attributes for spontaneous speech recognition. In: Fourth international conference on systems, pp 13–16

  36. Wang Y, Guan L (2008) Recognizing human emotional state from audiovisual signals*. Multimed IEEE Trans 10(5):936–946

    Article  Google Scholar 

  37. Merrett T (2008) Query by humming. McGill University, Montreal

    Google Scholar 

  38. Unal E, Chew E, Georgiou PG, Narayanan SS (2008) Challenging uncertainty in query by humming systems: a fingerprinting approach. Audio Speech Lang Process IEEE Trans 16(2):359–371

    Article  Google Scholar 

  39. Amarakeerthi S, Ranaweera R, Cohen M (2010) Speech-based emotion characterization using postures and gestures in CVEs. In: International conference on cyberworlds, pp 72–76

  40. Pittermann J, Schmitt A, El Sayed NF (2008) Integrating linguistic cues into speech-based emotion recognition. In: 4th international conference on intelligent environments, pp 1–4

  41. Huang T (2008) Audio-visual human computer interface. In: IEEE international symposium on consumer electronics, pp 1–1

  42. Kotti M, Kotropoulos C (2008) Gender classification in two emotional speech databases. In: 19th international conference on pattern recognition, pp 1–4

  43. Krishnan A, Fernandez M (2014) System and method for recognizing emotional state from a speech signal. Google Patents

  44. Liu Y, Sourina O, Nguyen MK (2010) Real-time EEG-based human emotion recognition and visualization. In: International conference on cyberworlds, pp 262–269

  45. Schaaff K, Schultz T (2009) Towards emotion recognition from electroencephalographic signals. In: Affective computing and intelligent interaction and workshops (ACII), pp 1–6

  46. Guangying Y, Shanxiao Y (2010) Emotion recognition of electromyography based on support vector machine. In: Third international symposium on intelligent information technology and security informatics, pp 298–301

  47. Murugappan M, Murugappan S (2013) Human emotion recognition through short time electroencephalogram (EEG) signals using fast fourier transform (FFT). In: IEEE 9th international colloquium on signal processing and its applications (CSPA), pp 289–294. doi:10.1109/CSPA.2013.6530058

  48. Schuller B, Rigoll G, Lang M (2004) Emotion recognition in the manual interaction with graphical user interfaces. In: IEEE international conference on multimedia and expo 2. pp 1215–1218

  49. Milanova M, Sirakov N (2008) Recognition of emotional states in natural human–computer interaction. In: IEEE international symposium on signal processing and information technology (ISSPIT), pp 186–191

  50. Kaklauskas A, Zavadskas EK, Seniut M, Dzemyda G, Stankevic V, Simkevičius C, Stankevic T, Paliskiene R, Matuliauskaite A, Kildiene S, Bartkiene L, Ivanikovas S, Gribniak V (2011) Web-based biometric computer mouse advisory system to analyze a user’s emotions and work productivity. Eng Appl Artif Intell 24(6):928–945. doi:10.1016/j.engappai.2011.04.006

    Article  Google Scholar 

  51. Monrose F, Rubin AD (2000) Keystroke dynamics as a biometric for authentication. Future Gener Comput Syst 16(4):351–359

    Article  Google Scholar 

  52. Wang R, Fang B (2008) Affective computing and biometrics based HCI surveillance system. In: International symposium on information science and engineering, pp 192–195

  53. Epp C, Lippold M, Mandryk RL (2011) Identifying emotional states using keystroke dynamics. In: Proceedings of the 2011 annual conference on human factors in computing systems, pp 715–724. doi:10.1145/1978942.1979046

  54. Waikato Uo (2010) Weka. http://www.cs.waikato.ac.nz/ml/weka/. Accessed 01 July 2010

  55. Bashir MG, Nagarajan R, Hazry D (2010) Facial emotion detection using GPSO and Lucas–Kanade algorithms. In: International conference on computer and communication engineering (ICCCE), pp 1–6. doi:10.1109/ICCCE.2010.5556754

  56. Ekman P, Friesen WV (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Malor Books, Cambridge

    Google Scholar 

  57. Kao ECC, Liu CC, Yang TH, Hsieh CT, Soo VW (2009) Towards text-based emotion detection. In: International conference on information management and engineering (ICIME), pp 70–74

  58. Yang H, Willis A, De Roeck A, Nuseibeh B (2012) A hybrid model for automatic emotion recognition in suicide notes. Biomed Inform Insights 5(Supp 1):17–30. doi:10.4137/BII.S8948

    Article  Google Scholar 

  59. Böhlen M (2009) Second order ambient intelligence. J Ambient Intell Smart Environ 1(1):63–67

    Google Scholar 

  60. Aarts E, de Ruyter B (2009) New research perspectives on ambient intelligence. J Ambient Intell Smart Environ 1(1):5–14

    Google Scholar 

  61. Sinek S (2011) First why and then trust. TEDx. http://sciencestage.com/v/42756/tedxmaastricht-simon-sinek-first-why-and-then-trust.html. Accessed 06 Jan 2012

  62. Larson R, Csikszentmihalyi M (1983) The experience sampling method. In: New directions for methodology of social & behavioral science, vol 15, pp 41–46

  63. Hertenstein MJ, Keltner D, App B, Bulleit BA, Jaskolka AR (2006) Touch communicates distinct emotions. Emotion 6(3):528

    Article  Google Scholar 

  64. Rezaee Jordehi A (2014) A chaotic-based big bang–big crunch algorithm for solving global optimisation problems. Neural Comput Appl 25(6):1329–1335. doi:10.1007/s00521-014-1613-1

    Article  Google Scholar 

  65. Rezaee Jordehi A (2014) Particle swarm optimisation for dynamic optimisation problems: a review. Neural Comput Appl 25(7–8):1507–1516. doi:10.1007/s00521-014-1661-6

    Article  Google Scholar 

  66. Han J, Kamber M, Pei J (2012) Data mining concepts and techniques, 3rd edn. Elsevier, USA

    Google Scholar 

  67. Bakhtiyari K, Taghavi M, Husain H (2014) Implementation of emotional-aware computer systems using typical input devices. In: Nguyen NT, Attachoo B, Trawiński B, Somboonviwat K (eds) Intelligent Information and Database Systems, vol 8397. Springer International Publishing, Bangkok, pp 364–374. doi:10.1007/978-3-319-05476-6_37

    Chapter  Google Scholar 

  68. Kemp F (2003) Applied multiple regression/correlation analysis for the behavioral sciences. J R Stat Soc Ser D (Stat) 52(4):691

    Article  Google Scholar 

  69. Cohen J (2003) Applied multiple regression/correlation analysis for the behavioral sciences. Lawrence Erlbaum Associates, Publishers

    Google Scholar 

  70. Hauske G (2003) Statistische Signaltheorie. In: Skriptum zur Vorlesung. Technische Universität München, München, Deutschland

  71. Xiao Z, Dellandrea E, Dou W, Chen L (2007) Automatic hierarchical classification of emotional speech. In: Ninth IEEE international symposium on multimedia workshops, pp 291–296

  72. Bakhtiyari K, Husain H (2013) Fuzzy model on human emotions recognition. In: 12th international conference on applications of computer engineering, pp 77–82

  73. Bakhtiyari K, Husain H (2014) Fuzzy model of dominance emotions in affective computing. Neural Comput Appl 25(6):1467–1477. doi:10.1007/s00521-014-1637-6

    Article  Google Scholar 

  74. Taghavi M, Bakhtiyari K, Scavino E (2013) Agent-based computational investing recommender system. In: Proceedings of the 7th ACM conference on recommender systems, pp 455–458. doi:10.1145/2507157.2508072

  75. Taghavi M, Bakhtiyari K, Taghavi H, Olyaee V, Hussain A (2014) Planning for sustainable development in the emerging information societies. J Sci Technol Policy Manag 5(3):178–211. doi:10.1108/JSTPM-04-2014-0013

    Article  Google Scholar 

Download references

Acknowledgments

Thanks to the students of Universiti Kebangsaan Malaysia (The National University of Malaysia), University of Duisburg-Essen, Germany, and members of AIESEC UKM and AIESEC ESSEN, the largest nonprofit organization run by students, who participated in the prototype system evaluation.

Author information

Authors and Affiliations

  1. Interactive Systems, Department of Computer and Cognitive Science, Faculty of Engineering, University of Duisburg-Essen, 47048, Duisburg, North Rhine-Westphalia, Germany

    Kaveh Bakhtiyari

  2. Department of Electrical, Electronics and Systems Engineering, Faculty of Engineering and Built Environment, Universiti Kebangsaan Malaysia, UKM (The National University of Malaysia), 43600, Bangi, Selangor Darul Ehsan, Malaysia

    Kaveh Bakhtiyari, Mona Taghavi & Hafizah Husain

Authors
  1. Kaveh Bakhtiyari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mona Taghavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hafizah Husain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kaveh Bakhtiyari.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bakhtiyari, K., Taghavi, M. & Husain, H. Hybrid affective computing—keyboard, mouse and touch screen: from review to experiment. Neural Comput & Applic 26, 1277–1296 (2015). https://doi.org/10.1007/s00521-014-1790-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-014-1790-y

Keywords

Search

Navigation