Mohammed Alharbi
ABSTRACT
Affective computing is considered one of the important areas in the field of human and computer interaction where software systems can recognize and understand human's behaviour and emotions. Affective computing integrates a variety of modalities of inputs that are used to recognize users' emotions and consequently respond to these emotions accordingly. In this paper, we first conducted a broad survey of the varieties of modalities that are used for incorporating affective computing in software systems. We then discussed, classified, and critically analyzed the different approaches in this field that can be used and incorporated in order to detect, analyze, and respond to users' inputs efficiently. The contribution of this paper is providing an up-to-date review about the current literature and discussing the current challenges that lead to some insights into the future work that can be done to make affective computing more effective.
- Jeremy D. Schwark. 2015. Toward a Taxonomy of Affective Computing. International Journal of Human-Computer Interaction 31, 761--768. DOI:https://doi.org/10.1080/10447318.2015.1064638Google Scholar
Cross Ref
- Shalom Greene, Himanshu Thapliyal, and Allison Caban-Holt. 2016. A Survey of Affective Computing for Stress Detection: Evaluating technologies in stress detection for better health. IEEE Consumer Electronics Magazine 5, 44-- 56. DOI:https://doi.org/10.1109/MCE.2016.2590178Google ScholarCross Ref
- Eric Lloyd, Shihong Huang, and Emmanuelle Tognoli. 2017. Improving Human-in-the-Loop Adaptive Systems Using Brain-Computer Interaction. Proceedings - 2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2017, 163--174. DOI:https://doi.org/10.1109/SEAMS.2017.1Google ScholarDigital Library
- Umer Farooq, Jonathan Grudin, Ben Shneiderman, Pattie Maes, and Xiangshi Ren. 2017. Human Computer Integration versus Powerful Tools. Proc. 2017 CHI Conf. Ext. Abstr. Hum. Factors Comput. Syst. - CHI EA '17 (2017), 1277--1282. DOI:https://doi.org/10.1145/3027063.3051137Google ScholarDigital Library
- Nagesh Jadhav and Rekha Sugandhi. 2019. Survey on Human Behavior Recognition Using Affective Computing. Proceedings - 2018 IEEE Global Conference on Wireless Computing and Networking, GCWCN 2018, 98--103. DOI:https://doi.org/10.1109/GCWCN.2018.8668632Google Scholar
- Henry Candra, Mitchell Yuwono, Rifai Chai, Ardi Handojoseno, Irraivan Elamvazuthi, Hung T. Nguyen, and Steven Su. 2015. Investigation of window size in classification of EEG-emotion signal with wavelet entropy and support vector machine. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS. DOI:https://doi.org/10.1109/EMBC.2015.7320065Google ScholarCross Ref
- Ruoyu Du and Hyo Jong Lee. 2016. Frontal alpha asymmetry during the audio emotional experiment revealed by event-related spectral perturbation. In Proceedings - 2015 8th International Conference on BioMedical Engineering and Informatics, BMEI 2015. DOI:https://doi.org/10.1109/BMEI.2015.7401562Google Scholar
- Zirui Lan, Olga Sourina, Lipo Wang, and Yisi Liu. 2016. Real-time EEG-based emotion monitoring using stable features. Visual Computer 32, 347--358. DOI:https://doi.org/10.1007/s00371-015-1183-yGoogle ScholarDigital Library
- Yuan Pin Lin, Chi Hong Wang, Tzyy Ping Jung, Tien Lin Wu, Shyh Kang Jeng, Jeng Ren Duann, and Jyh Horng Chen. 2010. EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. (2010). DOI:https://doi.org/10.1109/TBME.2010.2048568Google Scholar
- Raja Majid Mehmood and Hyo Jong Lee. 2015. Emotion classification of EEG brain signal using SVM and KNN. In 2015 IEEE International Conference on Multimedia and Expo Workshops, ICMEW 2015. DOI:https://doi.org/10.1109/ICMEW.2015.7169786Google ScholarCross Ref
- Rui Qiao, Chunmei Qing, Tong Zhang, Xiaofen Xing, and Xiangmin Xu. 2017. A novel deep-learning based framework for multi-subject emotion recognition. In ICCSS 2017 - 2017 International Conference on Information, Cybernetics, and Computational Social Systems. DOI:https://doi.org/10.1109/ICCSS.2017.8091408Google ScholarCross Ref
- Shiyi Wu, Xiangmin Xu, Lin Shu, and Bin Hu. 2017. Estimation of valence of emotion using two frontal EEG channels. In Proceedings - 2017 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2017. DOI:https://doi.org/10.1109/BIBM.2017.8217815Google ScholarCross Ref
- Xiaofen Xing, Zhenqi Li, Tianyuan Xu, Lin Shu, Bin Hu, and Xiangmin Xu. 2019. SAE+LSTM: A new framework for emotion recognition from multi-channel EEG. Front. Neurorobot. (2019). DOI:https://doi.org/10.3389/fnbot.2019.00037Google Scholar
- Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu. 2017. Identifying Stable Patterns over Time for Emotion Recognition from EEG. IEEE Trans. Affect. Comput. (2017). DOI:https://doi.org/10.1109/taffc.2017.2712143Google ScholarDigital Library
- Wei Long Zheng, Bo Nan Dong, and Bao Liang Lu. 2014. Multimodal emotion recognition using EEG and eye tracking data. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC 2014. DOI:https://doi.org/10.1109/EMBC.2014.6944757Google Scholar
- Foteini Agrafioti, Dimitris Hatzinakos, and Adam K. Anderson. 2012. ECG pattern analysis for emotion detection. IEEE Trans. Affect. Comput. (2012). DOI:https://doi.org/10.1109/T-AFFC.2011.28Google ScholarDigital Library
- Zi Cheng, Lin Shu, Jinyan Xie, and C. L.Philip Chen. 2018. A novel ECG-based real-time detection method of negative emotions in wearable applications. In 2017 International Conference on Security, Pattern Analysis, and Cybernetics, SPAC 2017. DOI:https://doi.org/10.1109/SPAC.2017.8304293Google Scholar
- Han Wen Guo, Yu Shun Huang, Chien Hung Lin, Jen Chien Chien, Koichi Haraikawa, and Jiann Shing Shieh. 2016. Heart Rate Variability Signal Features for Emotion Recognition by Using Principal Component Analysis and Support Vectors Machine. In Proceedings - 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering, BIBE 2016. DOI:https://doi.org/10.1109/BIBE.2016.40Google ScholarCross Ref
- Alejandro Rodriguez, Beatriz Rey, Ma Dolores Vara, Maja Wrzesien, Mariano Alcaniz, Rosa Ma Banos, and David Perez-Lopez. 2015. A VR-based serious game for studying emotional regulation in adolescents. IEEE Comput. Graph. Appl. (2015). DOI:https://doi.org/10.1109/MCG.2015.8Google ScholarDigital Library
- Gaetano Valenza, Luca Citi, Antonio Lanatá, Enzo Pasquale Scilingo, and Riccardo Barbieri. 2014. Revealing real-time emotional responses: A personalized assessment based on heartbeat dynamics. Sci. Rep. (2014). DOI:https://doi.org/10.1038/srep04998Google Scholar
- Gaetano Valenza, Antonio Lanata, and Enzo Pasquale Scilingo. 2012. The role of nonlinear dynamics in affective valence and arousal recognition. IEEE Trans. Affect. Comput. (2012). DOI:https://doi.org/10.1109/T-AFFC.2011.30Google ScholarDigital Library
- Yaqian Xu, Isabel Hubener, Ann Kathrin Seipp, Sandra Ohly, and Klaus David. 2017. From the lab to the real-world: An investigation on the influence of human movement on Emotion Recognition using physiological signals. In 2017 IEEE International Conference on Pervasive Computing and Communications Workshops, PerCom Workshops 2017. DOI:https://doi.org/10.1109/PERCOMW.2017.7917586Google Scholar
- Leila Mirmohamadsadeghi, Ashkan Yazdani, and Jean Marc Vesin. 2017. Using cardio-respiratory signals to recognize emotions elicited by watching music video clips. In 2016 IEEE 18th International Workshop on Multimedia Signal Processing, MMSP 2016. DOI:https://doi.org/10.1109/MMSP.2016.7813349Google Scholar
- Sylvia D. Kreibig. 2010. Autonomic nervous system activity in emotion: A review. Biological Psychology. DOI:https://doi.org/10.1016/j.biopsycho.2010.03.010Google Scholar
- Wee Ming Wong, Alan W C Tan, Chu Kiong Loo, and Wei Shiung Liew. 2010. PSO optimization of synergetic neural classifier for multichannel emotion recognition. In Proceedings - 2010 2nd World Congress on Nature and Biologically Inspired Computing, NaBIC 2010. DOI:https://doi.org/10.1109/NABIC.2010.5716292Google Scholar
- Priyanka Das, Anwesha Khasnobish, and D. N. Tibarewala. 2016. Emotion recognition employing ECG and GSR signals as markers of ANS. In Conference on Advances in Signal Processing, CASP 2016. DOI:https://doi.org/10.1109/CASP.2016.7746134Google ScholarCross Ref
- Ming Zher Poh, Tobias Loddenkemper, Nicholas C. Swenson, Shubhi Goyal, Joseph R. Madsen, and Rosalind W. Picard. 2010. Continuous monitoring of electrodermal activity during epileptic seizures using a wearable sensor. In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC'10. DOI:https://doi.org/10.1109/IEMBS.2010.5625988Google Scholar
- Ming Zher Poh, Nicholas C. Swenson, and Rosalind W. Picard. 2010. A wearable sensor for unobtrusive, long-term assessment of electrodermal activity. IEEE Trans. Biomed. Eng. (2010). DOI:https://doi.org/10.1109/TBME.2009.2038487Google Scholar
- Wanhui Wen, Guangyuan Liu, Nanpu Cheng, Jie Wei, Pengchao Shangguan, and Wenjin Huang. 2014. Emotion recognition based on multi-variant correlation of physiological signals. IEEE Trans. Affect. Comput. (2014). DOI:https://doi.org/10.1109/TAFFC.2014.2327617Google ScholarCross Ref
- Guanghua Wu, Guangyuan Liu, and Min Hao. 2010. The analysis of emotion recognition from GSR based on PSO. In Proceedings - 2010 International Symposium on Intelligence Information Processing and Trusted Computing, IPTC 2010. DOI:https://doi.org/10.1109/IPTC.2010.60Google ScholarDigital Library
- K. Gouizi, F. Bereksi Reguig, and C. Maaoui. 2011. Emotion recognition from physiological signals. Journal of Medical Engineering and Technology. DOI:https://doi.org/10.3109/03091902.2011.601784Google Scholar
- Hector P. Martinez, Yoshua Bengio, and Georgios Yannakakis. 2013. Learning deep physiological models of affect. IEEE Comput. Intell. Mag. (2013). DOI:https://doi.org/10.1109/MCI.2013.2247823Google ScholarDigital Library
- Zhongkai Tong, Xianxiang Chen, Zhengling He, Kai Tong, Zhen Fang, and Xianlong Wang. 2018. Emotion Recognition Based on Photoplethysmogram and Electroencephalogram. In Proceedings - International Computer Software and Applications Conference. DOI:https://doi.org/10.1109/COMPSAC.2018.10266Google ScholarCross Ref
- Arturo Nakasone, Helmut Prendinger, and Mitsuru Ishizuka. 2005. Emotion Recognition from Electromyography and Skin Conductance. 5th Int. Work. Biosignal Interpret. (2005).Google Scholar
- Guangying Yang and Shanxiao Yang. 2010. Emotion recognition of electromyography based on support vector machine. 3rd Int. Symp. Intell. Inf. Technol. Secur. Informatics, IITSI 2010 (2010), 298--301. DOI:https://doi.org/10.1109/IITSI.2010.122Google ScholarDigital Library
- Sabrina Stöckli, Michael Schulte-Mecklenbeck, Stefan Borer, and Andrea C. Samson. 2018. Facial expression analysis with AFFDEX and FACET: A validation study. Behavior Research Methods 50, 1446--1460. DOI:https://doi.org/10.3758/s13428-017-0996-1Google ScholarCross Ref
- Jose Maria Garcia-Garcia, Víctor M.R. Penichet, María Dolores Lozano, Juan Enrique Garrido, and Effie Lai Chong Law. 2018. Multimodal affective computing to enhance the user experience of educational software applications. Mobile Information Systems 2018. DOI:https://doi.org/10.1155/2018/8751426Google Scholar
- M Huang. 2012. Eye-Tracking Technology in Human-Computer Interaction. Cs.Bham.Ac.Uk. Retrieved from http://www.cs.bham.ac.uk/~rjh/courses/ResearchTopicsInH CI/2015-16/Submissions/huangmengsen.pdfGoogle Scholar
- Shlomo Berkovsky, Ronnie Taib, Irena Koprinska, Eileen Wang, Yucheng Zeng, Jingjie Li, and Sabina Kleitman. 2019. Detecting personality traits using eye-tracking data. Conference on Human Factors in Computing Systems -Proceedings. DOI:https://doi.org/10.1145/3290605.3300451Google ScholarDigital Library
- Hu Ni, Jianying Wang, Lan Wang, and Nan Yan. 2018. Track Your Emotional Perception of 3-D Virtual Talking Head in Human-computer Interaction. (2018), 298--303. DOI:https://doi.org/10.1109/CBS.2018.8612271Google Scholar
- Huan-huan Wang and Jing-wei Gu. 2018. The Applications of Facial Expression Recognition in Human-computer Interaction. 2018 IEEE Int. Conf. Adv. Manuf. (2018), 288-- 291.Google Scholar
- Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion 37, 98--125. DOI:https://doi.org/10.1016/j.inffus.2017.02.003Google ScholarDigital Library
- Kairos: Serving Businesses with Face Recognition. Retrieved November 8, 2019 from https://www.kairos.com/Google Scholar
- CrowdEmotion. Retrieved November 11, 2019 from https://www.crowdemotion.co.uk/Google Scholar
- Home - Affectiva: Affectiva. Retrieved November 11, 2019 from https://www.affectiva.com/Google Scholar
- Beyond Verbal. Retrieved November 11, 2019 from https://beyondverbal.com/Google Scholar
- NVISO | Fintech Artificial Intelligence. Retrieved November 11, 2019 from https://www.nviso.ai/enGoogle Scholar
- Eyeris. Retrieved November 12, 2019 from https://www.eyeris.ai/Google Scholar
- Stepan Mracek, Radim Dvorak, Jan Vana, Tomas Novotny, and Martin Drahansky. 2015. 3D face recognition utilizing a low-cost depth sensor. 2015 11th IEEE Int. Conf. Work. Autom. Face Gesture Recognition, FG 2015 (2015), 88579. DOI:https://doi.org/10.1109/FG.2015.7163169Google ScholarCross Ref
- Sension -- A Visual Interface. Retrieved November 12, 2019 from http://www.sension.co/Google Scholar
- Human Centric Software & Natural User Interfaces by gestigon. Retrieved November 12, 2019 from https://www.gestigon.com/home/Google Scholar
- Alejandro Jaimes and Nicu Sebe. 2007. Multimodal human-computer interaction: A survey. Comput. Vis. Image Underst. 108, 1--2 (2007), 116--134. DOI:https://doi.org/10.1016/j.cviu.2006.10.019Google ScholarDigital Library
- Rosalind W. Picard. 2003. Affective computing: Challenges. Int. J. Hum. Comput. Stud. (2003). DOI:https://doi.org/10.1016/S1071-5819(03)00052-1Google ScholarDigital Library
- Ciprian Corneanu, Fatemeh Noroozi, Dorota Kaminska, Tomasz Sapinski, Sergio Escalera, and Gholamreza Anbarjafari. 2018. Survey on Emotional Body Gesture Recognition. IEEE Trans. Affect. Comput. XX, X (2018), 1-- 19. DOI:https://doi.org/10.1109/TAFFC.2018.2874986Google ScholarDigital Library
- Nourah Alswaidan and Mohamed El Bachir Menai. 2020. A survey of state-of-the-art approaches for emotion recognition in text. Knowl. Inf. Syst. (2020). DOI:https://doi.org/10.1007/s10115-020-01449-0Google ScholarDigital Library
Index Terms
- A Survey of Incorporating Affective Computing for Human-System Co-adaptation
Recommendations
The affective connection: how and when users communicate emotion
CHI EA '04: CHI '04 Extended Abstracts on Human Factors in Computing SystemsAffective computer systems which recognize human emotions or use emotion in their displays have potential to enhance human-computer interaction (HCI). Wizard-of-Oz (WOZ) methods and experimental design have enabled recording, analysis and comparison of ...
Affective computing with primary and secondary emotions in a virtual human
We introduce the WASABI ([W]ASABI [A]ffect [S]imulation for [A]gents with [B]elievable [I]nteractivity) Affect Simulation Architecture, in which a virtual human's cognitive reasoning capabilities are combined with simulated embodiment to achieve the ...
Affective Guide for Museum: A System to Suggest Museum Paths Based on Visitors’ Emotions
Universal Access in Human-Computer Interaction. Design Methods and User ExperienceAbstractThis paper introduces a new recommendation system for museums able to profile the visitors and propose them the most suitable exhibition path accordingly, to improve visitors’ satisfaction. It consists of an interactive touch screen totem, which ...
Comments