Mina Alipour
ABSTRACT
Interactive computer systems’ designers emphasize the importance of considering humans, their emotions, and behaviors as first-class entities. Emotions are integral parts of human nature, and ignoring that can lead the interactive systems to failure, low quality, or discomfort. User interfaces (UIs) are increasingly becoming adaptive to users’ various characteristics, intending to improve users’ satisfaction, performance, and decisions. However, the previous approaches proposed for supervising such adaptations are not effectively adopted in real-life problems. This paper proposes the novel approach to adapting UIs to users’ emotions using Model-Free Reinforcement Learning (MFRL). The approach aims to maximize applying the essential adaptations and minimize the unnecessary ones towards users’ task completion and satisfaction. We chose emergency evacuation training as a suitable evaluation domain since people experience intense emotions in potential danger. We performed experiments with a mobile application we developed that acts as a recommender system in emergency training. By taking contextual input of the users’ basic emotions from face recognition, the application intelligently adapts its UI to quickly lead people to safe areas while arousing target emotions. The research includes literature analysis, surveys, and further adopting an iterative process in implementation and experimentation. The evaluation process confirms the efficiency and effectiveness of the MFRL in iterations, as well as compared to other possible UI adaptation techniques, i.e., rule-based and sequential adaptation.
- Mina Alipour, Sophie Dupuy-Chessa, and Eric Céret. 2021. An emotion-oriented problem space for ui adaptation: From a literature review to a conceptual framework. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1–8.Google Scholar
Cross Ref
- Mina Alipour, Sophie Dupuy-Chessa, and Eline Jongmans. 2020. Disaster mitigation using interface adaptation to emotions: a targeted literature review. In 10th International Conference on the Internet of Things Companion. 1–15.Google ScholarDigital Library
- Reem Alnanih, Olga Ormandjieva, and Thiruvengadam Radhakrishnan. 2013. Context-based and rule-based adaptation of mobile user interfaces in mHealth. Procedia Computer Science 21 (2013), 390–397.Google ScholarCross Ref
- Claudio Arbib, Davide Arcelli, Julie Dugdale, Mahyar T Moghaddam, and Henry Muccini. 2019. Real-time emergency response through performant IoT architectures. In International Conference on Information Systems for Crisis Response and Management (ISCRAM).Google Scholar
- Claudio Arbib, Mahyar T Moghaddam, and Henry Muccini. 2019. Iot flows: a network flow model application to building evacuation. A View of Operations Research Applications in Italy, 2018 (2019), 115–131.Google Scholar
- Claudio Arbib, Henry Muccini, and Mahyar Tourchi Moghaddam. 2018. Applying a network flow model to quick and safe evacuation of people from a building: a real case.RSFF 18 (2018), 50–61.Google Scholar
- Jonathan S Barnhoorn, Erwin Haasnoot, Bruno R Bocanegra, and Henk van Steenbergen. 2015. QRTEngine: An easy solution for running online reaction time experiments using Qualtrics. Behavior research methods 47, 4 (2015), 918–929.Google Scholar
- Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49–59.Google ScholarCross Ref
- Tobias Brosch, David Sander, Gilles Pourtois, and Klaus R Scherer. 2008. Beyond fear: Rapid spatial orienting toward positive emotional stimuli. Psychological science 19, 4 (2008), 362–370.Google Scholar
- Hartmut Dieterich, Uwe Malinowski, Thomas Kühme, and Matthias Schneider-Hufschmidt. 1993. State of the art in adaptive user interfaces. Human factors in information technology 10 (1993), 13–13.Google Scholar
- Julie Dugdale, Mahyar T Moghaddam, and Henry Muccini. 2020. Human behaviour centered design: developing a software system for cultural heritage. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Society. 85–94.Google ScholarDigital Library
- Paul Ekman. 1993. Facial expression and emotion.American psychologist 48, 4 (1993), 384.Google Scholar
- Paul Ekman. 1999. Basic emotions. Handbook of cognition and emotion 98, 45-60 (1999), 16.Google Scholar
- Paul Ekman, Wallace V Friesen, Maureen O’sullivan, Anthony Chan, Irene Diacoyanni-Tarlatzis, Karl Heider, Rainer Krause, William Ayhan LeCompte, Tom Pitcairn, Pio E Ricci-Bitti, 1987. Universals and cultural differences in the judgments of facial expressions of emotion.Journal of personality and social psychology 53, 4 (1987), 712.Google Scholar
- Sergio Firmenich, Alejandra Garrido, Fabio Paternò, and Gustavo Rossi. 2019. User interface adaptation for accessibility. In Web Accessibility. Springer, 547–568.Google Scholar
- Julián Andrés Galindo, Sophie Dupuy-Chessa, Nadine Mandran, and Eric Céret. 2018. Using user emotions to trigger UI adaptation. In 2018 12th International Conference on Research Challenges in Information Science (RCIS). IEEE, 1–11.Google ScholarCross Ref
- Jose Maria Garcia-Garcia, Victor MR Penichet, and Maria D Lozano. 2017. Emotion detection: a technology review. In Proceedings of the XVIII international conference on human computer interaction. 1–8.Google ScholarDigital Library
- Jan Gläscher, Nathaniel Daw, Peter Dayan, and John P O’Doherty. 2010. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 4 (2010), 585–595.Google ScholarCross Ref
- Donald Glowinski, Antonio Camurri, Gualtiero Volpe, Nele Dael, and Klaus Scherer. 2008. Technique for automatic emotion recognition by body gesture analysis. In 2008 IEEE Computer society conference on computer vision and pattern recognition workshops. IEEE, 1–6.Google Scholar
- Brad Hardin and Dave McCool. 2015. BIM and construction management: proven tools, methods, and workflows. John Wiley & Sons.Google Scholar
- Sabina Hunziker, Laura Laschinger, Simone Portmann-Schwarz, Norbert K Semmer, Franziska Tschan, and Stephan Marsch. 2011. Perceived stress and team performance during a simulated resuscitation. Intensive care medicine 37, 9 (2011), 1473–1479.Google Scholar
- Jeffrey O Kephart and David M Chess. 2003. The vision of autonomic computing. Computer 36, 1 (2003), 41–50.Google ScholarDigital Library
- Yanal Mahmoud Kilani. 2021. SMART Business Role in Supporting Marketing Strategies among Telecommunication Organizations Injordan. Turkish Journal of Computer and Mathematics Education (TURCOMAT) 12, 6 (2021), 4568–4582.Google Scholar
- Paul R Kleinginna and Anne M Kleinginna. 1981. A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and emotion 5, 4 (1981), 345–379.Google Scholar
- Amit Konar and Aruna Chakraborty. 2015. Emotion recognition: A pattern analysis approach. John Wiley & Sons.Google Scholar
- Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction. Morgan Kaufmann.Google Scholar
- Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, and Irina Rish. 2019. A story of two streams: Reinforcement learning models from human behavior and neuropsychiatry. arXiv preprint arXiv:1906.11286 (2019).Google Scholar
- David Lindlbauer, Anna Maria Feit, and Otmar Hilliges. 2019. Context-aware online adaptation of mixed reality interfaces. In Proceedings of the 32nd annual ACM symposium on user interface software and technology. 147–160.Google ScholarDigital Library
- Tiago V Maia and Michael J Frank. 2011. From reinforcement learning models to psychiatric and neurological disorders. Nature neuroscience 14, 2 (2011), 154–162.Google Scholar
- Christian Märtin, Sanim Rashid, and Christian Herdin. 2016. Designing responsive interactive applications by emotion-tracking and pattern-based dynamic user interface adaptation. In International Conference on Human-Computer Interaction. Springer, 28–36.Google ScholarCross Ref
- Elizabeth Martin. 2006. Survey questionnaire construction. Survey methodology 13 (2006), 2006.Google Scholar
- Sascha Meudt, Miriam Schmidt-Wack, Frank Honold, Felix Schüssel, Michael Weber, Friedhelm Schwenker, and Günther Palm. 2016. Going further in affective computing: how emotion recognition can improve adaptive user interaction. In Toward Robotic Socially Believable Behaving Systems-Volume I. Springer, 73–103.Google Scholar
- Nesrine Mezhoudi. 2013. User interface adaptation based on user feedback and machine learning. In Proceedings of the companion publication of the 2013 international conference on Intelligent user interfaces companion. 25–28.Google ScholarDigital Library
- Mahyar T Moghaddam, Mina Alipour, and Mikkel Baun Kjærgaard. 2023. User Interface and Architecture Adaption Based on Emotions and Behaviors. In 2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C). IEEE, 101–105.Google Scholar
- Mahyar T Moghaddam, Henry Muccini, and Julie Dugdale. 2022. Intelligent Building Evacuation: From Modeling Systems to Behaviors. In Disaster Management and Information Technology: Professional Response and Recovery Management in the Age of Disasters. Springer, 111–129.Google Scholar
- Mahyar T Moghaddam, Henry Muccini, Julie Dugdale, and Mikkel Baun Kjægaard. 2022. Designing Internet of Behaviors Systems. In 2022 IEEE 19th International Conference on Software Architecture (ICSA). IEEE, 124–134.Google Scholar
- Patricia Moravec, Lu Lucy Yan, and Alfonso Pedraza Martinez. 2021. Wildfire Response Operations: Intentional Fear Reduction through Social Media Updates. Kelley School of Business Research Paper2021-05 (2021).Google Scholar
- Vivian Genaro Motti and Jean Vanderdonckt. 2013. A computational framework for context-aware adaptation of user interfaces. In IEEE 7th International Conference on Research Challenges in Information Science (RCIS). IEEE, 1–12.Google ScholarCross Ref
- Henry Muccini, Romina Spalazzese, Mahyar T Moghaddam, and Mohammad Sharaf. 2018. Self-adaptive IoT architectures: An emergency handling case study. In Proceedings of the 12th European Conference on Software Architecture: Companion Proceedings. 1–6.Google ScholarDigital Library
- Daniel M Oppenheimer, Tom Meyvis, and Nicolas Davidenko. 2009. Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of experimental social psychology 45, 4 (2009), 867–872.Google ScholarCross Ref
- Jonathan Posner, James A Russell, and Bradley S Peterson. 2005. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology 17, 3 (2005), 715–734.Google Scholar
- Roger Pressman and Bruce Maxim. 2019. ISE Software Engineering: A Practitioner’s Approach (9 ed.). McGraw-Hill Education, Columbus, OH.Google Scholar
- Arman Savran, Koray Ciftci, Guillaume Chanel, Javier Mota, Luong Hong Viet, Blent Sankur, Lale Akarun, Alice Caplier, and Michele Rombaut. 2006. Emotion detection in the loop from brain signals and facial images. (2006).Google Scholar
- Annett Schirmer and Ralph Adolphs. 2017. Emotion perception from face, voice, and touch: comparisons and convergence. Trends in cognitive sciences 21, 3 (2017), 216–228.Google Scholar
- Hanan Shteingart and Yonatan Loewenstein. 2014. Reinforcement learning and human behavior. Current Opinion in Neurobiology 25 (2014), 93–98.Google ScholarCross Ref
- Jean-Sébastien Sottet, Gaëlle Calvary, Jean-Marie Favre, Joëlle Coutaz, Alexandre Demeure, and Lionel Balme. 2006. Towards Model Driven Engineering of Plastic User Interfaces. In Satellite Events at the MoDELS 2005 Conference, Jean-Michel Bruel (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 191–200.Google Scholar
- Jean-Sébastien Sottet, Vincent Ganneau, Gaëlle Calvary, Joëlle Coutaz, Alexandre Demeure, Jean-Marie Favre, and Rachel Demumieux. 2007. Model-driven adaptation for plastic user interfaces. In IFIP Conference on Human-Computer Interaction. Springer, 397–410.Google ScholarCross Ref
- Hamed Taherdoost. 2016. How to design and create an effective survey/questionnaire; A step by step guide. International Journal of Academic Research in Management (IJARM) 5, 4 (2016), 37–41.Google Scholar
- Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta. 2021. Adapting user interfaces with model-based reinforcement learning. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8, 3-4 (1992), 279–292.Google Scholar
- Danny Weyns, Bradley Schmerl, Vincenzo Grassi, Sam Malek, Raffaela Mirandola, Christian Prehofer, Jochen Wuttke, Jesper Andersson, Holger Giese, and Karl M Göschka. 2013. On patterns for decentralized control in self-adaptive systems. In Software Engineering for Self-Adaptive Systems II. Springer, 76–107.Google Scholar
- Timothy D Wilson and Daniel T Gilbert. 2003. Affective forecasting. (2003).Google Scholar
- Eldad Yechiam, Jerome R Busemeyer, Julie C Stout, and Antoine Bechara. 2005. Using cognitive models to map relations between neuropsychological disorders and human decision-making deficits. Psychological science 16, 12 (2005), 973–978.Google Scholar
- Enes Yigitbas, André Hottung, Sebastian Mansfield Rojas, Anthony Anjorin, Stefan Sauer, and Gregor Engels. 2019. Context-and data-driven satisfaction analysis of user interface adaptations based on instant user feedback. Proceedings of the ACM on Human-Computer Interaction 3, EICS (2019), 1–20.Google ScholarDigital Library
- Mao Zheng, Qian Xu, and Hao Fan. 2016. Modeling The Adaption Rule in Context-aware Systems. arXiv preprint arXiv:1609.01614 (2016).Google Scholar
Recommendations
Emoticontrol: Emotions-based Control of User-Interfaces Adaptations
EICSEmotions are integral to human nature, and their existence, duration, and evolution could lead to specific behaviors. If emotions and behaviors are ignored in the design of socio-technical systems, they will fail or cause discomfort. User interfaces (...
Dealing with Drift of Adaptation Spaces in Learning-based Self-Adaptive Systems Using Lifelong Self-Adaptation
Recently, machine learning (ML) has become a popular approach to support self-adaptation. ML has been used to deal with several problems in self-adaptation, such as maintaining an up-to-date runtime model under uncertainty and scalable decision-making. ...
Emotional Internet of Behaviors: A QoE-QoS Adjustment Mechanism
Artificial Intelligence in HCIAbstractThe Internet of Behaviors (IoB) approach supports developing socio-technical systems based on humans’ goals, characteristics, behaviors, and emotions. This paper shows how emotions and behaviors could impact the quality of software systems. We ...
Comments