Ricardo Cordeiro Galvão Sant'Ana van Erven
Edna Dias Canedo
ABSTRACT
Context: Measuring user satisfaction is an essential tool to create value in digital transformation. Measurement allows you to identify future purchase intentions, user loyalty, and retention. The traditional way of measuring satisfaction using self-assessment has problems, such as subjectivity. Therefore, a more objective approach is needed that allows automatic measurement of satisfaction. Objective: This research aims to make a systematic literature mapping (SLM) to identify the state-of-the-art use of Artificial Intelligence tools and techniques related to emotion recognition visually and then use the results to propose a model that can lead to an automated user satisfaction measurement. We present the model with a work process, an exploratory experiment, and an application project to measure the user experience from visual emotion recognition, automatically calculate the satisfaction assessment of a user when using a digital product, and present the positive, neutral, and negative points of using this product. Methods: A systematic literature mapping was used to identify the primary studies related to techniques, benefits, and challenges that associate the measurement of user satisfaction through the recognition of emotions with the use of Artificial Intelligence (AI). In the proposed model, an experiment was planned in two phases to validate research hypotheses related to the objective of the work. The experiment, which will apply the projected application, will be carried out by videoconferencing with the approach of exploratory usability testing on a web product. Results: 10 primary studies were identified in different areas of knowledge: restaurants, television systems, retail stores, artistic shows, and usability testing in a call center system. Two systematic reviews of the literature were also identified. The technique most commonly used in primary studies is the convolutional neural network (CNN). The use of cloud services for emotion recognition was also verified. Benefits related to user feedback, such as user profile mapping, were reported, and challenges for emotion recognition were found, such as user privacy and capture environment inadequacies. Besides, a model to automatically measure user satisfaction in a scalable and replicable way was proposed. Conclusions: The possibilities of applying emotion recognition are countless in terms of contexts, techniques, forms, and components. Despite this, it was possible to identify good practices that could guide the creation of a tool to measure the satisfaction of using a digital product through emotion recognition and Artificial Intelligence. The main contribution of this work is the proposal of a model that guides the creation of an automatic satisfaction measurement tool to identify positive, neutral, and negative points in the use of a digital product. This research also defines a measurement experimentation process that any organization or researcher can reproduce.
- Sanket Agrawal, Rucha Rangnekar, Aditya Das, Shantanu Gawde, and Sudhir Dhage. 2019. Gauging Customer Interest Using Skeletal Tracking and Convolutional Neural Network. In 2019 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT). Institute of Electrical and Electronics Engineers Inc., Coimbatore, Tamil Nadu, 1–6. https://doi.org/10.1109/ICECCT.2019.8869045Google Scholar
Cross Ref
- Ali Alawneh, Hasan Al-Refai, and Khaldoun Batiha. 2013. Measuring user satisfaction from e-Government services: Lessons from Jordan. Government Information Quarterly 30, 3 (7 2013), 277–288. https://doi.org/10.1016/j.giq.2013.03.001Google ScholarCross Ref
- Jessie R. Balbin, Charmaine C. Paglinawan, Mary Josanne A. De Castro, Jared Kobe C. Llamas, Mikka Ellah T. Medina, John Jomel O. Pangilinan, and Flordeliza L. Valiente. 2019. Augmented Reality Aided Analysis of Customer Satisfaction based on Taste-Induced Facial Expression Recognition Using Affdex Software Developer’s Kit. In ACM International Conference Proceeding Series. Association for Computing Machinery, Tokyo, 204–209. https://doi.org/10.1145/3326172.3326221Google ScholarDigital Library
- Boris Bartikowski and Sylvie Llosa. 2004. Customer satisfaction measurement: Comparing four methods of attribute categorisations. Service Industries Journal 24, 4 (7 2004), 67–82. https://doi.org/10.1080/0264206042000275190Google ScholarCross Ref
- A. Birch-Jensen, I. Gremyr, J. Hallencreutz, and Rönnbäck. 2020. Use of customer satisfaction measurements to drive improvements. Total Quality Management and Business Excellence 31, 5-6 (4 2020), 569–582. https://doi.org/10.1080/14783363.2018.1436404Google ScholarCross Ref
- U. Bititci, P. Garengo, V. Dörfler, and S. Nudurupati. 2012. Performance Measurement: Challenges for Tomorrow. International Journal of Management Reviews 14, 3 (2012), 305–327. https://doi.org/10.1111/j.1468-2370.2011.00318.xGoogle ScholarCross Ref
- Simone Borsci, Robert D Macredie, Julie Barnett, Jennifer Martin, Jasna Kuljis, and Terry Young. 2013. Reviewing and Extending the Five-User Assumption: A Grounded Procedure for Interaction Evaluation. ACM Trans. Comput.-Hum. Interact. 20, 5 (11 2013), 1–23. https://doi.org/10.1145/2506210Google ScholarDigital Library
- Silvia Ceccacci, Andrea Generosi, Luca Giraldi, and Maura Mengoni. 2023. Emotional Valence from Facial Expression as an Experience Audit Tool: An Empirical Study in the Context of Opera Performance. Sensors 23, 5 (3 2023), 2688. https://doi.org/10.3390/s23052688Google ScholarCross Ref
- Michela Chimienti, Ivan Danzi, Vincenzo Gattulli, Donato Impedovo, Giuseppe Pirlo, and Davide Veneto. 2022. Behavioral Analysis for User Satisfaction. In Proceedings - 2022 IEEE 8th International Conference on Multimedia Big Data, BigMM 2022. Institute of Electrical and Electronics Engineers Inc., Virtual, Online, 113–119. https://doi.org/10.1109/BigMM55396.2022.00027Google ScholarCross Ref
- Wansuk Choi, Taeseok Choi, and Seoyoon Heo. 2023. A Comparative Study of Automated Machine Learning Platforms for Exercise Anthropometry-Based Typology Analysis: Performance Evaluation of AWS SageMaker, GCP VertexAI, and MS Azure. Bioengineering 10, 8 (8 2023), 891. https://doi.org/10.3390/bioengineering10080891Google ScholarCross Ref
- R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J.G. Taylor. 2001. Emotion recognition in human-computer interaction. IEEE Signal Processing Magazine 18, 1 (2001), 32–80. https://doi.org/10.1109/79.911197Google ScholarCross Ref
- Udaya Dampage, D. A. Egodagamage, A. U. Waidyaratne, D. A.W. DIssanayaka, and A. G.N.M. Senarathne. 2021. Spatial Augmented Reality Based Customer Satisfaction Enhancement and Monitoring System. IEEE Access 9 (2021), 97990–98004. https://doi.org/10.1109/ACCESS.2021.3093829Google ScholarCross Ref
- T. Dybå, T. Dingsøyr, and G.K. Hanssen. 2007. Applying systematic reviews to diverse study types: An experience report. In Proceedings - 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 2007. Institute of Electrical and Electronics Engineers Inc., Madrid, Spain, 225–234. https://doi.org/10.1109/ESEM.2007.21Google ScholarDigital Library
- P. Ekman. 1992. Are There Basic Emotions?Psychological Review 99, 3 (1992), 550–553. https://doi.org/10.1037/0033-295X.99.3.550Google ScholarCross Ref
- Camila Favoretto, Glauco Henrique de Sousa Mendes, Moacir Godinho Filho, Maicon Gouvea de Oliveira, and Gilberto Miller Devós Ganga. 2022. Digital transformation of business model in manufacturing companies: challenges and research agenda., 748–767 pages. https://doi.org/10.1108/JBIM-10-2020-0477Google ScholarCross Ref
- Markus Gahler, Jan F. Klein, and Michael Paul. 2022. Customer Experience: Conceptualization, Measurement, and Application in Omnichannel Environments. Journal of Service Research 26, 2 (5 2022), 191–211. https://doi.org/10.1177/10946705221126590Google ScholarCross Ref
- Shantanu Godbole and Shourya Roy. 2008. Text to Intelligence: Building and Deploying a Text Mining Solution in the Services Industry for Customer Satisfaction Analysis. In 2008 IEEE International Conference on Services Computing, Vol. 2. Institute of Electrical and Electronics Engineers Inc., Honolulu, HI, USA, 441–448. https://doi.org/10.1109/SCC.2008.99Google ScholarDigital Library
- Sari Kujala, Virpi Roto, Kaisa Väänänen-Vainio-Mattila, Evangelos Karapanos, and Arto Sinnelä. 2011. UX Curve: A method for evaluating long-term user experience. Interacting with Computers 23, 5 (2011), 473–483. https://doi.org/10.1016/j.intcom.2011.06.005Google ScholarDigital Library
- Agnieszka Landowska. 2015. Towards emotion acquisition in IT usability evaluation context. In ACM International Conference Proceeding Series, Vol. 29-30-Jun-2015. Association for Computing Machinery, Warsaw Poland, 1–9. https://doi.org/10.1145/2814464.2814470Google ScholarDigital Library
- Heloise Acco Tives Leão and Edna Dias Canedo. 2018. Best Practices and Methodologies to Promote the Digitization of Public Services Citizen-Driven: A Systematic Literature Review. Inf. 9, 8 (2018), 197. https://doi.org/10.3390/info9080197Google ScholarCross Ref
- J.-S. Lee and D.-H. Shin. 2014. The relationship between human and smart TVs based on emotion recognition in HCI. Vol. 8582 LNCS. Springer Verlag, Guimaraes, PORTUGA. 652–667 pages. https://doi.org/10.1007/978-3-319-09147-1_47Google ScholarCross Ref
- L. Li. 2013. Study on the interactive relationship between customer’s emotional response and the brand trust - In the view of online shopping. In Proceedings of 2013 IEEE International Conference on Service Operations and Logistics, and Informatics, SOLI 2013. Institute of Electrical and Electronics Engineers Inc., Dongguan, PEOPLES R CHINA, 245–248. https://doi.org/10.1109/SOLI.2013.6611418Google ScholarCross Ref
- Rensis Likert. 1932. A technique for the measurement of attitudes.Archives of psychology 22, 140 (1932), 5–55.Google Scholar
- Anna Beatriz Marques, Alex Felipe Costa, Ismayle Santos, and Rossana Andrade. 2023. Enriching User Stories with Usability Features in a Remote Agile Project: A Case Study. In Proceedings of the XXI Brazilian Symposium on Software Quality (Curitiba, Brazil) (SBQS ’22). Association for Computing Machinery, New York, NY, USA, Article 24, 10 pages. https://doi.org/10.1145/3571473.3571496Google ScholarDigital Library
- Eleonora Pantano, Charles Dennis, and Eleftherios Alamanos. 2022. Retail Managers’ Preparedness to Capture Customers’ Emotions: A New Synergistic Framework to Exploit Unstructured Data with New Analytics. British Journal of Management 33, 3 (7 2022), 1179–1199. https://doi.org/10.1111/1467-8551.12542Google ScholarCross Ref
- Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64 (8 2015), 1–18. https://doi.org/10.1016/j.infsof.2015.03.007Google ScholarDigital Library
- M. Petticrew and H. Roberts. 2008. Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing Ltd, Published Online. 1–336 pages. https://doi.org/10.1002/9780470754887Google ScholarCross Ref
- R.W. Picard. 2010. Affective Computing: From laughter to IEEE. IEEE Transactions on Affective Computing 1, 1 (2010), 11–17. https://doi.org/10.1109/T-AFFC.2010.10Google ScholarDigital Library
- R.W. Picard, E. Vyzas, and J. Healey. 2001. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 10 (2001), 1175–1191. https://doi.org/10.1109/34.954607Google ScholarDigital Library
- Priti Rai Jain, S. M.K. Quadri, and Muskan Lalit. 2021. Recent trends in artificial intelligence for emotion detection using facial image analysis. In ACM International Conference Proceeding Series. Association for Computing Machinery, Noida, India, 18–36. https://doi.org/10.1145/3474124.3474205Google ScholarDigital Library
- Shourya Roy, Ragunathan Mariappan, Sandipan Dandapat, Saurabh Srivastava, Sainyam Galhotra, and Balaji Peddamuthu. 2016. QART: A System for Real-Time Holistic Quality Assurance for Contact Center Dialogues. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence(AAAI’16). AAAI Press, Phoenix, 3768–3775.Google Scholar
- Mohammad Soleymani, David Garcia, Brendan Jou, Björn Schuller, Shih Fu Chang, and Maja Pantic. 2017. A survey of multimodal sentiment analysis. Image and Vision Computing 65 (9 2017), 3–14. https://doi.org/10.1016/j.imavis.2017.08.003Google ScholarCross Ref
- X. Tian, W. Hou, and K. Yuan. 2008. A study on the method of satisfaction measurement based on emotion space. In 9th International Conference on Computer-Aided Industrial Design and Conceptual Design: Multicultural Creation and Design - CAIDCD 2008. Institute of Electrical and Electronics Engineers Inc., Beijing, China, 39–43. https://doi.org/10.1109/CAIDCD.2008.4730515Google ScholarCross Ref
- Peter C. Verhoef, Thijs Broekhuizen, Yakov Bart, Abhi Bhattacharya, John Qi Dong, Nicolai Fabian, and Michael Haenlein. 2021. Digital transformation: A multidisciplinary reflection and research agenda. Journal of Business Research 122 (1 2021), 889–901. https://doi.org/10.1016/j.jbusres.2019.09.022Google ScholarCross Ref
- André Ximenes, Carina Alves, Jéssyka Vilela, and Iveruska Jatobá. 2022. A Competency Management Model to Support Digital Transformation Initiatives in a Public Organization. In Anais do XXI Simpósio Brasileiro de Qualidade de Software (Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 209–218. https://sol.sbc.org.br/index.php/sbqs/article/view/23307Google Scholar
- Zhihong Zeng, Maja Pantic, Glenn I. Roisman, and Thomas S. Huang. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 1 (2009), 39–58. https://doi.org/10.1109/TPAMI.2008.52Google ScholarDigital Library
Index Terms
- Measurement of User's Satisfaction of Digital Products through Emotion Recognition
Recommendations
Emotion recognition in doctor-patient interactions from real-world clinical video database: Initial development of artificial empathy
Highlights- The first real-world-based video-recorded corpus of doctor-patient interactions in clinical practices.
- Our database combines objective and subjective evaluation based on doctor-patient interactions.
- Artificial empathy based on ...
Abstract Background and objectiveThe promising use of artificial intelligence (AI) to emulate human empathy may help a physician engage with a more empathic doctor-patient relationship. This study demonstrates the application of artificial empathy based ...
Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations
AbstractEmotion recognition is the ability to precisely infer human emotions from numerous sources and modalities using questionnaires, physical signals, and physiological signals. Recently, emotion recognition has gained attention because of its diverse ...
Highlights- A systematic review of emotion recognition comprising journal articles of the last decade culled from renowned databases.
- Inclusion of physical and physiological signals used for emotion recognition using artificial intelligence.
- A ...
EmotiW 2016: video and group-level emotion recognition challenges
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal InteractionThis paper discusses the baseline for the Emotion Recognition in the Wild (EmotiW) 2016 challenge. Continuing on the theme of automatic affect recognition `in the wild', the EmotiW challenge 2016 consists of two sub-challenges: an audio-video based ...
Comments