Abstract
Because of the COVID-19 pandemic, telework policies have required many user experience (UX) labs to restrict their research activities to remote user testing. Automatic Facial Expression Analysis (AFEA) is an accessible psychophysiological measurement that can be easily implemented in remote user tests. However, to date, the literature on Human Computer Interaction (HCI) has provided no guidelines for remote moderated user tests that collect facial expression data and synchronize them with the state of a dynamic stimulus such as a webpage. To address this research gap, this article offers guidelines for effective AFEA data collection that are based on a methodology developed in a concrete research context and on the lessons learned from applying it in four remote moderated user testing projects. Since researchers have less control over test environment settings, we maintain that they should pay greater attention to factors that can affect face detection and\or emotion classification prior, during, and after remote moderated user tests. Our study contributes to the development of methods for including psychophysiological and neurophysiological measurements in remote user tests that offer promising opportunities for information systems (IS) research, UX design, and even digital health research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Evalyzer, HatchTank, iTracks, Lookback.io, Qualaroo, Recollective, Trymyui, Userlook, Userlytics, Usertesting, UXCam, Validately.
References
vom Brocke, J., Hevner, A., Léger, P.M., Walla, P., Riedl, R.: Advancing a NeuroIS research agenda with four areas of societal contributions. Eur. J. Inf. Syst. 29(1), 9–24 (2020)
Alvarez, J., Brieugne, D., Léger, P.M., Sénécal, S., Frédette, M.: Towards agility and speed in enriched UX evaluation projects. In: Human 4.0-From Biology to Cybernetic, IntechOpen (2019)
Alvarez, J., Léger, P.M., Fredette, M., Chen, S.L., Maunier, B., Senecal, S.: An enriched customer journey map: how to construct and visualize a global portrait of both lived and perceived users’ experiences? Designs 4(3), 29 (2020)
Black, J., Abrams, M.: Remote usability testing. In: The Wiley Handbook of Human Computer Interaction, pp. 277–297 (2018)
Bolt, N., Tulathimutte, T.: Remote Research: Real Users, Real Time, Real Research, 1st edn. Rosenfeld Media, LLC, Brooklyn (2010)
Remote Moderated Usability Tests: How to Do Them. https://www.nngroup.com/articles/moderated-remote-usability-test/. Accessed 06 Feb 2020
Skiendziel, T., Ro, A.G.: Assessing the convergent validity between the automated emotion recognition software Noldus FaceReader 7 and facial action coding system scoring. PLoS One 1–18 (2019)
Ekman, P., Friesen, W.V.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)
Noldus FaceReader methodology. https://info.noldus.com/free-white-paper-on-facereader-methodology. Accessed 15 Jan 2020
Cohen, B.H., Davidson, R.J., Senulis, J.A., Saron, C.D., Weisman, D.R.: Muscle tension patterns during auditory attention. Biol. Psychol. 33, 133–156 (1992)
Peper, E, Gibney, K.H.: muscle biofeedback at the computer: a manual to prevent repetitive strain injury (RSI) by taking the guesswork out of assessment, monitoring, and training. Association for Applied Psychophysiology & Biofeedback (2006)
Léger, P.-M., Courtemanche, F., Fredette, M., Sénécal, S.: A cloud-based lab management and analytics software for triangulated human-centered research. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B. (eds.) Information Systems and Neuroscience. LNISO, vol. 29, pp. 93–99. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-01087-4_11
Courtemanche, F., et al.: Ambient facial emotion recognition: a pilot study. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B., Fischer, T. (eds.) NeuroIS 2020. LNISO, vol. 43, pp. 284–290. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60073-0_33
Courtemanche, F., Léger, P.-M., Dufresne, A., Fredette, M., Labonté-LeMoyne, É., Sénécal, S.: Physiological heatmaps: a tool for visualizing users’ emotional reactions. Multimed. Tools Appl. 77(9), 11547–11574 (2017). https://doi.org/10.1007/s11042-017-5091-1
Courtemanche, F., et al.: U.S. Patent No. 10,368,741. U.S. Patent and Trademark Office, Washington, DC (2019)
Giroux-Huppé, C., Sénécal, S., Fredette, M., Chen, S.L., Demolin, B., Léger, P.-M.: Identifying psychophysiological pain points in the online user journey: the case of online grocery. In: Marcus, A., Wang, W. (eds.) HCII 2019. LNCS, vol. 11586, pp. 459–473. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23535-2_34
Lamontagne, C., et al.: User test: how many users are needed to find the psychophysiological pain points in a journey map? In: Ahram, T., Taiar, R., Colson, S., Choplin, A. (eds.) Human Interaction and Emerging Technologies. IHIET 2019, vol. 1018, pp. 136–142. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25629-6_22
Larson, K., Hazlett, R.L., Chaparro, B.S., Picard, R.W.: Measuring the aesthetics of reading. In: Bryan-Kinns, N., Blanford, A., Curzon, P., Nigay, L. (eds.) People and Computers XX—Engage, pp 41–56. Springer, London (2007). https://doi.org/10.1007/978-1-84628-664-3_4
Kappas, A.: What facial activity can and cannot tell us about emotions. In: Katsikitis, M. (ed.) The human face: Measurement and Meaning, pp. 215–234. Springer, Boston (2003). https://doi.org/10.1007/978-1-4615-1063-5_11
Roy, A., Sénécal, S., Léger, P.-M., Demolin, B., Bigras, É., Gagne, J.: Measuring users’ psychophysiological experience in non-linear omnichannel environment. In: Stephanidis, C., Marcus, A., Rosenzweig, E., Rau, P.-L., Moallem, A., Rauterberg, M. (eds.) HCII 2020. LNCS, vol. 12423, pp. 762–779. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60114-0_50
Veilleux, M., et al.: Visualizing a user’s cognitive and emotional journeys: a fintech case. In: Marcus, A., Rosenzweig, E. (eds.) HCII 2020. LNCS, vol. 12200, pp. 549–566. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49713-2_38
Giroux, F., et al.: Haptic stimulation with high fidelity vibro-kinetic technology psychophysiologically enhances seated active music listening experience. In: 2019 IEEE World Haptics Conference, pp. 151–156 (2019)
Giroux, F., Boasen, J., Sénécal, S., Léger, P.-M.: Hedonic multitasking: the effects of instrumental subtitles during video watching. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B., Fischer, T. (eds.) NeuroIS 2020. LNISO, vol. 43, pp. 330–336. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60073-0_38
Falconnet, A., et al.: Beyond system design: the impact of message design on recommendation acceptance. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B., Fischer, T. (eds.) NeuroIS 2020. LNISO, vol. 43, pp. 185–190. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60073-0_21
Beauchesne, A., et al.: User-centered gestures for mobile phones: exploring a method to evaluate user gestures for UX designers. In: Marcus, A., Wang, W. (eds.) HCII 2019. LNCS, vol. 11584, pp. 121–133. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23541-3_10
Brissette-Gendron, R., Léger, P.M., Courtemanche, F., Chen, S.L., Ouhnana, M., Sénécal, S.: The response to impactful interactivity on spectators’ engagement in a digital game. Multimodal Technol. Interact. 4(4), 89 (2020)
Le Pailleur, F., Huang, B., Léger, P.-M., Sénécal, S.: A new approach to measure user experience with voice-controlled intelligent assistants: a pilot study. In: Kurosu, M. (ed.) HCII 2020. LNCS, vol. 12182, pp. 197–208. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49062-1_13
Passalacqua, M., et al.: Playing in the backstore: interface gamification increases warehousing workforce engagement. Ind. Manage. Data Syst. (2020)
Ruer, P., et al.: Improving driving behavior with an insurance telematics mobile application. In: Davis, F.D., Riedl, R., vom Brocke, J., Léger, P.-M., Randolph, A.B., Fischer, T. (eds.) NeuroIS 2020. LNISO, vol. 43, pp. 198–203. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60073-0_23
Vasseur, A., et al.: Distributed remote psychophysiological data collection for UX evaluation: a pilot project. In: International Conference on Human-Computer Interaction (forthcoming)
Demazure, T., et al.: Distributed remote EEG data collection for NeuroIS research: a methodological framework. In: International Conference on Human-Computer Interaction (forthcoming)
Riedl, R., Davis, F.D., Leger, P.M., Muller-Putz, G.: NeuroIS: status, rewarding research questions, and future directions. practice development workshop. In: Forty-First International Conference on Information Systems, India (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Giroux, F. et al. (2021). Guidelines for Collecting Automatic Facial Expression Detection Data Synchronized with a Dynamic Stimulus in Remote Moderated User Tests. In: Kurosu, M. (eds) Human-Computer Interaction. Theory, Methods and Tools. HCII 2021. Lecture Notes in Computer Science(), vol 12762. Springer, Cham. https://doi.org/10.1007/978-3-030-78462-1_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-78462-1_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78461-4
Online ISBN: 978-3-030-78462-1
eBook Packages: Computer ScienceComputer Science (R0)