Abstract
Much was learned in the development of an emotion sensing virtual standardized patient (VSP) which employs various inputs to obtain an emotional read and VSP-oriented judgment of the human participant during a simulated medical interview. Such a virtual patient changes behavior based on its sensing and alters its behavior and degree of honest disclosure to the human ‘doctor’ accordingly. We have discovered several important human factor and technology strategies to produce practical and impactful affective simulations. It is important to recognize overpromises of AI vendors and fallacies associated with affective computing. Essential technologies for affective applications include natural language understanding (NLU), computer vision, auditory processing of speech tonal characteristics and contextual assessments. Important factors for effective sensing include methods to identify context of the conversation in progress, a model of the VSP’s emotional state and use of evocative stimuli to provoke an emotional reaction when conducting measurements.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Talbot, T.B., Hackett, M.: Perceptive patient: a multimodal sensing virtual standardized patient experiment to create an emotionally judging ai-based patient responsive to non-verbal behaviors and other affective indicators. International Meeting on Simulation in Healthcare (2020)
Hamm, J., Kohler, C.G., Gur, R.C., Verma, R.: Automated Facial Action Coding System for dynamic analysis of facial expressions in neuropsychiatric disorders. J. Neurosci. Methods 200(2), 237–56. doi:https://doi.org/10.1016/j.jneumeth.2011.06.023. PMC 3402717. PMID 21741407
Tausczik, Y.R., Pennebaker, J.W.: The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29(1), 24–54 (2010). https://doi.org/10.1177/0261927X09351676
Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)
Tian, Y., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97–115 (2001)
Pantic, M.: Machine analysis of facial behaviour: naturalistic and dynamic behavior. Philosoph. Trans. Roy. Soc. B: Biol. Sci. 364, 3505–3513 (2009)
Giannakopoulos, T.: pyAudioAnalysis: an open-source python library for audio signal analysis. PLoS ONE 10(12), e0144610 (2015). https://doi.org/10.1371/journal.pone.0144610
Tawari, A., Trivedi, M.M.: Speech emotion analysis: exploring the role of context. IEEE Trans. Multimedia 12(6), 502–509 (2010). https://doi.org/10.1109/TMM.2010.2058095
Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L.: OpenFace 2.0: facial behavior analysis toolkit. In: 13th IEEE International Conference on Automatic Face & Gesture Recognition Xi'an, 2018, pp. 59–66, (2018). https://doi.org/10.1109/FG.2018.00019
Stratou, G., Morency, L.P.: MultiSense—context-aware nonverbal behavior analysis framework: a psychological distress use case. IEEE Trans. Affect. Comput. 8(2), 190–203 (2017). https://doi.org/10.1109/TAFFC.2016.2614300
Acknowledgments
The project or effort depicted was or is sponsored by the U.S. Army Research Laboratory (ARL) under contract W911NF-16-C-0034, and that the content of the information does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Talbot, T.B., Hackett, M., Pike, W. (2021). Perceptive Patient: Important Factors for Practical Emotion Sensing in Conversational Human-Computer. In: Wright, J.L., Barber, D., Scataglini, S., Rajulu, S.L. (eds) Advances in Simulation and Digital Human Modeling. AHFE 2021. Lecture Notes in Networks and Systems, vol 264. Springer, Cham. https://doi.org/10.1007/978-3-030-79763-8_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-79763-8_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-79762-1
Online ISBN: 978-3-030-79763-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)