Abstract
To analyze successful consulting processes using multimodal analysis, the aim of this research is to develop a model for recognizing when a client is persuaded by a consultant using multimodal features. These models enable us to analyze the utterances of highly skilled professional consultants in persuading clients. For this purpose, first, we collect a multimodal counseling interaction corpus including audio and spoken dialogue content (manual transcription) on dialogue sessions between a professional beauty counselor and five clients. Second, we developed a recognition model of persuasion labels using acoustic and linguistic features that are extracted from a multimodal corpus by training a machine learning model as a binary classification task. The experimental results show that the persuasion was 0.697 for accuracy and 0.661 for F1-score with bidirectional LSTM.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)
Chen, L., Feng, G., Joe, J., Leong, C.W., Kitchen, C., Lee, C.M.: Towards automated assessment of public speaking skills using multimodal cues. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 200–203 (2014)
Cortes, C., Vapnik, V.: Support vector networks. Mach. Learn. 20, 273–297 (1995)
DeVault, D., et al.: SimSensei kiosk: a virtual human interviewer for healthcare decision support. In: Proceedings of the International Conference on Autonomous Agents and Multi-agent Systems, pp. 1061–1068 (2014)
Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18, 602–610 (2005)
Higashiyama, M., Inui, K., Matsumoto, Y.: Learning sentiment of nouns from selectional preferences of verbs and adjectives. In: Proceedings of the Annual Meeting of the Association for Natural Language Processing, pp. 584–587 (2008)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hoque, M.E., Courgeon, M., Martin, J.-C., Mutlu, B., Picard, R.W.: MACH: My automated conversation coach. In: Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 697–706 (2013)
Ishii, R., Otsuka, K., Kumano, S., Higashinaka, R., Tomita, J.: Analyzing gaze behavior and dialogue act during turn-taking for estimating empathy skill level. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 31–39 (2018)
Kingma, D., Adam, J.B.: A method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations, pp. 1–15 (2015)
Nozomi, K., Kentaro, I., Yuji, M., Kenji, T.: Collecting evaluative expressions for opinion extraction. J. Nat. Lang. Process. 12(3), 203–222 (2005)
Kudo, T., Yamamoto, K., Matsumoto, Y.: Applying conditional random fields to Japanese morphological analysis. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 230–237 (2004)
Nair, V., Hinton, G.E.: Rectified linear units improve Restricted Boltzmann machines. In: Proceedings of the International Conference on Machine Learning, pp. 807–814 (2010)
Nguyen, L.S., Frauendorfer, D., Mast, M.S., Gatica-Perez, D.: Hire me: Computational inference of hirability in employment interviews based on nonverbal behavior. IEEE Trans. Multimed. 16(4), 1018–1031 (2014)
Okada, S., et al.: Estimating communication skills using dialogue acts and nonverbal features in multiple discussion datasets. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 169–176 (2016)
Park, S., Shim, H.S., Chatterjee, M., Sagae, K., Morency, L.-P.: Computational analysis of persuasiveness in social multimedia: a novel dataset and multimodal prediction approach. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 50–57 (2014)
Ramanarayanan, V., Leong, C.W., Chen, L., Feng, G., Suendermann-Oeft, D.: Evaluating speech, face, emotion and body movement time-series features for automated multimodal presentation scoring. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 23–30 (2015)
Sanchez-Cortes, D., Aran, O., Mast, M.S., Gatica-Perez, D.: A nonverbal behavior approach to identify emergent leaders in small groups. IEEE Trans. Multimed. 14, 816–832 (2012)
Schuller, B., Steidl, S., Batliner, A., Burkhardt, F., Devillers, L., Müller, C., Narayanan, S.: The interspeech 2010 paralinguistic challenge. In: Proceedings of the Annual Conference of the International Speech Communication Association, pp. 2794–2797 (2010)
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Suzuki, M., Matsuda, K., Sekine, S., Okazaki, N., Inui, K.: A joint neural model for fine-grained named entity classification of wikipedia articles. IEICE Trans. Inf. Syst. E101.D(1), 73–81 (2018)
Tanaka, H., Negoro, H., Iwasaka, H., Nakamura, S.: Listening skills assessment through computer agents. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 492–496 (2018)
Tanaka, H., et al.: Automated social skills trainer. In: Proceedings of the ACM International Conference on Inteligent User Interface, pp. 17–27 (2015)
Tavabi, L., Stefanov, K., Gilani, S.N., Traum, D., Soleymani, M.: Multimodal learning for identifying opportunities for empathetic responses. In: Proceedings of the ACM International Conference on Multimodal Interaction, pp. 95–104 (2019)
Tianqi, C., Carlos, G.: XGBoost: a scalable tree boosting system. In: Proceedings of the International Conference on Knowledge Discovery and Data Mining, pp. 785–794 (2016)
Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: survey of an emerging domain. Image Vis. Comput. 27(12), 1743–1759 (2009)
Wörtwein, T., Chollet, M., Schauerte, B., Morency, L.-P., Stiefelhagen, R., Scherer, S.: Multimodal public speaking performance assessment. In: Proceedings of ACM International Conference on Multimodal Interaction, pp. 43–50 (2015)
Xiao, B., Imel, Z.E., Georgiou, P., Atkins, D.C., Narayanan, S.S.: Computational analysis and simulation of empathic behaviors: a survey of empathy modeling with behavioral signal processing framework. Curr. Psych. Rep. 18(5), 1–11 (2016)
Xiao, B., Imel, Z.E., Georgiou, P.G., Atkins, D.C., Narayanan, S.S.: “rate my therapist”: Automated detection of empathy in drug and alcohol counseling via speech and language processing. PLOS ONE 10(12), 1–15 (2015)
Acknowledgements
We sincerely appreciate Be \(\cdot \) Fine Co. ltd. and Ms. Teruko Kobayashi who is the professional beauty counselor.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Amari, Y., Okada, S., Matsumoto, M., Sadamitsu, K., Nakamoto, A. (2021). Multimodal Analysis of Client Persuasion in Consulting Interactions: Toward Understanding Successful Consulting. In: Meiselwitz, G. (eds) Social Computing and Social Media: Applications in Marketing, Learning, and Health. HCII 2021. Lecture Notes in Computer Science(), vol 12775. Springer, Cham. https://doi.org/10.1007/978-3-030-77685-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-77685-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77684-8
Online ISBN: 978-3-030-77685-5
eBook Packages: Computer ScienceComputer Science (R0)