Berardina De Carolis
Marco de Gemmis
ABSTRACT
A conversational recommender system should interactively assist users in order to understand their needs and preferences and produce personalized recommendations accordingly. While traditional recommender systems use a single-shot approach, the conversational ones refine their suggestions during the conversation since they gain more knowledge about the user. This approach can be useful in case the recommender is embodied in a conversational agent acting as a shopping assistant in a smart retail context. In this case, knowledge about the user preferences may be acquired during the conversation and by observing the user behavior. In such a setting, besides "rational" information, the agent may grasp information also about extra-rational factors such as attitudes, emotions, likes and dislikes. This paper describes the study performed in order to develop a multimodal framework for recognizing the shopping attitude of the user during the interaction with DIVA, a Dress-shopping InteractiVe Assistant. In particular, speech prosody, gestures and facial expressions have been taken into account for providing feedback to the system and refining the recommendation accordingly.
- http://perso.telecom-paristech.fr/~pelachau/Greta/Google Scholar
- I. Poggi. Mind markers. In N. T. M. Rector, I. Poggi, editor, Gestures. Meaning and use. University Fernando Pessoa Press, Oporto, Portugal, 2003.Google Scholar
- James R. Bettman, Mary F. Luce, and John W. Payne, "Constructive consumer choice processes," Journal of consumer research, vol. 25, no. 3, pp. 187--217, 1998.Google Scholar
- Robert J. Donovan, John R. Rossiter, Gilian Marcoolyn, and Nesdale Andrew, "Store atmosphere and purchasing behavior," Journal of Retailing, vol. 70, no. 3, pp. 283--294, 1994.Google Scholar
- M. Tkalčič, U. Burnik, A. Odic, A. Koßsir, and J. Tasic, "Emotion-aware recommender systems-a framework and a case study," ICT Innovations 2012: Secure and Intelligent Systems, vol. 207, pp. 141--150, 2013.Google Scholar
- Karen A. Machleit, Susan Powell Mantel. (2001). Emotional response and shopping satisfaction. Moderating effects of shopper attributions. Journal of Business Research 54. Elsevier Science Inc. pp: 97--106Google Scholar
- Jensen, F.V. Bayesian Networks and Decision Graphs. Springer. (2001). Google ScholarDigital Library
- Reeves, B. and Nass, C. The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge: Cambridge University Press. (1996). Google ScholarDigital Library
- Reilly WSN (1996) Believable Social and Emotional Agents. PhD thesis.Google Scholar
- Bickmore T (2003) Relational Agents: Effecting Change through Human-Computer Relationships. PhD Thesis, Media Arts & Sciences, Massachusetts Institute of Technology.Google Scholar
- Qiu, L., & Benbasat, I. (2009). Evaluating Anthropomorphic Product Recommendation Agents: A Social Relationship Perspective to Designing Information Systems. Journal of Management Information Systems. Google ScholarDigital Library
- Mumme, C., Pinkwart, N., Loll, F.: Design and Implementation of a Virtual Salesclerk. In Proceedings of IVA 2009, 379--385. Springer, Heidelberg (2009) Google ScholarDigital Library
- B. De Carolis, I. Mazzotta, N. Novielli, S. Pizzutilo (2013). User Modeling in Social Interaction with a Caring Agent. User Modeling and Adaptation for Daily Routines Human--Computer Interaction Series 2013, pp 89--116Google ScholarCross Ref
- R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, J. Taylor (2001). Emotion Recognition in Human-Computer Interaction. IEEE Signal Processing Magazine January 2001. 32--8Google Scholar
- Vinciarelli, A., Pantic, M., Bourlard, H., and Pentland, A. (2008) Social signals, their function, and automatic analysis: a survey. In: ICMI '08 Proceedings of the 10th International Conference on Multimodal Interfaces, October 20--22, 2008, Chania, Greece. Google ScholarDigital Library
- De Carolis, B., Novielli N., Recognizing signals of social attitude in interacting with Ambient Conversational Systems, Journal on Multimodal User Interfaces, March 2014, Volume 8, Issue 1, pp 43--60.Google ScholarCross Ref
- Zhihong Zeng, Maja Pantic, Glenn I. Roisman, Thomas S. Huang: A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1): 39--58 (2009) Google ScholarDigital Library
- Pantic, M., Rothkrantz, L.J.M. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, Volume: 91 Issue: 9, Sept. 2003. Page(s): 1370--1390.Google ScholarCross Ref
- G. Caridakis, A. Raouzaiou, K. Karpouzis, S. Kollias, Synthesizing Gesture Expressivity Based on Real Sequences, Workshop on multimodal corpora: from multimodal behaviour theories to usable models, LREC Conference, Genoa, Italy, 24--26 May, 2006.Google Scholar
- Wagner J, Lingenfelser F, André E (2011) The social signal interpretation framework (SSI) for real time signal processing and recognition. In: Proceedings of Interspeech 2011Google Scholar
- Bruno Lepri, Nadia Mana, Alessandro Cappelletti, Fabio Pianesi, Massimo Zancanaro: Modeling the Personality of Participants During Group Interactions. UMAP 2009: 114--125. Google ScholarDigital Library
- Candace L. Sidner, Cory D. Kidd, Christopher Lee, and Neal Lesh. 2004. Where to look: a study of human-robot engagement. In Proceedings of the 9th international conference on Intelligent user interfaces (IUI '04). ACM, New York, NY, USA, 78--84. DOI=10.1145/964442.964458 http://doi.acm.org/10.1145/964442.964458 Google ScholarDigital Library
- Nakano YI, Ishii R (2010) Estimating user's engagement from eye-gaze behaviors in human-agent conversations. In: 2010 international conference on intelligent user interfaces (IUI2010) Google ScholarDigital Library
- L. C. De Silva and P. C. Ng, "Bimodal emotion recognition," in IEEE International Conf. on Automatic Face and Gesture Recognition, March 2000, pp. 332--335. Google ScholarDigital Library
- L. S. Chen and T. S. Huang, "Emotional expressions in audiovisual human computer interaction," in International Conf. on Multimedia and Expo (ICME), 2000, pp. 423--426.Google Scholar
- N. Sebe, I. Cohen, T. Gevers, and T.S. Huang, "Emotion Recognition Based on Joint Visual and Audio Cues," Proc. 18th Int'l Conf. Pattern Recognition (ICPR '06), pp. 1136--1139, 2006. Google ScholarDigital Library
- T. Balomenos, A. Raouzaiou, S. Ioannou, A. Drosopoulos, K. Karpouzis, and S. Kollias. Emotion Analysis in Man-Machine Interaction Systems, pages 175{200. 3D Modeling and Animation: Synthesis and Analysis Techniques, Idea Group Publ., 2005.Google Scholar
- Hatice Gunes and Massimo Piccardi. 2007. Bi-modal emotion recognition from expressive face and body gestures. J. Netw. Comput. Appl. 30, 4 (November 2007), 1334--1345. DOI=10.1016/j.jnca.2006.09.007 http://dx.doi.org/10.1016/j.jnca.2006.09.007 Google ScholarDigital Library
- G. Castellano, L. Kessous, G. Caridakis, "Multimodal emotion recognition from expressive faces, body gestures and speech", C. Peter, R. Beale (eds), Affect and Emotion in Human-Computer Interaction, Lecture Notes in Computer Science, Springer.Google Scholar
- Ekman, P. & Friesen, W. V. 1969 The repertoire of nonverbal behavior. Semiotica 1, 49--98.Google ScholarCross Ref
- Ambady, N. & Rosenthal, R. 1992 Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychol. Bull. 111, 256--274.Google ScholarCross Ref
- Whittaker S., Walker M., and Moore J., Fish or Fowl: A Wizard of Oz evaluation of dialogue strategies in the restaurant domain. Language Resources and Evaluation Conference. (2002).Google Scholar
- Russell J.A. A circumplex model of affect. Journal of Personality and Social Psychology.1980;39:1161--1178.Google Scholar
- Sundberg, J., Patel, S., Björkner, E., Scherer, K.R. (2011). Interdependencies among voice source parameters in emotional speech. IEEE Transactions on Affective Computing, 2(3). Google ScholarDigital Library
- Feldman, R.S. (1992). Applications of Nonverbal Behavioral Theories and Research. New Jersey: Lawrence Erlbaum Associates, Inc.Google Scholar
- Knapp, M. and Hall, J. (1992). Nonverbal communication in human interaction. Orlando, FL: Holt, Rinehart&Winsten, Inc.Google Scholar
- Prendinger H., Mori J. and Ishizuka, M. (2005). Recognizing, modeling, and responding to users affective states. In Proc. of User Modeling 2005, Lecture Notes in Computer Science, Volume 3538/2005, 149. Google ScholarDigital Library
- Conati C. (2002) Probabilistic assessment of user's emotions in educational games. Applied Artificial Intelligence, 16:555--575.Google ScholarCross Ref
- Vogt, T., Adre' E., and Bee N. 2008. EmoVoice - A Framework for Online Recognition of Emotions from Voice. In Proceedings of the 4th IEEE workshop on Perception and Interactive Technologies for Speech-Based Systems: Perception in Multimodal Dialogue Systems (PIT '08), Springer-Verlag, Berlin, Heidelberg, 188--199. Google ScholarDigital Library
- Boersma, P. and Weenink, D. (2007). Praat: doing phonetics by computer (version 4.5.15) {computer program}.Google Scholar
- KinectDTW: http://kinectdtw.codeplex.com/Google Scholar
- Y. Zheng, B. Mobasher, and R. D. Burke. The role of emotions in context-aware recommendation. Proceedings of the 3rd Workshop on Human Decision Making in Recommender Systems, in conjunction with the 7th ACM Conference on Recommender Systems (RecSys 2013), vol. 1050 of CEUR Workshop Proceedings, pp.21--28. CEURWS.org, 2013.Google Scholar
Index Terms
- A Multimodal Framework for Recognizing Emotional Feedback in Conversational Recommender Systems
Recommendations
Recognizing users feedback from non-verbal communicative acts in conversational recommender systems
A multimodal framework for monitoring users attitude and profile.A dynamic model for monitoring user liking attitude.We integrated the module in a content-based recommender system. Conversational recommender systems produce personalized recommendations ...
Unifying Recommender Systems and Conversational User Interfaces
CUI '22: Proceedings of the 4th Conference on Conversational User InterfacesThis paper considers unifying research on conversational user interfaces and recommender systems. Studies on conversational user interfaces (CUIs) typically examine how conversations can be facilitated (i.e., optimizing the means). Recommender systems ...
Initiative transfer in conversational recommender systems
RecSys '23: Proceedings of the 17th ACM Conference on Recommender SystemsConversational recommender systems (CRS) are increasingly designed to offer mixed-initiative dialogs in which the user and the system can take turns in starting a communicative exchange, for example, by asking questions or stating preferences. However, ...
Comments