Skip to main content
SpringerLink
Log in

HMM modeling of user engagement in advice-giving dialogues

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

This research aims at defining a real-time probabilistic model of user’s engagement in advice-giving dialogues. We propose an approach based on Hidden Markov Models (HMMs) to describe the differences in the dialogue pattern due to the different level of engagement experienced by the users. We train our HMM models on a corpus of natural dialogues with an Embodied Conversational Agent (ECA) in the domain of healthy-eating. The dialogues are coded in terms of Dialogue Acts associated to each system or user move. Results are quite encouraging: HMMs are a powerful formalism for describing the differences in the dialogue patterns, due to the different level of engagement of users and they can be successfully employed in real-time user’s engagement detection. Though, the HMM learning process shows a lack of robustness when using low-dimensional and skewed corpora. Therefore we plan a further validation of our approach with larger corpora in the near future.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Mazzotta I, De Rosis F, Carofiglio V (2007) PORTIA: A user-adapted persuasion system in the healthy eating domain. IEEE Intell Syst 22(6):42–51

    Article  Google Scholar 

  2. Mazzotta I, Novielli N, Silvestri E, de Rosis F (2007) ‘O Francesca ma che sei grulla?’. Emotions and irony in persuasion dialogues. In: Procs of the 10th conference of AI*IA—Special track on ‘AI for expressive media’. AI*IA 2007: artificial intelligence and human-oriented computing, Rome. LNCS, vol 4733. Springer, Berlin, pp 602–613

    Chapter  Google Scholar 

  3. Prochaska J, Di Clemente C, Norcross H (1992) In search of how people change: applications to addictive behavior. Am Psychol 47:1102–1114

    Article  Google Scholar 

  4. Petty RE, Cacioppo JT (1986) The elaboration likelihood model of persuasion. In: Berkowitz L (ed) Advances in experimental social psychology, vol 19. Academic Press, New York, pp 123–205

    Google Scholar 

  5. de Rosis F, Novielli N, Carofiglio V, De Carolis B (2006) User modeling and adaptation in health promotion dialogs with an animated character. J Biomed Inf 39(5):514–531

    Article  Google Scholar 

  6. de Rosis F, Batliner A, Novielli N, Steidl S (2007) ‘You are soo cool Valentina!’ Recognizing social attitude in speech-based dialogues with an ECA. In: Procs of ACII 2007, Lisbon

  7. Charniak E (1993) Statistical language learning. MIT Press, Cambridge

    Google Scholar 

  8. Picard RW (2003) Affective computing: challenges. Int J Human-Comput Stud 59(12):55–64

    Article  Google Scholar 

  9. Batliner A, Steidl S, Hacker C, Noth E, Niemann E (2005) Private emotions vs social interaction: towards new dimensions in research on emotions. In: Carberry S, De Rosis F (eds) Procs of the workshop on adapting the interaction style to affective factors

  10. Bosma W, Andre’ E (2004) Exploiting emotions to disambiguate dialogue acts. In: Proceedings of the international conference on intelligent user interfaces. Island of Madeira

  11. Litman D, Forbes K, Silliman S (2003) Towards emotion prediction in spoken tutoring dialogues. In: Proceedings of HLT/NAACL

  12. J Gill A, Oberlander J (2002) Taking care of the linguistic features of extraversion. In: Proceedings of the 24th annual conference of the cognitive science society

  13. Bickmore T, Cassell J (2005) Social dialogue with embodied conversational agents. In: van Kuppevelt J, Dybkjaer L, Bernsen N (eds) Advances in natural, multimodal dialogue systems. Kluwer Academic, New York

    Google Scholar 

  14. Peters C, Pelachaud C, Mancini M, Bevacqua E (2005) Engagement capabilities for ECAs, workshop “Creating bonds with ECAs”. Fourth international joint conf. on autonomous agents & multi-agent systems

  15. Bickmore T, Picard R (2005) Establishing and maintaining long-term human-computer relationships. ACM Trans Comput-Human Interact

  16. Sidner C, Lee C (2003) An architecture for engagement in collaborative conversations between a robot and a human. MERL Technical Report, TR2003-12

  17. Sidner C, Lee C (2003) An architecture for engagement in collaborative conversations between a robot and a human. MERL Technical Report, TR2003-12

  18. Campbell N (2008) Multimodal processing of discourse information; the effect of synchrony. In: Procs of the 2008 second international symposium on universal communication, pp 12–15

  19. Yu C, Aoki PM, Woodruff A (2004) Detecting user engagement in everyday conversations. In: Procs of international conference on spoken language processing, pp 1329–1332

  20. Pentland A (2005) Socially aware computation and communication. Computer 38(3):33–40

    Article  Google Scholar 

  21. Core MG, Moore JD, Zinn C (2003) The role of initiative in tutorial dialogue. In: Procs of 10th conference of the European chapter of the association for computational linguistics, Budapest, Hungary, April 2003

  22. Shah F (1997) Recognizing and responding to student plans in an intelligent tutoring system: CIRCSIM-Tutor. Ph.D. thesis, Illinois Institute of Technology

  23. Linell P, Gustavsson L, Juvonen P (1988) Interactional dominance in dyadic communication: a presentation of initiative-response analysis. Linguistics 26:415–442

    Article  Google Scholar 

  24. O’Brien HL, Toms EG, Kelloway EK, Kelley E Developing and evaluating a reliable measure of user engagement

  25. Collins R (2004) Interaction ritual chains. Princeton University Press, Princeton

    Google Scholar 

  26. Said NS (2004) An engaging multimedia design model. In: Procs of the 2004 conference in interaction design and children

  27. Smardon R (2004) Streetwise science: toward a theory of the code of the classroom. Mind Cult Act 11:201–223

    Article  Google Scholar 

  28. Olitsky S (2007) Promoting student engagement in science: Interaction rituals and the pursuit of a community of practice. J Res Sci Teach 44(1):33–56

    Article  Google Scholar 

  29. Woodruff A, Aoki PM (2004) Conversation analysis and the user experience. Digit Creat 15(4):232–238

    Article  Google Scholar 

  30. Walton D (2006) Examination dialogue: an argumentation framework for critically questioning an expert opinion. J Pragmat 38:745–777

    Article  Google Scholar 

  31. Martalo’ A, Novielli N, de Rosis F (2008) Attitude display in dialogue patterns. In: Proceedings of AISB’08, symposium on ‘affective language in human and machine’

  32. Whittaker S (2003) Theories and methods in mediated communication. In: Handbook of discourse processes, LEA, Mahwah, NJ

  33. Rabiner LR (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

    Article  Google Scholar 

  34. Levin E, Pieraccini R, Eckert W (1998) Using Markov decision process for learning dialogue strategies. In: Proceedings of the IEEE international conference on acoustic, speech and signal processing, vol 1, pp 201–204

  35. Stolcke A, Coccaro N, Bates R, Taylor P, Van Ess-Dykema C, Ries K, Shriberg E, Jurafsky D, Martin R, Meteer M (2000) Dialogue act modeling for automatic tagging and recognition of conversational speech. Comput Linguist 26:3

    Article  MathSciNet  Google Scholar 

  36. Twitchell DP, Adkins M, Nunamaker JF, Burgoon JK (2004) Using speech act theory to model conversations for automated classification and retrieval. In: Procs of the 9th international working conference on the language-action perspective on communication modelling

  37. Soller A (2004) Computational modeling and analysis of knowledge sharing in collaborative distance learning. UMUAI 14(4):351–381

    Google Scholar 

  38. Clarizio G, Mazzotta I, Novielli N, de Rosis F (2006) Social attitude towards a conversational character. In: Proceedings of the 15th IEEE international symposium on robot and human interactive communication. RO-MAN 2006. Hatfield, UK, pp 2–7

  39. Carletta J (1996) Assessing agreement on classification tasks. The Kappa statistics. Comput Linguist 22

  40. Levinson SE, Rabiner LR, Sondhi MM (1983) An introduction to the application of the theory of probabilistic functions of a Markov process to automatic speech recognition. Bell Syst Tech J 62(4):1035–1074

    MATH  MathSciNet  Google Scholar 

  41. Juang B-H, Rabiner LR (1985) A probabilistic distance measure for Hidden Markov models. ATT Tech J 64(2):391–408

    MathSciNet  Google Scholar 

  42. Carofiglio V, De Carolis B, Mazzotta I, Novielli N, Pizzutilo S (2009) Towards a socially intelligent ECA. In: Proceedings of CHItaly ’09, Roma, Italy, pp 99–106

  43. Novielli N, Strapparava C (2009) Towards unsupervised recognition of dialogue acts, proceedings of human language technologies: the 2009 annual conference of the North American chapter of the association for computational linguistics, companion volume: student research workshop and doctoral consortium, pp 84–89

  44. Carofiglio V, de Rosis F, Novielli N (2005) Dynamic user modeling in health promotion dialogs. In: Tao J, Tan T, Picard RW (eds) Affective computing and intelligent interaction, pp 723–730

  45. Carter N, Young D, Ferryman J (2006) A combined Bayesian Markovian approach for behaviour recognition. In: Procs of the 18th international conference on pattern recognition

Download references

Author information

Authors and Affiliations

  1. Department of Informatics, University of Bari, via Orabona, 4, 70125, Bari (BA), Italy

    Nicole Novielli

Authors
  1. Nicole Novielli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicole Novielli.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Novielli, N. HMM modeling of user engagement in advice-giving dialogues. J Multimodal User Interfaces 3, 131–140 (2010). https://doi.org/10.1007/s12193-009-0026-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-009-0026-4

Search

Navigation