Skip to main content
SpringerLink
Log in

Self-talk Discrimination in Human–Robot Interaction Situations for Supporting Social Awareness

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Being aware of the presence, activities and is fundamental for Human–Robot Interaction and assistive applications. In this paper, we describe (1) designing triadic situations for cognitive stimulation for elderly users; (2) characterizing social signals that describe social context: system directed speech (SDS) and self-talk (ST); and (3) estimating an interaction efficiency measure that reveals the quality of interaction. The proposed triadic situation is formed by a user, a computer providing cognitive exercises and a robot that provides encouragement and help using verbal and non-verbal signals. The methodology followed to design this situation is presented. Wizard-of-Oz experiments have been performed and analyzed through eye-contact behaviors and dialog acts (SDS and ST). We show that users employ two interaction styles characterized by different prosody features. Automatic recognition systems of these dialog acts is proposed using k-NN, decision tree and SVM classifiers trained with pitch, energy and rhythmic-based features. The best recognition system achieves an accuracy of 71 %, showing that interaction styles can be discriminated on the basis of prosodic features. An Interaction Efficiency (IE) metric is proposed to characterize interaction styles. This metric exploits on-view/off-view discrimination, semantic analysis and ST/SDS discrimination. Experiments on collected data prove the effectiveness of the IE measure in evaluating the individual’s quality of interaction of elderly patients during the cognitive stimulation task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. www-valoria.univ-ubs.fr.

References

  1. Feil-Seifer DJ, Mataric MJ (2005) Defining socially assistive robotics. In: International conference on rehabilitation robotics, Chicago, IL, pp 465–468

    Google Scholar 

  2. Fasola J, Mataric MJ (2010) Robot exercise instructor: a socially assistive robot system to monitor and encourage physical exercise for the elderly. In: 19th IEEE international symposium in robot and human interactive communication (Ro-Man 2010), Viareggio, Italy

    Google Scholar 

  3. Mataric MJ, Tapus A, Winstein CJ, Eriksson J (2009) Socially assistive robotics for stroke and mild TBI rehabilitation. In: Gaggioli A, Keshner EA, (Tamar) Weiss PL, Riva G (eds) Advanced technologies in rehabilitation. IOS Press, Amsterdam, pp 249–262.

    Google Scholar 

  4. Vinciarelli A, Pantic M, Bourlard H (2009) Social signal processing: survey of an emerging domain. Image Vis Comput J 27(12):1743–1759

    Article  Google Scholar 

  5. Saint-Georges C, Cassel RS, Cohen D, Chetouani M, Laznik M-C, Maestro S, Muratori F (2010) What studies of family home movies can teach us about autistic infants: a literature review. Res Autism Spectr Disord 4(3):355–366

    Article  Google Scholar 

  6. Cassell J, Bickmore J, Billinghurst M, Campbell L, Chang K, Vilhjàlmsson H, Yan H (1999) Embodiment in conversational interfaces: rea. In: CHI’99, Pittsburgh, pp 520–527

    Google Scholar 

  7. Wrede B, Kopp S, Rohlfing K, Lohse M, Muhl C (2010) Appropriate feedback in asymmetric interactions. J Pragmat 42(9):2369–2384

    Article  Google Scholar 

  8. Al Moubayed S, Baklouti M, Chetouani M, Dutoit T, Mahdhaoui A, Martin J-C, Ondas S, Pelachaud C, Urbain J, Yilmaz M (2009) Generating robot/agent backchannels during a storytelling experiment. In: ICRA’09, IEEE international conference on robotics and automation, Kobe, Japan

    Google Scholar 

  9. Chetouani M, Wu YH, Jost C, Le Pevedic B, Fassert C, Cristancho-Lacroix V, Lassiaille S, Granata Tapus A, Duhaut D, Rigaud AS (2010) Cognitive services for elderly people: the ROBADOM project. In: ECCE 2010 workshop: robots that care, European conference on cognitive ergonomics

    Google Scholar 

  10. Yanguas J, Buiza C, Etxeberria I, Urdaneta E, Galdona N, González MF (2008) Effectiveness of a non-pharmacological cognitive intervention on elderly factorial analysis of Donostia Longitudinal Study. Adv Gerontol 3:30–41

    Google Scholar 

  11. Young J, Sung JY, Voida A, Sharlin E, Igarashi T, Christensen H, Grinter R (2011) Evaluating human-robot interaction. Int J Soc Robot 3(1):53–67

    Article  Google Scholar 

  12. Sciutti A, Bisio A, Nori F, Metta G, Fadiga L, Pozzo T, Sandini G (2012) Measuring human-robot interaction through motor resonance. Int J Soc Robot 4(3):223–234

    Article  Google Scholar 

  13. Klein G, Woods DD, Bradshaw JM, Hoffman RR, Feltovich PJ (2004) Ten challenges for making automation a “team player” in joint human-agent activity. IEEE Intell Syst 10(6):91–95

    Article  Google Scholar 

  14. Delaherche E, Chetouani M, Mahdhaoui M, Saint-Georges C, Viaux S, Cohen D (2012) Interpersonal synchrony: a survey of evaluation methods across disciplines. IEEE Trans Affect Comput 3(3):34–365

    Google Scholar 

  15. Sidner CL, Kidd CD, Lee C, Lesh N (2004) Where to look: a study of human-robot engagement. In: Proceedings of the 9th international conference on intelligent user interfaces (IUI’04)

    Google Scholar 

  16. Poggi I (2007) Mind, hands, face and body. A goal and belief view of multimodal communication. Weidler, Berlin

    Google Scholar 

  17. Kulyukin V (2006) On natural language dialog with assistive robots. In: Proceedings of the 2006 ACM conference on human-robot interaction (HRI 2006), Salt Lake City, Utah, pp 164–171

    Google Scholar 

  18. Oppermann D, Schiel F, Steininger S, Beringer N (2001) Off-talk—a problem for human-machine-interaction? In: EUROSPEECH-2001, pp 2197–2200

    Google Scholar 

  19. Couture-Beil A, Vaughan R, Mori G (2010) Selecting and commanding individual robots in a vision-based multi-robot system. In: Seventh Canadian conference on computer and robot vision (CRV)

    Google Scholar 

  20. Castellano G, Pereira A, Leite I, Paiva A, McOwan PW (2009) Detecting user engagement with a robot companion using task and social interaction-based features. In: Proceedings of the 2009 international conference on multimodal interfaces (ICMI-MLMI’09), pp 119–126

    Chapter  Google Scholar 

  21. Ishii R, Shinohara Y, Nakano T, Nishida T (2011) Combining multiple types of eye-gaze information to predict user’s conversational engagement. In: 2nd workshop on eye gaze on intelligent human machine interaction

    Google Scholar 

  22. Nakano YI, Ishii R (2010) Estimating user’s engagement from eye-gaze behaviors in human-agent conversations. In: 2010 international conference on intelligent user interfaces (IUI2010)

    Google Scholar 

  23. Goffman E (1963) Behavior in public places: notes on the social organization of gatherings. The Free Press, New York

    Google Scholar 

  24. Argyle M, Cook M (1976) Gaze and mutual gaze. Cambridge University Press, Cambridge

    Google Scholar 

  25. Duncan S (1972) Some signals and rules for taking speaking turns in conversations. J Pers Soc Psychol 23(2):283–292

    Article  Google Scholar 

  26. Goodwin C (1986) Gestures as a resource for the organization of mutual attention. Semiotica 62(1/2):29–49

    Google Scholar 

  27. Kendon A (1967) Some functions of gaze direction in social interaction. Acta Psychol 26:22–63

    Article  Google Scholar 

  28. Klotz D, Wienke J, Peltason J, Wrede B, Wrede S, Khalidov V, Odobez JM (2011) Engagement-based multi-party dialog with a humanoid robot. In: Proceedings of SIGDIAL 2011: the 12th annual meeting of the special interest group on discourse and dialog, pp 341–343

    Google Scholar 

  29. Mutlu B, Shiwa T, Kanda T, Ishiguro H, Hagita N (2009) Footing in human-robot conversations: how robots might shape participants roles using gaze cues. In: Proc of ACM conf human robot interaction

    Google Scholar 

  30. Rich C, Ponsler B, Holroyd A, Sidner CL (2010) Recognizing engagement in human-robot interaction. In: Proc of ACM conf human robot interaction

    Google Scholar 

  31. Shi C, Shimada M, Kanda T, Ishiguro H, Hagita N (2011) Spatial formation model for initiating conversation. In: Proceedings of robotics: science and systems

    Google Scholar 

  32. Michalowski MP, Sabanovic S, Simmons R (2006) A spatial model of engagement for a social robot. In: IEEE international workshop on advanced motion control, pp 762–767

    Google Scholar 

  33. Mower E, Mataric MJ, Narayanan S (2011) A framework for automatic human emotion classification using emotional profiles. IEEE Trans Audio Speech Lang Process 19(5):1057–1070

    Article  Google Scholar 

  34. Zong C, Chetouani M (2009) Hilbert-Huang transform based physiological signals analysis for emotion recognition. In: IEEE symposium on signal processing and information technology (ISSPIT’09)

    Google Scholar 

  35. Peters C, Castellano G, de Freitas S (2009) An exploration of user engagement in HCI. In: Proceedings of AFFINE’09

    Google Scholar 

  36. Payr S, Wallis P, Cunningham S, Hawley M (2009) Research on social engagement with a rabbitic user interface. In: Tscheligi M, de Ruyter B, Soldatos J, Meschtscherjakov A, Buiza C, Streitz N, Mirlacher T (eds) Roots for the future of ambient intelligence. Adjunct proceedings, 3rd European conference on ambient intelligence (AmI09). ICT&S Center, Salzburg

    Google Scholar 

  37. Klamer T, Ben Allouch S (2010) Acceptance and use of a social robot by elderly users in a domestic environment. In: ICST PERVASIVE health 2010

    Google Scholar 

  38. Heerink M, Krose BJA, Wielinga BJ, Evers V (2006) The influence of a robot’s social abilities on acceptance by elderly users. In: Proceedings RO-MAN, Hertfordshire, September 2006, pp 521–526

    Google Scholar 

  39. Mataric MJ (2005) The role of embodiment in assistive interactive robotics for the elderly. In: AAAI fall symposium on caring machines: AI for the elderly, Arlington, VA

    Google Scholar 

  40. Tapus A, Tapus C, Mataric MJ (2009) The use of socially assistive robots in the design of intelligent cognitive therapies for people with dementia. In: Proceedings, international conference on rehabilitation robotics (ICORR-09), Kyoto, Japan

    Google Scholar 

  41. Xiao B, Lunsford R, Coulston R, Wesson M, Oviatt S (2003) Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences. In: Proceedings of the 5th international conference on multimodal interfaces

    Google Scholar 

  42. Lunsford R (2004) Private speech during multimodal human-computer interaction. In: International conference on multimodal interfaces (ICMI’04), p 346

    Google Scholar 

  43. Batliner A, Hacker C, Kaiser M, Mogele H, Noth E (2007) Taking into account the user’s focus of attention with the help of audio-visual information: towards less artificial human-machine communication. In: Auditory-visual speech processing (AVSP 2007)

    Google Scholar 

  44. Lunsford R, Oviatt S, Coulston R (2005) Audio-visual cues distinguishing self- from system-directed speech in younger and older adults. In: Proceedings of the 7th international conference on multimodal interfaces (ICMI’05), pp 167–174

    Google Scholar 

  45. Diaz R, Berk LE (eds) (1992) Private speech: from social interaction to self regulation. Erlbaum, Hillsdale

    Google Scholar 

  46. ten Bosch L, Boves L (2004) Survey of spontaneous speech phenomena in a multimodal dialog system and some implications for ASR. In: Proceedings, international conference on spoken language processing, October 2004, South Korea

    Google Scholar 

  47. Petersen RC, Doody R, Kurtz A, Mohs RC, Morris JC, Rabins PV, Ritchie K, Rossor M, Thal L, Winblad B (2001) Current concepts in mild cognitive impairment. Arch Neurol 58:1985–1992

    Article  Google Scholar 

  48. Wu YH, Fassert C, Rigaud AS (2012) Designing robots for the elderly: appearance issue and beyond. Arch Gerontol Geriatr 54(1):121–126

    Article  Google Scholar 

  49. Yngve VH (1970) On getting a word in edgewise. In: Proceedings of the sixth regional meeting of the Chicago linguistic society

    Google Scholar 

  50. Shibata T, Wada K, Saito T, Tanie K (2001) Mental commit robot and its application to therapy of children. In: IEEE/ASME international conference on AIM’01

    Google Scholar 

  51. Saint-Aime S, Le Pevedic B, Duhaut D (2008) EmotiRob: an emotional interaction model. In: IEEE RO-MAN 2008, 17th international symposium on robot and human interactive communication

    Google Scholar 

  52. Lee J, Nam T-J (2006) Augmenting emotional interaction through physical movement. In: UIST2006, the 19th annual ACM symposium on user interface software and technology

    Google Scholar 

  53. Steinberger J, Jarek K (2004) Using latent semantic analysis in text summary evaluation. In: Proceedings of ISIM’04 2004, pp 93–100

    Google Scholar 

  54. Schuller B, Batliner A, Seppi D, Steidl S, Vogt T, Wagner J, Devillers L, Vidrascu L, Amir N, Kessous L, Aharonson V (2007) The relevance of feature type for the automatic classification of emotional user states: low level descriptors and functionals. In: Proceedings of interspeech, pp 2253–2256

    Google Scholar 

  55. Mahdhoui A, Chetouani M (2011) Supervised and semi-supervised infant-directed speech classification for parent-infant interaction analysis. Speech Commun 9–10:1149–1161

    Article  Google Scholar 

  56. Breazeal C, Aryananda L (2002) Recognizing affective intent in robot directed speech. Auton Robots 12(1):83–104

    Article  MATH  Google Scholar 

  57. Hacker C, Batliner A, Noth E (2006) Are you looking at me, are you talking with me: multimodal classification of the focus of attention. In: Sojka P, Kopêcek I, Pala K (eds) TSD 2006. LNAI, vol 4188. Springer, Berlin, pp 581–588

    Google Scholar 

  58. Truong K, van Leeuwen D (2007) Automatic discrimination between laughter and speech. Speech Commun 49:144–158

    Article  Google Scholar 

  59. Boersma P, Weenink D (2005) Praat, doing phonetics by computer. Tech. rep, Institute of Phonetic Sciences, University of Amsterdam, Pays-Bas. URL www.praat.org

  60. Shami M, Verhelst W (2007) An evaluation of the robustness of existing supervised machine learning approaches to the classification of emotions. Speech Commun 49(3):201–212

    Article  Google Scholar 

  61. Tilsen S, Johnson K (2008) Low-frequency Fourier analysis of speech rhythm. J Acoust Soc Am 124(2):EL34–EL39

    Google Scholar 

  62. Ringeval F, Chetouani M, Schuller B (2012) Novel metrics of speech rhythm for the assessment of emotion. In: Interspeech 2012

    Google Scholar 

  63. Zellner-Keller B, Keller E (1998) The chaotic nature of speech rhythm: hints for fluency in the language acquisition process. Integrating speech technology in language learning. Swets Zeitlinger, Lisse

    Google Scholar 

  64. Duda R, Hart P, Stork D (2000) Pattern classification, 2nd edn. Wiley, New York

    Google Scholar 

  65. Vapnik V (1995) The nature of statistical learning theory. Springer, Berlin

    Book  MATH  Google Scholar 

  66. Sanghvi J, Castellano G, Leite I, Pereira A, McOwan PW, Paiva A (2011) Automatic analysis of postures and body motion to detect engagement with a game companion. In: Proc of ACM conf human robot interaction

    Google Scholar 

  67. Olsen DR, Goodrich M (2003) Metrics for evaluating human-robot interaction. In: PERMIS 2003

    Google Scholar 

  68. Delaherche E, Chetouani M (2010) Multimodal coordination: exploring relevant features and measures. In: Second international workshop on social signal processing. ACM Multimedia, New York

    Google Scholar 

  69. Dahlbaeck N, Joensson A, Ahrenberg L (1993) Wizard of oz studies ? Why and how. In: Proceedings of the 1993 international workshop on intelligent user interfaces (IUI193). ACM Press, New York, pp 193–200

    Google Scholar 

  70. Xiao B, Lunsford R, Coulston R, Wesson M, Oviatt S (2003) Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences. In: Proceedings of the 5th international conference on multimodal interfaces. Vancouver, British Columbia, Canada

    Google Scholar 

  71. Light J (1997) Communication is the essence of human kife: reflections on communicative competence. Augment Altern Commun 13(2):61–70

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the Broca hospital for their work: Ya-Huei Wu, Christine Fassert, Victoria Cristancho-Lacroix and Anne-Sophie Rigaud.

Author information

Authors and Affiliations

  1. ISIR UMR 7222, Université Pierre et Marie Curie, Paris, France

    Jade Le Maitre & Mohamed Chetouani

Authors
  1. Jade Le Maitre
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Chetouani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohamed Chetouani.

Additional information

This work has been supported by French National Research Agency (ANR) through TecSan program (Robadom project ANR-09-TECS-012).

Rights and permissions

Reprints and permissions

About this article

Cite this article

Le Maitre, J., Chetouani, M. Self-talk Discrimination in Human–Robot Interaction Situations for Supporting Social Awareness. Int J of Soc Robotics 5, 277–289 (2013). https://doi.org/10.1007/s12369-013-0179-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-013-0179-x

Keywords

Search

Navigation