Skip to main content
SpringerLink
Log in

Smile and Laughter Detection for Elderly People-Robot Interaction

  • Conference paper
  • First Online:
Social Robotics (ICSR 2015)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9388))

Included in the following conference series:

Abstract

Affect bursts play an important role in non-verbal social interaction. Laughter and smile are some of the most important social markers in human-robot social interaction. Not only do they contain affective information, they also may reveal the user’s communication strategy. In the context of human robot interaction, an automatic laughter and smile detection system may thus help the robot to adapt its behavior to a given user’s profile by adopting a more relevant communication scheme. While many interesting works on laughter and smile detection have been done, only few of them focused on elderly people. Elderly people data are relatively rare and often carry a significant challenge to a laughter and smile detection system due to face wrinkles and an often lower voice quality. In this paper, we address laughter and smile detection in the ROMEO2 corpus, a multimodal (audio and video) corpus of elderly people-robot interaction. We show that, while a single modality yields a given performance, a fair improvement can be reached by combining the two modalities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Deniz, O., Castrillon, M., Lorenzo, J., Anton, L., Bueno, G.: Smile detection for user interfaces. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Remagnino, P., Porikli, F., Peters, J., Klosowski, J., Arns, L., Chun, Y.K., Rhyne, T.-M., Monroe, L. (eds.) ISVC 2008, Part II. LNCS, vol. 5359, pp. 602–611. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  2. Abhinav, D., Akshay, A., Roland, G., Tom, G.: Emotion recognition using PHOG and LPQ features. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 878–883. IEEE (2011)

    Google Scholar 

  3. Ekman, P., Friesen, W.V., Hager, J.C.: Facs manual. A Human Face (2002)

    Google Scholar 

  4. Feng, X., Pietikäinen, M., Hadid, A.: Facial expression recognition based on local binary patterns. Pattern Recognition and Image Analysis 17(4), 592–598 (2007)

    Article  Google Scholar 

  5. Akinori, I., Xinyue, W., Motoyuki, S., Shozo, M.: Smile and laughter recognition using speech processing and face recognition from conversation video. In: International Conference on Cyberworlds, pp. 8. IEEE (2005)

    Google Scholar 

  6. Takeo, K., Cohn, J.F., Yingli, T.: Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, Proceedings, pp. 46–53. IEEE (2000)

    Google Scholar 

  7. Kennedy, L.S., Ellis, D.P.W.: Laughter detection in meetings. In NIST ICASSP 2004 Meeting Recognition Workshop, Montreal, pp. 118–121. National Institute of Standards and Technology (2004)

    Google Scholar 

  8. Kowalik, U., Aoki, T., Yasuda, H.: BROAFERENCE - a next generation multimedia terminal providing direct feedback on audience’s satisfaction level. In: Costabile, M.F., Paternó, F. (eds.) INTERACT 2005. LNCS, vol. 3585, pp. 974–977. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  9. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (cert). In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 298–305. IEEE (2011)

    Google Scholar 

  10. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J: Coding facial expressions with gabor wavelets. In: Third IEEE International Conference on Automatic Face and Gesture Recognition, Proceedings, pp. 200–205. IEEE (1998)

    Google Scholar 

  11. Moore, S., Bowden, R.: Local binary patterns for multi-view facial expression recognition. Computer Vision and Image Understanding 115(4), 541–558 (2011)

    Article  Google Scholar 

  12. Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7), 971–987 (2002)

    Article  MATH  Google Scholar 

  13. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: IEEE International Conference on Multimedia and Expo, ICME 2005, pp. 5. IEEE (2005)

    Google Scholar 

  14. Petridis, S., Pantic, M.: Audiovisual discrimination between speech and laughter: Why and when visual information might help. IEEE Transactions on Multimedia 13(2), 216–234 (2011)

    Article  Google Scholar 

  15. Salamin, H., Polychroniou, A., Vinciarelli, A.: Automatic detection of laughter and fillers in spontaneous mobile phone conversations. In: 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 4282–4287. IEEE (2013)

    Google Scholar 

  16. Schuller, B., Eyben, F., Rigoll, G.: Static and dynamic modelling for the recognition of non-verbal vocalisations in conversational speech. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds.) PIT 2008. LNCS (LNAI), vol. 5078, pp. 99–110. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  17. Sehili, M., Yang, F., Leynaert, V., Devillers, L.: A corpus of social interaction between nao and elderly people. In: 5th International Workshop on Emotion, Social Signals, Sentiment & Linked Open Data (ES3LOD2014). LREC (2014)

    Google Scholar 

  18. Shinohara, Y., Otsu, N.: Facial expression recognition using fisher weight maps. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition, Proceedings. pp. 499–504. IEEE (2004)

    Google Scholar 

  19. Truong, K., Van Leeuwen, D.: Evaluating automatic laughter segmentation in meetings using acoustic and acoustics-phonetic features. In: Proc Workshop on the Phonetics of Laughter at the 16th International Congress of Phonetic Sciences (ICPhS), pp. 49–53 (2007)

    Google Scholar 

  20. Khiet, P.: Truong and David A Van Leeuwen. Automatic discrimination between laughter and speech. Speech Communication 49(2), 144–158 (2007)

    Article  Google Scholar 

  21. Valstar, M., Pantic, M.: Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proc. 3rd Intern. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, pp. 65 (2010)

    Google Scholar 

  22. Valstar, M.F., Pantic, M., Ambadar, Z., Cohn, J.F.: Spontaneous vs. posed facial behavior: automatic analysis of brow actions. In: Proceedings of the 8th International Conference on Multimodal Interfaces, pp. 162–170. ACM (2006)

    Google Scholar 

  23. Viola, P., Jones, M.: Robust real-time object detection. International Journal of Computer Vision 4, 51–52 (2001)

    Google Scholar 

  24. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6), 915–928 (2007)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Human-Machine Communication, LIMSI-CNRS, Orsay, France

    Fan Yang, Mohamed A. Sehili, Claude Barras & Laurence Devillers

  2. Department of Computer Science, University Paris-Sud, Orsay, France

    Fan Yang & Claude Barras

  3. University Paris-Sorbonne, Paris, France

    Laurence Devillers

Authors
  1. Fan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed A. Sehili
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claude Barras
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Laurence Devillers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fan Yang .

Editor information

Editors and Affiliations

  1. Robotics and Computer Vision Lab, ENSTA-ParisTech, Palaiseau, France

    Adriana Tapus

  2. Inst für Informatik, Human-Ctr Multimedi, Universität Augsburg, Augsburg, Bayern, Germany

    Elisabeth André

  3. LIMSI-CNRS, Orsay, France

    Jean-Claude Martin

  4. Robotics and Computer Vision Lab, ENSTA-ParisTech, Palaiseau, France

    François Ferland

  5. LIMSI-CNRS, Orsay, France

    Mehdi Ammi

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Yang, F., Sehili, M.A., Barras, C., Devillers, L. (2015). Smile and Laughter Detection for Elderly People-Robot Interaction. In: Tapus, A., André, E., Martin, JC., Ferland, F., Ammi, M. (eds) Social Robotics. ICSR 2015. Lecture Notes in Computer Science(), vol 9388. Springer, Cham. https://doi.org/10.1007/978-3-319-25554-5_69

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-25554-5_69

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-25553-8

  • Online ISBN: 978-3-319-25554-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation