Skip to main content
SpringerLink
Log in

A New Robotic Platform for Sign Language Tutoring

Humanoid Robots as Assistive Game Companions for Teaching Sign Language

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

This paper presents a socially interactive humanoid robot-assisted system for sign language (SL) tutoring for children with communication impairments by means of imitation-based interaction games. In this study, a five-fingered robot platform Robovie R3 is used to express a set of chosen words in Turkish sign language (TSL) using hand and body movements combined with facial expressions. The robot is able to recognize signs through a RGB-D camera and give vocal, visual and motional (as signs) feedback. The proposed game consists of an introductory phase where participants are introduced to the robot and the signs; an imitation-based learning phase where participants are motivated to imitate the signs demonstrated by the robot, and a test phase where the signs taught in the previous phases are tested within a guessing game. The current paper presents results from the studies with three different test groups. The presented humanoid robot is used as an assistive social companion in the game context using sign language and visual clues to interact with the children. The robot is evaluated according to the participant’s sign recognition ability within different setups. The results indicate that the robot has a significant effect on the sign learning performance of participants.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Kushalnagar P, Mathur G, Moreland CJ, Napoli DJ, Osterling W, Padden C, Rathmann C (2010) Infants and children with hearing loss need early language access. J Clin Ethics 21(2):143–154

    Google Scholar 

  2. Kose H, Yorganci R. (2011) Tale of a robot: humanoid robot assisted sign language tutoring. In: Proceedings of IEEE-RAS international conference on humanoid robots, pp 105–111

  3. Morasso P, Casadio M, Giannoni P, Masia L, Sanguineti V, Squeri V, Vergaro E (2009) Desirable features of a “Humanoid” robot-therapist. In: Proceedings of annual international conference of the IEEE on engineering in medicine and biology society (EMBC), pp 2418–2421

  4. Riener R (2005) Control of robots for rehabilitation. In: Proceedings of international conference on computer as a tool (EUROCON), vol 1, pp 33–36

  5. Ismail L, Shamsuddin S, Yussof H, Hashim H, Bahari S, Jaafar A, Zahari I (2011) Face detection technique of humanoid robot nao for application in robotic assistive therapy. In: Proceedings of IEEE international conference on control system, computing and engineering (ICCSCE), pp 517–521

  6. Vygotsky LS (1978) Mind and society: the development of higher mental processes. Harvard University Press, Cambridge

    Google Scholar 

  7. Spitzer SL (2008) Play in children with autism: structure and experience. In: Parham LD, Fazio LS (eds) Play in occupational therapy for children. Mosby, St. Louis, pp 351–374

    Chapter  Google Scholar 

  8. Iacono I, Lehmann H, Marti P, Robins B, Dautenhahn K (2001) Robots as social mediators for children with autism—a preliminary analysis comparing two different robotic platforms. In: Proceedings of IEEE international conference on development and learning (ICDL), vol 2, pp 1–6

  9. Haberdar H, Albayrak S (2005) Real-time isolated Turkish sign language recognition from video using hidden Markov models with global features. In: Yolum P, Gungor T, Gurgen F, Ozturan C (eds) Lecture notes in computer science, computer and information sciences (ISCIS), vol 3733. Springer, Berlin, pp 677–687

    Google Scholar 

  10. Kadous W (1995) Grasp: recognition of Australian sign language using instrumented gloves. Dissertation, University of New South Wales

  11. Starner T, Weaver J, Pentland A (1998) Real-time American sign language recognition using desk and wearable computer based video. IEEE Trans Pattern Anal Mach Intell 20(12):1371–1375

  12. Braffort A, Bolot L, Segouat J (2011) Virtual signer co-articulation in octopus, a sign language generation platform. In: Proceedings of the 9th international gesture workshop, gesture in embodied communication and human-computer interaction

  13. Aran O, Akarun L (2010) A multi-class classification strategy for fisher scores: application to signer independent sign language recognition. Pattern Recogn 43(5):1776–1788

    Article  MATH  Google Scholar 

  14. Keskin C, Akarun L (2009) Sign tracking and recognition system using input–output HMMs. Pattern Recognit Lett 30(12):1086–1095

    Article  Google Scholar 

  15. Caplier A, Stillittano S, Aran O, Akarun L, Bailly G, Beautemps D, Aboutabit N, Burger T (2007) Image and Video for Hearing Impaired people. In: Proceedings of EURASIP journal on image and video processing, vol 1, p 045641

  16. Luis-Perez FE, Trujillo-Romero F, Martnez-Velazco W (2011) Control of a service robot using the mexican sign language. In: Batyrshin I, Sidorov G (eds) Lecture notes in computer science, advances in soft computing, vol 7095. Springer, Berlin, pp 419–430

    Google Scholar 

  17. Anastasiou D (2012) Gestures in assisted living environments. In: Efthimiou E, Kouroupetroglou G, Fotinea SE (eds) Lecture notes in computer science, gesture and sign language in human-computer interaction and embodied communication, vol 7206. Springer, Berlin, pp 1–12

    Chapter  Google Scholar 

  18. Jaffe D (1994) Evolution of mechanical finger spelling hands for people who are deaf-blind. Journal of rehabilitation research and development 31:236–244

    Google Scholar 

  19. Hersh MA, Johnson MA (2003) Anatomy and physiology of hearing, hearing impairment and treatment. In: Hersh MA, Johnson MA (eds) Assistive technology for the hearing-impaired deaf and deafblind. Springer, London, pp 1–39

    Chapter  Google Scholar 

  20. Gibet S (2011) analysis and synthesis of sign language gestures: from meaning to movement production. In: Proceedings of the 9th international gesture workshop, gesture in embodied communication and human-computer interaction

  21. Kose H, Yorganci R, Algan EH, Syrdal DS (2012) Evaluation of the robot assisted sign language tutoring using video-based studies. Int J Soc Robot 4(3):273–283

    Article  Google Scholar 

  22. Akalin N, Uluer P, Kose H (2013) Ispy-usign humanoid assisted interactive sign language tutoring games. In: Proceedings of IEEE RO-MAN, pp 290–291

  23. Huenerfauth MA (2004) Multi-path architecture for machine translation of english text into American sign language animation. In: Proceedings of the student research workshop at HLT-NAACL association for computational linguistics, pp 25–30

  24. Kipp M, Heloir A, Nguyen Q (2001) Sign language avatars: animation and comprehensibility. In: Vilhjalmsson H, Kopp S, Marsella S, Thorisson KR (eds) Lecture notes in computer science, intelligent virtual agents, vol 6895. Springer, Berlin, pp 113–126

    Chapter  Google Scholar 

  25. Lave J, Wenger E (1991) Situated learning: legitimate peripheral participation. Cambridge University Press, New York

    Book  Google Scholar 

  26. Bruner JS (1990) Acts of meaning. Harvard University Press, London

    Google Scholar 

  27. Powell S (2000) Helping children with autism to learn. Taylor & Francis, Boca Raton

    Google Scholar 

  28. Hakkarainen P (1999) Play and motivation. In: Engestrom Y, Miettinen R, Punamaki RL (eds) Perspectives on activity theory. Cambridge University Press, New York, pp 231–250

    Chapter  Google Scholar 

  29. Ho-Sub Y, Su-Young C (2006) Visual processing of rock, scissors, paper game for human robot interaction. In: Proceedings of international joint conference SICE-ICASE, pp 326–329

  30. Chao C, Jinhan L, Begum M, Thomaz AL (2011) Simon plays Simon says: the timing of turn-taking in an imitation game. In: Proceedings of IEEE RO-MAN, pp 235–240

  31. Changchun L, Conn K, Sarkar N, Stone W (2008) Online affect detection and robot behavior adaptation for intervention of children with autism. IEEE transactions on robotics 24(4):883–896

    Article  Google Scholar 

  32. Kanda T, Hirano T, Eaton D, Ishiguro H (2004) Interactive robots as social partners and peer tutors for children: a field trial. Hum-comput Interact 19(1):61–84

    Article  Google Scholar 

  33. Isaacs EA, Clark HH (1987) References in conversation between experts and novices. J Exp Psychol 116(1):26–37

    Article  Google Scholar 

  34. Nalin M, Baroni I, Kruijff-Korbayova I, Canamero L, Lewis M, Beck A, Cuayahuitl H, Sanna A (2012) Children’s adaptation in multi-session interaction with a humanoid robot. In: Proceedings of IEEE RO-MAN, pp 351–357

  35. Michaud F, Clavet A, Lachiver G, Lucas M (2000) Designing toy robots to help autistic children—an open design project for electrical and computer engineering education. In: Proceedings of American Society for Engineering Education

  36. Robins B, Dickerson P, Stribling P, Dautenhahn K (2004) Robot-mediated joint attention in children with autism: a case study in robot–human interaction. Interact Stud 5(2):161–198

    Article  Google Scholar 

  37. Robins B, Dautenhahn K, Bo ekhorst T, Billard A (2005) Robotic assistants in therapy and education of children with autism: can a small humanoid robot help encourage social interaction skills? Univers Access Inf Soc 4(2):105–120

    Article  Google Scholar 

  38. Salber D, Coutaz J (1993) Applying the wizard of Oz technique to the study of multimodal systems. In: Bass LJ, Gornostaev J, Unger C (eds) Lecture notes in computer science, human-computer interaction, vol 753. Springer, Berlin, pp 219–230

    Google Scholar 

  39. Kose H, Akalin N, Uluer P (2014) Socially interactive robotic platforms as sign language tutors. Int J Hum Robot 11(1):1450003

    Article  Google Scholar 

  40. Akalin N, Uluer P, Kose H, Ince G (2013) Humanoid robots communication with participants using sign language: an interaction based sign language game. In: Proceedings of IEEE workshop on advanced robotics and its social impacts (ARSO), pp 181–186

  41. Kose H, Yorganci R, Itauma II (2011) Humanoid robot assisted interactive sign language tutoring game. In: Proceedings of IEEE international conference on robotics and biomimetics (ROBIO), pp 2247–2248

  42. Kose-Bagci H, Yorganci R, Algan EH (2011) Evaluation of the robot sign language tutoring assistant using video-based studies. In: Lilienthal AJ (ed) Proceedings of the European Conference on Mobile Robots (ECMR), pp 109–114

  43. Ertugrul BS, Kivrak H, Daglarli E, Kulaglic A, Tekelioglu A, Kavak S, Ozkul A, Yorganci R, Kose H (2012) iSign: interaction games for humanoid assisted sign language tutoring. In: Proceedings of international workshop on human–agent interaction (iHAI)

  44. Ertugrul BS, Gurpinar C, Kivrak H, Kose H (2013) Gesture recognition for humanoid assisted interactive sign language tutoring. In: Proceedings of signal processing and communications applications conference (SIU), pp 1–4

  45. Kivrak H, Ertugrul BS, Yorganci R, Daglarli E, Kulaglic A, Kose A (2012) Humanoid assisted sign language tutoring. In: 5th workshop on human-friendly robotics (HFR)

  46. Turkish Sign Language Dictionary. URL http://www.tdk.gov.tr/index.php?option=com_content&id=264

Download references

Acknowledgments

We would like to thank the managers, teachers and students of Ferahevler Primary School, Dosteller Secondary School for Hearing-Impaired Children and Turkish Hearing-Impaired Association for their voluntary participation, their criticisms and contributions to this study. And also we would like to thank TSL tutors Sumru Özsoy and Feride Korkmaz for their guidance. Research supported by the Scientific and Technological Research Council of Turkey under the contract TUBITAK KARIYER 111E283.

Author information

Authors and Affiliations

  1. Faculty of Engineering and Technology, Galatasaray University, Istanbul, Turkey

    Pınar Uluer

  2. Faculty of Computer and Informatics, Istanbul Technical University, Istanbul, Turkey

    Pınar Uluer, Neziha Akalın & Hatice Köse

Authors
  1. Pınar Uluer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Neziha Akalın
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hatice Köse
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pınar Uluer.

Appendix

Appendix

The sample set of TSL words, their demonstration and the flashcards representing the signs are displayed in Fig. 7.

Fig. 7
figure 7

The demonstration for the 10 TSL signs and the flashcards representing them: a “mother”, b “to throw”, c “spring”, d “baby”, e “I/me”, f “big”, g “mountain”, h “to come”, i “table”, j “black”

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Uluer, P., Akalın, N. & Köse, H. A New Robotic Platform for Sign Language Tutoring. Int J of Soc Robotics 7, 571–585 (2015). https://doi.org/10.1007/s12369-015-0307-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-015-0307-x

Keywords

Search

Navigation