Skip to main content
Springer Nature Link
Log in

Human–Robot Gesture Analysis for Objective Assessment of Autism Spectrum Disorder

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

In this paper we study the use of human robot interaction as a mean to objectively evaluate imitation deficits in children with autism. Robot control and data analysis methods were combined to assess the quality of interaction between the robot and the subjects. Humanoid robot Zeno was used to execute upper body gestures which the subjects were asked to imitate. The resulting motion of the subject was acquired through a motion capture system and compared with the intended motion performed by Zeno using the dynamic time warping (DTW) algorithm. During this study, the clinical hypothesis was that the resulting DTW cost can serve as a quantitative measure for the similarity between the motions, and thus can be used to objectively assess the severity of imitation deficits exhibited by the child. To validate this hypothesis, we present two sets of experiments, one with a set of healthy adults and the other with a group of children, some with autism spectrum disorder. The experiment with adult subjects serves as a statistically significant test to demonstrate the viability of the DTW cost as a similarity measure for the gesture analysis, whereas the experiment with child subjects is a pilot study to differentiate imitation performance for children with autism.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Kanner L (1943) Autistic disturbances of affective contact. Nerv Child 2:217–250

    Google Scholar 

  2. Geschwind DH (2009) Advances in autism. Annu Rev Med 60(1):367–380

    Article  Google Scholar 

  3. Centers for Disease Control and Prevention (2014) Prevalence of autism spectrum disorder among children aged 8 years—autism and developmental disabilities monitoring network, 11 sites, United States, 2010. http://www.cdc.gov/mmwr/preview/mmwrhtml/ss6302a1.htm. Accessed Mar 2014

  4. Steffenburg S, Gillberg C, Hellgren L, Andersson L, Gillberg IC, Jakobsson G, Bohman M (1989) A twin study of autism in denmark, finland, iceland, norway and sweden. J Child Psychol Psychiatry 30(3):405–416

    Article  Google Scholar 

  5. Whyatt C, Craig C (2013) Sensory-motor problems in autism. Front Integr Neurosci 7:51

    Article  Google Scholar 

  6. Scassellati B, Admoni H, Mataric M (2012) Robots for use in autism research. Annu Rev Biomed Eng 14(1):275–294

    Article  Google Scholar 

  7. Shamsuddin S, Yussof H, Ismail LI, Mohamed S, Hanapiah FA, Zahari NI (2012) Initial response in hri—a case study on evaluation of child with autism spectrum disorders interacting with a humanoid robot nao. In: International symposium on robotics and intelligent sensors 2012

  8. Robins B, Dautenhahn K (2014) Tactile interactions with a humanoid robot: novel play scenario implementations with children with autism. Int J Soc Robot 6(3):397–415

    Article  Google Scholar 

  9. Robins B, Dautenhahn K, Dubowski J (2006) Does appearance matter in the interaction of children with autism with a humanoid robot? Interact Stud 7(3):509–542

    Article  Google Scholar 

  10. Tapus A, Peca A, Aly A, Pop C, Jisa L, Pintea S, Rusu AS, David DO (2012) Children with autism social engagement in interaction with nao, an imitative robot. Interact Stud 13(3):315–347

    Article  Google Scholar 

  11. Kozima H, Michalowski M, Nakagawa C (2009) Keepon: a playful robot for research, therapy, and entertainment. Int J Soc Robot 1(1):3–18

    Article  Google Scholar 

  12. Huskens B, Verschuur R, Gillesen J, Didden R, Barakova E (2013) Promoting question-asking in school-aged children with autism spectrum disorders: effectiveness of a robot intervention compared to a human-trainer intervention. Dev Neurorehabil 16(5):345–356

    Article  Google Scholar 

  13. Costa S, Lehmann H, Dautenhahn K, Robins B, Soares F (2014) Using a humanoid robot to elicit body awareness and appropriate physical interaction in children with autism. Int J Soc Robot 7(2):265–278

    Article  Google Scholar 

  14. Boccanfuso L, O’Kane JM (2011) CHARLIE: an adaptive robot design with hand and face tracking for use in autism therapy. Int J Soc Robot 3(4):337–347

    Article  Google Scholar 

  15. Wainer J, Dautenhahn K, Robins B, Amirabdollahian F (2013) A pilot study with a novel setup for collaborative play of the humanoid robot kaspar with children with autism. Int J Soc Robot 6(1):45–65

    Article  Google Scholar 

  16. Cabibihan JJ, Javed H, Ang M Jr, Aljunied SM (2013) Why robots? A survey on the roles and benefits of social robots in the therapy of children with autism. Int J Soc Robot 5(4):593–618

    Article  Google Scholar 

  17. Rinehart N, Bradshaw J, Brereton A, Tonge B (2001) Movement preparation in high-functioning autism and asperger disorder: a serial choice reaction time task involving motor reprogramming. J Autism Dev Disord 31(1):79–88

    Article  Google Scholar 

  18. Ricks DJ, Colton MB (2010) Trends and considerations in robot-assisted autism therapy. In: 2010 IEEE international conference on robotics and automation (ICRA), pp. 4354–4359, iD: 1

  19. Fasola J, Mataric M (2012) Using socially assistive human–robot interaction to motivate physical exercise for older adults. Proc IEEE 100(8):2512–2526

    Article  Google Scholar 

  20. Fujimoto I, Matsumoto T, Silva PRSD, Kobayashi M, Higashi M (2011) Mimicking and evaluating human motion to improve the imitation skill of children with autism through a robot. Int J Soc Robot 3(4):349–357

    Article  Google Scholar 

  21. Goncalves N, Costa S, Rodrigues J, Soares F (2014) Detection of stereotyped hand flapping movements in autistic children using the kinect sensor: a case study. In: 2014 IEEE international conference on autonomous robot systems and competitions (ICARSC), pp 212–216

  22. Ranatunga I, Rajruangrabin J, Popa DO, Makedon F (2011) Enhanced therapeutic interactivity using social robot zeno. In: Proceedings of the 4th international conference on PErvasive Technologies related to assistive environments, ser. PETRA ’11. ACM, New York, pp 57:1–57:6

  23. Ranatunga I, Torres NA, Patterson R, Bugnariu N, Stevenson M, Popa D (2012) Rodica: a human–robot interaction system for treatment of childhood autism spectrum disorders. In: Proceedings of the 5th international conference on PErvasive Technologies related to assistive environments, ser. PETRA ’12. ACM, New York

  24. Torres NA, Clark N, Ranatunga I, Popa D (2012) Implementation of interactive armplayback behaviors of social robot zeno for autism spectrum disorder therapy. In: Proceedings of the 5th international conference on PErvasive technologies related to assistive environments, ser. PETRA ’12. ACM, New York

  25. Baer D, Peterson R, Sherman J (1967) The development of imitation by reinforcing behavioral similarity to a model. J Exp Anal Behav 10(5):405

    Article  Google Scholar 

  26. Bellman R, Dreyfus S (1959) Functional approximations and dynamic programming. Math Tables Other Aids Comput 13(68):247–251. doi:10.1090/S0025-5718-1959-0107376-8

  27. Itakura F (1975) Minimum prediction residual principle applied to speech recognition. IEEE Trans Acoust Speech Sig Process 23(1):67–72

    Article  Google Scholar 

  28. Sakoe H, Chiba S (1978) Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans Acoust Speech Sig Process 26(1):43–49

    Article  MATH  Google Scholar 

  29. Christiansen R, Rushforth CK (1997) Detecting and locating key words in continuous speech using linear predictive coding. IEEE Trans Acoust Speech Sig Process 25(5):361–367

    Article  Google Scholar 

  30. Ferre F, Clote P (2006) Btw: a web server for boltzmann time warping of gene expression time series. Nucleic Acids Res 34:482–485

    Article  Google Scholar 

  31. Lijffijt J, Papapetrou P, Hollmen J, Athitsos V (2010) Benchmarking dynamic time warping for music retrieval. In: 3rd international conference on pervasive technologies related to assistive environments

  32. Keogh E, Ratanamahatana CA (2005) Exact indexing of dynamic time warping. Knowl Inf Syst 7:358–386. doi:10.1007/s10115-004-0154-9

    Article  Google Scholar 

  33. Jangyodsuk P, Conly C, Athitsos V (2014) Sign language recognition using dynamic time warping and hand shape distance based on histogram of oriented gradient features. In: 7th international conference on pervasive technologies related to assistive environments

  34. Wang H, Stefan A, Moradi S, Athitsos V, Neidle C, Kamangar F (2012) A system for large vocabulary sign search. Lect Notes Comput Sci 6553:342–353

    Article  Google Scholar 

  35. Ferguson S, Schubert E, Stevens CJ (2014) Dynamic dance warping: using dynamic time warping to compare dance movement performed under different conditions. In: Proceedings of the 2014 international workshop on movement and computing, p 94

  36. Brooks DA, Howard AM (2012) Quantifying upper-arm rehabilitation metrics for children through interaction with a humanoid robot. Appl Bion Biomech 9:157–172

    Article  Google Scholar 

  37. Zheng Z, Das S, Young EM, Swanson A, Warren Z, Sarkar N (2014) Autonomous robot-mediated imitation learning for children with autism. In: IEEE international conference on robotics and automation (ICRA) 2014, pp 2707–2712

  38. University of Leeds (2012) Kinesthesia—a kinect based rehabilitation and surgical analysis system. https://decibel.ni.com/content/docs/DOC-20973

  39. Ranatunga I, Beltran M, Torres NA, Bugnariu N, Patterson RM, Garver C, Popa DO (2013) Human–robot upper body gesture imitation analysis for autism spectrum disorders. In: Proceedings of the 5th international conference on social robotics

Download references

Acknowledgments

This work was supported by the TxMRC consortium grant: Human–Robot Interaction System for Early Diagnosis and Treatment of Childhood Autism Spectrum Disorders (RoDiCA) and by the US National Science Foundation Grant CPS 1035913. The authors wish to thank David Hanson, Richard Margolin, Matt Stevenson and Joshua Jach of RoboKind for their help with Zeno R-30. The authors thank Robert Longnecker, Janet Trammell, David Cummings and Carolyn Carr from UNTHSC for their help with the experimental data collection presented in this paper. Thanks also go to Carolyn Garver of Dallas Autism Treatment Center for the help provided with subject recruitment. The authors also thank Abhishek Thakurdesai from the NGS group for help with Zeno programming.

Author information

Authors and Affiliations

  1. University of Louisville, Louisville, KY, 40292, USA

    Indika B. Wijayasinghe & Dan O. Popa

  2. University of Texas at Arlington, Arlington, TX, 76010, USA

    Isura Ranatunga & Namrata Balakrishnan

  3. University of North Texas Health Science Center, 3500 Camp Bowie Blvd., Fort Worth, TX, 76107, USA

    Nicoleta Bugnariu

Authors
  1. Indika B. Wijayasinghe
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Isura Ranatunga
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Namrata Balakrishnan
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Nicoleta Bugnariu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Dan O. Popa
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Indika B. Wijayasinghe.

Appendix

Appendix

Table 5 Sample means of DTW costs
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wijayasinghe, I.B., Ranatunga, I., Balakrishnan, N. et al. Human–Robot Gesture Analysis for Objective Assessment of Autism Spectrum Disorder. Int J of Soc Robotics 8, 695–707 (2016). https://doi.org/10.1007/s12369-016-0379-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-016-0379-2

Keywords