Skip to main content
SpringerLink
Log in

Beat gesture recognition and finger motion control of a piano playing robot for affective interaction of the elderly

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

This paper introduces a piano playing robot in views of smart house and assistive robot technology to care the affective states of the elderly. We address the current issues in this research area and propose a piano playing robot as a solution. For affective interaction based on music, we first present a beat gesture recognition method to synchronize the tempo of a robot playing a piano with the desired tempo of the user. To estimate the period of an unstructured beat gesture expressed by any part of a body or an object, we apply an optical flow method, and use the trajectories of the center of gravity and normalized central moments of moving objects in images. In addition, we also apply a motion control method by which robotic fingers are trained to follow a set of trajectories. Since the ability to track the trajectories influences the sound a piano generates, we adopt an iterative learning control method to reduce the tracking error.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bien ZZ, Park KH, Jung JW and Do JH (2005). Intention reading is essential in human-friendly interfaces for the elderly and the handicapped. IEEE Trans Ind Electron 52(6): 1500–1505

    Article  Google Scholar 

  2. Bondi P, Casalino G and Gambardella L (1988). On the iterative learning control theory for robotic manipulators. IEEE Trans Rob Autom 4(1): 14–22

    Article  Google Scholar 

  3. Cohen C, Conway L (1996) Dynamical system representation, generation, and recognition of basic oscillatory motion gestures. In: Proceedings of the 2nd international conference on automatic face and gesture recognition, pp 60–65

  4. Cutler R and Davis L (2000). Robust real-time periodic motion detection, analysis and applications. IEEE Trans Pattern Anal Mach Intell 22(8): 781–796

    Article  Google Scholar 

  5. Davis J, Bobick A, Richards W (2000) Categorical representation and recognition of oscillatory motion patterns. In: Proceedings of the IEEE conference on computer vision and pattern recognition, vol 1, pp 628–635

  6. Fawcett J, Scheftner W, Fogg L, Clark D, Young M, Hedeker D and Gibbons R (1990). Time-related predictors of suicide in major affective disorder. Am J Psychiatry 147: 1189–1194

    Google Scholar 

  7. Filova V, Solina F and Lenarcic J (1998). Automatic reconstruction of 3d human arm motion from a monocular image sequence. Mach Vis Appl 10: 223–231

    Article  Google Scholar 

  8. Kato I, Ohteru S, Shirai K, Narita S, Sugano S, Matsushima T, Kobayashi T and Fujisawa E (1987). The robot musician ‘wabot-2’. Robotics 3(2): 143–155

    Article  Google Scholar 

  9. Lee HE, Kim YM, Park KH and Bien ZZ (2005). Development of a steward robot for intelligent sweet home. Int J Hum-friendly Welf Rob Syst 6(4): 57–64

    Google Scholar 

  10. Liu F, Picard R (1998) Finding periodicity in space and time. In: Proceedings of the 6th international conference on computer vision, pp 376–383

  11. Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: Proceedings of the DARPA image understanding workshop, pp 121–130

  12. Luukinen H, Laippala P and Huikuri H (2003). Depressive symptoms and the risk of sudden cardiac death among the elderly. Eur Heart J 24(22): 2021–2026

    Article  Google Scholar 

  13. Moore K (1993). Iterative learning control for deterministic systems. Springer, Heidelberg

    MATH  Google Scholar 

  14. Oh J, Choi Y, Park J and Zheng Y (2004). Complete coverage navigation of cleaning robots using triangular-cell-based map. IEEE Trans Ind Electron 51(3): 718–726

    Article  Google Scholar 

  15. Palacin J, Salse J, Valganon I and Clua X (2004). Building a mobile robot for a floor-cleaning operation in domestic environments. IEEE Trans Instrum Meas 53(5): 1418–1424

    Article  Google Scholar 

  16. Paul R (1982). Robot manipulators: mathematics, programming and control. MIT Press, Cambridge, p 198

    Google Scholar 

  17. Polana R, Nelson R (1993) Detecting activities. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2–7

  18. Seitz S, Dyer C (1994) Affine invariant detection of periodic motion. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, pp 970–975

  19. Sugano S, Kato I (1987) Wabot-2: autonomous robot with dexterous finger-arm. In: Proceedings of the IEEE international conference on robotics and automation, pp 90–97

  20. Tsai P, Shah M, Keiter K and Kasparis T (1994). Cyclic motion detection for motion based recognition. Pattern Recognit 27(12): 1591–1603

    Article  Google Scholar 

  21. Watanabe K, Kosaka H (1995) Evaluation of keyboard switches based on kansei (human sensitivity) information. In: Proceedings of the 4th IEEE international workshop on robot and human communication, Tokyo, pp 65–70

  22. Yoshikawa T (1990). Foundations of robotics, analysis and control chap 3. MIT Press, Cambridge

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Science, Korea Advanced Institute of Science and Technology, 373-1Guseong-dong, Yuseong-gu, Daejeon, 305-701, Republic of Korea

    Kwang-Hyun Park, Sung-Hoon Jeong & Z. Zenn. Bien

  2. Institute of Automation, University of Bremen, Otto-Hahn-Allee, NW1, 28359, Bremen, Germany

    Christopher Pelczar

Authors
  1. Kwang-Hyun Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sung-Hoon Jeong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christopher Pelczar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Z. Zenn. Bien
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kwang-Hyun Park.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Park, KH., Jeong, SH., Pelczar, C. et al. Beat gesture recognition and finger motion control of a piano playing robot for affective interaction of the elderly. Intel Serv Robotics 1, 185–193 (2008). https://doi.org/10.1007/s11370-008-0018-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-008-0018-3

Keywords

Search

Navigation