Skip to main content
SpringerLink
Log in

Music to Motion: Using Music Information to Create Expressive Robot Motion

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

In a robot theater, developing high-quality motions for a humanoid robot requires significant time and effort. People with artistic and animation skills are required to create expressive gestures and movements on the robots. We observe that dancers can translate music into motions that are perceived as expressive by the audience. In this study, we developed a program to exploit melodic and dynamics information from music to create expressive robot motion. The program was used on two applications: producing motions strictly from the music information and controlling the execution a pre-programmed sequence of actions. Both applications are intended for a robot theater. The former was applied on an arm robot, while the latter on a humanoid KHR-1 robot. Two surveys were done to analyze the impact of our method. The results suggest that the robot motions produced by our program can be perceived as being more expressive and more dynamic than the motions created by a person or without music information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. The MIDI beat resolution used by MusE is 384 ticks per beat.

  2. The KHR-1 robot has since been discontinued by its manufacturer. Information about the robot can still be found here: [28].

  3. URL: https://youtu.be/9QHp9B-0QK0.

  4. The whole music is played for the Mozart Sonata No. 16 because the channel used to generate the motion data does not sound good when played by itself—the main melody is constructed by the combination of multiple channels.

  5. URL: https://youtu.be/9QHp9B-0QK0.

  6. URL: https://youtu.be/ASWy6lHl4zs.

References

  1. Amaya K, Bruderlin A, Calvert T (1996) Emotion from motion. In: Graphics interface, vol. 96. Toronto, Canada, pp 222–229

  2. Bhutada A (2011) Universal event and motion editor for robots’ theatre. Master’s thesis, Portland State University, 1900 SW Fourth Ave., Suite 160, Portland, OR 97201

  3. Breazeal C (1999) A context-dependent attention system for a social robot. In: 1999 International joint conference on artificial intelligence

  4. Bruderlin A, Williams L (1995) Motion signal processing. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques. ACM, pp 97–104

  5. Busso C, Deng Z, Neumann U, Narayanan S (2005) Natural head motion synthesis driven by acoustic prosodic features. Comput Anim Virtual Worlds 16(3–4):283–290

    Article  Google Scholar 

  6. Camurri A, Volpe G, Poli GD, Leman M (2005) Communicating expressiveness and affect in multimodal interactive systems. IEEE Multimed 12(1):43–53

    Article  Google Scholar 

  7. Fernando R (2004) GPU gems: programming techniques. Tips and tricks for real-time graphics. Pearson Higher Education, Upper Saddle River

    Google Scholar 

  8. Fisher RA (1925) Statistical methods for research workers. Genesis Publishing Pvt Ltd, New Delhi

    MATH  Google Scholar 

  9. Hein E (2010) Scales and emotions. http://www.ethanhein.com/wp/2010/scales-and-emotions. Accessed 29 Jan 2017

  10. Hollander M, Wolfe DA, Chicken E (2013) Nonparametric statistical methods. Wiley, Hoboken

    MATH  Google Scholar 

  11. Johnson R (2014) Choosing between parametric and non-parametric tests. J Undergrad Res Minn State Univ Mankato 9(1):6

    Google Scholar 

  12. Lee D, Park S, Hahn M, Lee N (2014) Robot actors and authoring tools for live performance system. In: 2014 International conference on information science and applications (ICISA). IEEE, pp 1–3

  13. Sergey L (2009) Body language animation synthesis from prosody. Undergraduate honors thesis, Stanford University

  14. Mannell R, C F, Harrington J (2014) An introduction to phonetics and phonology. http://clas.mq.edu.au/speech/phonetics/index.html. Accessed 29 Jan 2017

  15. McNeill D (1992) Hand and mind: what gestures reveal about thought. University of Chicago Press, Chicago

    Google Scholar 

  16. Medonis.com (2008) Asc16 - advanced servo controller. http://www.medonis.com/asc16.html. Accessed 29 Jan 2017

  17. Meek GE, Ozgur C, Dunning K (2007) Comparison of the t vs. wilcoxon signed-rank test for likert scale data and small samples. J Mod Appl Stat Methods 6(1):10

    Article  Google Scholar 

  18. Merriam-Webster.com (2017) Dynamic. https://www.merriam-webster.com/dictionary/dynamic. Accessed 19 June 2017

  19. Merriam-Webster.com (2017) Expressive. https://www.merriam-webster.com/dictionary/expressive. Accessed 19 June 2017

  20. Michalowski MP, Sabanovic S, Kozima H (2007) A dancing robot for rhythmic social interaction. In: 2007 2nd ACM/IEEE international conference on human–robot interaction (HRI). IEEE, pp 89–96

  21. MIDI.org (2016) The complete midi 1.0 detailed specification. https://www.midi.org/specifications/item/the-midi-1-0-specification. Accessed 3 Feb 2017

  22. MIDI.org (2016) Summary of midi messages. https://www.midi.org/specifications/item/table-1-summary-of-midi-message. Accessed 3 Feb 2017

  23. Muse-sequencer.org (2016) Main page. http://www.muse-sequencer.org/. Accessed 19 June 2017

  24. Perkowski M, Bhutada A, Lukac M, Sunardi M (2013) On synthesis and verification from event diagrams in a robot theatre application. In: 2013 IEEE 43rd international symposium on multiple-valued logic (ISMVL). IEEE, pp 77–83

  25. Perkowski M, Sasao T, Kim JH, Lukac M, Allen J, Gebauer S (2005) Hahoe kaist robot theatre: learning rules of interactive robot behavior as a multiple-valued logic synthesis problem. In: 2005. Proceedings. 35th International symposium on multiple-valued logic. IEEE, pp 236–248

  26. Perlin K (1985) An image synthesizer. ACM Siggraph Comput Graphics 19(3):287–296

    Article  Google Scholar 

  27. Pilhofer M, Day H (2015) Music theory for dummies. Wiley, Hoboken

    Google Scholar 

  28. Robosavvy.com (2015) Khr-1 humanoid robot kit + roboard embedded 1ghz linux board. https://robosavvy.com/store/khr-1-humanoid-robot-kit-roboard-embedded-1ghz-linux-board.html. Accessed 29 Jan 2017

  29. Sargin ME, Erzin E, Yemez Y, Tekalp AM, Erdem AT, Erdem C, Ozkan M (2007) Prosody-driven head-gesture animation. In: 2007 IEEE international conference on acoustics, speech and signal processing-ICASSP’07, vol 2. IEEE, pp II–677

  30. Scheeff M, Pinto J, Rahardja K, Snibbe S, Tow R (2002) Experiences with sparky, a social robot. In: Socially intelligent agents. Springer US, pp 173–180

  31. Scheffé H (1952) An analysis of variance for paired comparisons. J Am Stat Assoc 47(259):381–400

    MathSciNet  MATH  Google Scholar 

  32. Stanislavski C (2013) Building a character. A & C Black, London

    Google Scholar 

  33. Sunardi MI (2010) Expressive motion synthesis for robot actors in robot theatre. Master’s thesis, Portland State University, 1900 SW Fourth Ave., Suite 160, Portland, OR 97201

  34. Thomas F, Johnston O, Thomas F (1995) The illusion of life: disney animation. Hyperion New York, New York

    Google Scholar 

  35. Unuma M, Anjyo K, Takeuchi R (1995) Fourier principles for emotion-based human figure animation. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques. ACM, pp 91–96

  36. Von Laban R, Lange R (1975) Laban’s principles of dance and movement notation. Princeton Book Co Pub, Princeton

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering Department, Portland State University, 1900 SW Fourth Ave., Suite 160, Portland, OR, 97201, USA

    Mathias Sunardi & Marek Perkowski

Authors
  1. Mathias Sunardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marek Perkowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mathias Sunardi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human Participants

Survey participants were informed that their participation in the survey is entirely voluntary and they can choose to not participate. The participants were asked only to provide gender and age information, and informed that their providing this information is also voluntary.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sunardi, M., Perkowski, M. Music to Motion: Using Music Information to Create Expressive Robot Motion. Int J of Soc Robotics 10, 43–63 (2018). https://doi.org/10.1007/s12369-017-0432-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-017-0432-9

Keywords

Search

Navigation