Skip to main content
SpringerLink
Log in

A novel approach to learning music and piano based on mixed reality and gamification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Learning music has been demonstrated to provide many benefits for children. However, music students, especially beginners, often suffer from lack of motivation and even can be frustrated if their musical skills do not improve as they practice over and over. In such situations, they usually end up dropping out of music school. To face this challenge, in this work a novel approach based on mixed reality and gamification is proposed to motivate music students. This approach has been validated thanks to HoloMusic XP, a multimedia tool that helps students learn music and piano. The devised architecture that supports HoloMusic XP has been designed and developed to scale when new music concepts must be addressed. Thanks to the use of mixed reality, the usually steep learning curve for beginner students can be mitigated and complex music concepts can be simplified due to the use of visual metaphors. The system has been evaluated in a real environment by teachers and students to measure its effectiveness and usability. After conducting the experiments, an increase in the students’ motivation and a general understanding of the multimedia representation have been achieved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Akçayır M, Akçayır G (2017) Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ Res Rev 20:1–11

    Article  Google Scholar 

  2. Cai M, Amrizal MA, Abe T, Suganuma T (2019) Design and implementation of ar-supported system for piano learning. In: 2019 IEEE 8th global conference on consumer electronics (GCCE). IEEE, pp 49–50

  3. Chou YK (2013) Octalysis: complete gamification framework. Yu-Kai Chou & Gamification

  4. Chow J, Feng H, Amor R, Wünsche BC (2013) Music education using augmented reality with a head mounted display. In: Proceedings of the fourteenth Australasian user interface conference, vol 139. Australian Computer Society, Inc., pp 73–79

  5. Comport AI, Marchand E, Pressigout M, Chaumette F (2006) Real-time markerless tracking for augmented reality: the virtual visual servoing framework. IEEE Trans Visual Comput Graph 12(4):615–628

    Article  Google Scholar 

  6. Costa-Giomi E, Flowers PJ, Sasaki W (2005) Piano lessons of beginning students who persist or drop out: teacher behavior, student behavior, and lesson progress. J Res Music Educ 53(3):234–247

    Google Scholar 

  7. Costanza P, Russell T (2017) Methodologies in music education. In: Critical essays in music education. Routledge, pp 255–266

  8. Dannenberg RB, Sanchez M, Joseph A, Capell P, Joseph R, Saul R (1990) An expert system for teaching piano to novices. In: ICMC

  9. Davis FD (1993) User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. Int J Man-Machine Stud 38 (3):475–487

    Article  Google Scholar 

  10. Degli Innocenti E, Geronazzo M, Vescovi D, Nordahl R, Serafin S, Ludovico LA, Avanzini F (2019) Mobile virtual reality for musical genre learning in primary education. Comput Educ 139:102–117

    Article  Google Scholar 

  11. Fletcher C, Hulusic V, Amelidis P (2019) Virtual reality ear training system: a study on spatialised audio in interval recognition. In: 2019 11th International conference on virtual worlds and games for serious applications (VS-Games). IEEE, pp 1–4

  12. Hackl D, Anthes C (2017) Holokeys-an augmented reality application for learning the piano. In: Forum media technology, pp 140–144

  13. Huang F, Zhou Y, Yu Y, Wang Z, Du S (2011) Piano ar: a markerless augmented reality based piano teaching system. In: 2011 Third international conference on intelligent human-machine systems and cybernetics, vol 2. IEEE, pp 47–52

  14. Kalantari M, Rauschnabel P (2018) Exploring the early adopters of augmented reality smart glasses: the case of microsoft hololens. In: Augmented reality and virtual reality. Springer, pp 229–245

  15. Martin-Gutierrez J, Del Rio Guerra MS, Lopez-Chao V, Soto Gastelum RH, Valenzuela Bojórquez JF (2020) Augmented reality to facilitate learning of the acoustic guitar. Appl Sci 10(7):2425

    Article  Google Scholar 

  16. McPherson GE, O’Neill SA (2010) Students’ motivation to study music as compared to other school subjects: a comparison of eight countries. Res Stud Music Educ 32(2):101–137

    Article  Google Scholar 

  17. Molloy W, Huang E, Wünsche BC (2019) Mixed reality piano tutor: a gamified piano practice environment. In: 2019 International conference on electronics, information, and communication (ICEIC). IEEE, pp 1–7

  18. Moog RA (1986) Midi: musical instrument digital interface. J Audio Eng Soc 34(5):394–404

    Google Scholar 

  19. Overy K (1998) Can music really “improve” the mind? Psychol Music 26(1):97–99

    Article  Google Scholar 

  20. Pan Z, Cheok AD, Yang H, Zhu J, Shi J (2006) Virtual reality and mixed reality for virtual learning environments. Comput Graph 30(1):20–28

    Article  Google Scholar 

  21. Pesek M, Vučko ž, Šavli P, Kavčič A, Marolt M (2020) Troubadour: a gamified e-learning platform for ear training. IEEE Access 8:97090–97102

    Article  Google Scholar 

  22. Rio-Guerra D, Sylvia M, Martin-Gutierrez J, Lopez-Chao VA, Flores Parra R, Ramirez Sosa MA (2019) Ar graphic representation of musical notes for self-learning on guitar. Appl Sci 9(21):4527

    Article  Google Scholar 

  23. Rusiñol M, Chazalon J, Diaz-Chito K (2018) Augmented songbook: an augmented reality educational application for raising music awareness. Multimed Tools Appl 77(11):13773–13798

    Article  Google Scholar 

  24. Smoliar SW, Waterworth JA, Kellock PR (1995) Pianoforte: a system for piano education beyond notation literacy. In: ACM multimedia, vol 95

  25. Walsh A (2009) Quick response codes and libraries. Library Hi Tech News 26(5/6):7–9

    Article  Google Scholar 

  26. Yin J, Wang Y, Hsu D (2005) Digital violin tutor: an integrated system for beginning violin learners. In: Proceedings of the 13th annual ACM international conference on multimedia. ACM, pp 976–985

  27. Zeng H, He X, Pan H (2019) Funpianoar: a novel ar application for piano learning considering paired play based on multi-marker tracking. J Phys: Conf Ser 1229:012072. IOP Publishing

    Google Scholar 

Download references

Acknowledgements

This research was partially funded by the Department of Technologies and Information Systems (grant number 00421372) and by the University of Castilla-La Mancha (AIR Research Group, grant number 2020-GRIN-28886).

Author information

Authors and Affiliations

  1. University of Castilla-La Mancha, Paseo de la Universidad 4, 13071, Ciudad Real, Spain

    D. Molero, S. Schez-Sobrino, D. Vallejo, C. Glez-Morcillo & J. Albusac

Authors
  1. D. Molero
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Schez-Sobrino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Vallejo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. C. Glez-Morcillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. J. Albusac
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to D. Vallejo.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Molero, D., Schez-Sobrino, S., Vallejo, D. et al. A novel approach to learning music and piano based on mixed reality and gamification. Multimed Tools Appl 80, 165–186 (2021). https://doi.org/10.1007/s11042-020-09678-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09678-9

Keywords

Search

Navigation