Skip to main content

Advertisement

Springer Nature Link
Log in

Interactive piano training using augmented reality and the Internet of Things

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

The technology of augmented reality in education is a significant attempt to reform the training mode and develop education in general. Piano training usually includes using well-known computer means of teaching music on-screen and hardware. In this context, augmented reality (AR), alone and integrated with the Internet of Things (IoT), is a promising solution for interactive piano learning. The present study seeks to investigate the impacts of an AR and IoT assisted system for piano education on learners’ musical literacy and piano skills (resonance, rhythm, coherence, pace, fingering, playing with both hands and legato playing). As an interactive learning environment, the study integrates HoloKeys, an AR application. This research builds on an empirical approach through a quasi-experiment. It involved 100 students of Tourism College of Zhejiang, China. The study participants were divided into control and experimental groups. The results showed that the experimental group achieved significantly higher results in a piano course than the control group. Hence, it confirms the positive perspective of using AR and the IoT in piano training. The research results can be useful to design and implement modern piano training courses. Thanks to the advantages that AR and IoT technologies offer when shaping interactive learning environments, the application of the present findings goes beyond music education and has the potential to boost learning in other specialties. Future research should focus on designing and developing AR learning tools and additional learning resources. These instruments will provide students with high-quality technical support for piano training.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Data will be available on request.

References

  • Bai, B. (2021). Piano learning in the context of schooling during China’s “Piano craze” and beyond: Motivations and pathways. Music Education Research, 23(4), 512–526. https://doi.org/10.1080/14613808.2021.1929139.

    Article  Google Scholar 

  • Bauer, V., Nagele, A., Baume, C., Cowlishaw, T., Cooke, H., Pike, C., & Healey, P. G. (2019). Designing an interactive and collaborative experience in audio augmented reality. In International Conference on Virtual Reality and Augmented Reality (pp. 305–311). Springer. https://doi.org/10.1007/978-3-030-31908-3_20.

  • Birhanu, A., & Rank, S. (2017). KeynVision: exploring piano pedagogy in mixed reality. In Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play (pp. 299–304). ACM. https://doi.org/10.1145/3130859.3131336.

  • Cai, M., Amrizal, M. A., Abe, T., & Suganuma, T. (2019). Design and implementation of AR-supported system for piano learning. In 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE) (pp. 49–50). IEEE. https://doi.org/10.1109/GCCE46687.2019.9015530.

  • Chang, K. E., Zhang, J., Huang, Y. S., Liu, T. C., & Sung, Y. T. (2020). Applying augmented reality in physical education on motor skills learning. Interactive Learning Environments, 28(6), 685–697. https://doi.org/10.1080/10494820.2019.1636073.

    Article  Google Scholar 

  • Chow, J., Feng, H., Amor, R., & Wünsche, B. C. (2013). Music education using augmented reality with a head mounted display. In Proceedings of the Fourteenth Australasian User Interface Conference (Vol. 139, pp. 73–79). ACM.

  • Das, S., Glickman, S., Hsiao, F. Y., & Lee, B. (2017). Music everywhere–Augmented reality piano improvisation learning system. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME). Aalborg University Copenhagen, Copenhagen, Denmark (pp. 511–512). NIME.

  • Desbruslais, S. (2018). The Music and Music Theory of Paul Hindemith. Boydell Press.

    Book  Google Scholar 

  • Di Serio, Á., Ibáñez, M. B., & Kloos, C. D. (2013). Impact of an augmented reality system on students’ motivation for a visual art course. Computers & Education, 68, 586–596. https://doi.org/10.1016/j.compedu.2012.03.002.

    Article  Google Scholar 

  • Fernandez, C. A. T., Paliyawan, P., Yin, C. C., & Thawonmas, R. (2016). Piano learning application with feedback provided by an AR virtual character. In 2016 IEEE 5th Global Conference on Consumer Electronics (pp. 1–2). Kyoto, Japan. IEEE. https://doi.org/10.1109/GCCE.2016.7800380.

  • Hackl, D., & Anthes, C. (2017). HoloKeys-An augmented reality application for learning the piano. In Forum Media Technology (pp. 140–144). CEUR.

  • Huang, T. C., Chen, C. C., & Chou, Y. W. (2016). Animating eco-education: To see, feel, and discover in an augmented reality-based experiential learning environment. Computers & Education, 96, 72–82. https://doi.org/10.1016/j.compedu.2016.02.008.

    Article  Google Scholar 

  • Julia, J., Kurnia, D., Isrokatun, I., Wulandari, H., & Aisyah, I. (2019). The use of the Synthesia application to simplify Angklung learning. In Journal of Physics: Conference Series (Vol. 1318, No. 1, p. 012040). IOP Publishing. https://doi.org/10.1088/1742-6596/1318/1/012040.

  • Lemos, B., Corrêa, A. G., Nascimento, M., & Lopes, R. D. D. (2017). Augmented reality musical app to support children’s musical education. Computer Science and Information Technology, 5(4), 121–127. https://doi.org/10.13189/csit.2017.050401.

  • Li, L. (2018). Application of augmented reality technology in piano teaching system design. Educational Sciences: Theory & Practice, 18(5), 1712–1721. https://doi.org/10.12738/estp.2018.5.070.

  • Lv, Z., Lloret, J., & Song, H. (2020). Internet of Things and augmented reality in the age of 5G. Computer Communications, 164, 158–161. https://doi.org/10.1016/j.comcom.2020.08.019.

    Article  Google Scholar 

  • MacDonald, R. A. (2013). Music, health, and well-being: A review. International Journal of Qualitative Studies on Health and Well-Being, 8(1), 20635. https://doi.org/10.3402/qhw.v8i0.20635.

    Article  Google Scholar 

  • Molero, D., Schez-Sobrino, S., Vallejo, D., Glez-Morcillo, C., & Albusac, J. (2021). A novel approach to learning music and piano based on mixed reality and gamification. Multimedia Tools and Applications, 80(1), 165–186. https://doi.org/10.1007/s11042-020-09678-9.

    Article  Google Scholar 

  • Pirgon, Y. (2013). Piyano eğitiminde günlük çalişma yapmanin öğrenci başarisi üzerine etkisi. Journal of Research in Education and Teaching, 2(3), 41–49.

    Google Scholar 

  • Rigby, L., Wünsche, B. C., & Shaw, A. (2020). piARno-an augmented reality piano tutor. In 32nd Australian Conference on Human-Computer Interaction (pp. 481–491). ACM. https://doi.org/10.1145/3441000.3441039.

  • Robb, A., White, C., Cordar, A., Wendling, A., Lampotang, S., & Lok, B. (2015). A comparison of speaking up behavior during conflict with real and virtual humans. Computers in Human Behavior, 52, 12–21. https://doi.org/10.1016/j.chb.2015.05.043.

    Article  Google Scholar 

  • Santini, G. (2020). Augmented piano in augmented reality. In Proceedings of the International Conference New Interfaces Musical Expression (pp. 411–415). NIME.

  • Simion, A., Iftene, A., & Gîfu, D. (2021). An augmented reality piano learning tool. In Proceedings of the 18th International Conference on Human-Computer Interaction RoCHI (pp. 16–17). Bucharest, Romania / Online, September 16-17, 2021. RoCHI.

  • SmartMusic (2022). Music Learning Software for Educators & Students. Retrieved April 15, 2022, from https://www.smartmusic.com/.

  • Spieker, M. H. (2016). AP music theory applied. General Music Today, 30(1), 16–19. https://doi.org/10.1177/1048371316641462.

    Article  Google Scholar 

  • SynthesiaGame (2022). Synthesia, Piano for Everyone. Retrieved April 15, 2022, from https://www.synthesiagame.com/.

  • Trujano, F., Khan, M., & Maes, P. (2018). ARPiano efficient music learning using augmented reality. In International Conference on Innovative Technologies and Learning (pp. 3–17). Springer. https://doi.org/10.1007/978-3-319-99737-7_1.

  • Wang, C. Y., & Cheng, Y. (2016). Summary of research on augmented reality educational application products. Modern Educational Technology, 26(5), 95–101.

    MathSciNet  Google Scholar 

  • Weing, M., Röhlig, A., Rogers, K., Gugenheimer, J., Schaub, F., Könings, B., Rukzio, E., & Weber, M. (2013). PIANO: enhancing instrument learning via interactive projected augmentation. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication (pp. 75–78). ACM. https://doi.org/10.1145/2494091.2494113.

  • XiaoSquared (2022a). Andante: a walking tempo. Retrieved April 15, 2022a, from http://portfolio.xiaosquared.com/Andante.

  • XiaoSquared (2022b). Perpetual Canon. Retrieved April 15, 2022b, from http://portfolio.xiaosquared.com/Perpetual-Canon.

  • Zeng, H., He, X., & Pan, H. (2019). FunPianoAR: A novel AR application for piano learning considering paired play based on multi-marker tracking. In Journal of Physics: Conference Series (Vol. 1229, No. 1, p. 012072). IOP Publishing. https://doi.org/10.1088/1742-6596/1229/1/012072.

  • Zhang, J. (2020). Real-time detection of energy consumption of IoT network nodes based on artificial intelligence. Computer Communications, 153, 188–195. https://doi.org/10.1016/j.comcom.2020.02.015.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Academy of Arts, Tourism College of Zhejiang, Hangzhou, China

    Yiqun Chen

Authors
  1. Yiqun Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yiqun Chen.

Ethics declarations

Conflict of interests

Authors declare that they have no conflict of interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A

Appendix A

1.1 Piano theory test

  1. 1.

    Types of polyphony.

  2. 2.

    The concept of imitative polyphony.

  3. 3.

    Methods for creating polyphonic texture.

  4. 4.

    Key stages in polyphonic music development.

  5. 5.

    The specifics of sound production and intonation in cantilena.

  6. 6.

    Ways to achieve a high-quality virtuoso performance.

  7. 7.

    Means of musical expression: dynamics, accent, pedalization, and articulation.

  8. 8.

    Phrasing as an artistic and semantic division of the musical language.

  9. 9.

    Things to consider when learning and playing large-form piano pieces (sonata/concert/variations).

  10. 10.

    Sonata, concert and variation forms and their performance interpretation.

  11. 11.

    The concept of individual performing style.

  12. 12.

    Fingering, playing with both hands, the freedom of the performing apparatus, the interplay of artistic and technical skills.

  13. 13.

    The vision of the piano pieces, its development and embodiment during rehearsals and performance.

  14. 14.

    An objective meaning of the piano piece and its subjective interpretation; compliance with the composer’s intention, cultural and historical contexts, genre and style requirements, and the principles of musical dramaturgy.

  15. 15.

    The interpretation of music pieces written for an ensemble.

  16. 16.

    A creative approach to performance as an indicator of high musical culture.

  17. 17.

    How musicians improve their artistic taste and develop individual performing styles.

  18. 18.

    Interpretation of the piano score.

  19. 19.

    Modeling and practical implementation of one's own vision for the piano performance in accordance with the genre, style and artistic characteristics of the music piece.

  20. 20.

    Musical dramaturgy of a large-form piano piece.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y. Interactive piano training using augmented reality and the Internet of Things. Educ Inf Technol 28, 6373–6389 (2023). https://doi.org/10.1007/s10639-022-11443-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-022-11443-4

Keywords