Skip to main content

Advertisement

SpringerLink
Log in

Interactive multi-sensory and volumetric content integration for music education applications

  • 1215: Multimodal Interaction and IoT Applications
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Taiwan’s heritage in terms of the local music culture has been gradually fading in recent years. Thus, boosting and passing on the local music culture to pre-school and elementary school students are urgent requirements. This study proposes an interactive integration of multi-sensory and volumetric content for music education in Taiwan into applications for children. Further, the study introduces a technological multi-sensory pop-up sketch book created in collaboration with the National Taiwan Symphony Orchestra (NTSO) and Industrial Technology Research Institute (ITRI). Both organizations collaborate to integrate emerging media technologies, including augmented reality (AR) and volumetric capture for content production, and creative music teaching methods, derived from traditional pop-up sketch books. The final product featured 3D animated videos to achieve interactive learning with digital formation additional to the real worlds. This AR multi-sensory pop-up sketch book utilizes advanced volumetric capture technology to capture the motions of main actors in vivid 3D animation. Besides, modularized pop-up cards of musical instruments provide a haptic experience to complement the story. The book targets children aged between 3 and 12 years. An actual reading survey was conducted on 497 students from five kindergartens in Taiwan. Satisfaction with the book was rated using a five-rank scale. The cluster random sampling method was used for data analysis. Results of t-test produced an average score of 4.9980, which indicated that the target audience ranked the book with high levels of satisfaction. The results also confirmed that the story line was familiar among young children within the target age range. Additionally, using digital audio–visual augmented reality technology will be conducive to young children in terms of acceptance and recognition of traditional music culture. 3D animations of famous intellectual property characters aroused the interest of readers in learning about music through interactive contents. Moreover, this study provided evidence that the STEAM education model, which represents a cross-curricular approach that integrates Science, Technology, Engineering, Art, and Mathematics, is applicable to the inheritance and development of the local music culture.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Alakärppä I, Jaakkola E, Väyrynen J, Häkkilä J (2017) Using nature elements in mobile AR for education with children. MobileHCI ‘17: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, 1–13. https://doi.org/10.1145/3098279.3098547

  2. Barnes J, FakhrHosseini SM, Vasey E, Park CH, Jeon M (2019) Informal STEAM education case study: child-robot musical theater. CHI EA ‘19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–6. https://doi.org/10.1145/3290607.3312890

  3. Guo K, Lincoln P, Davidson P, Busch J, Yu X, Whalen M, … Izadi S (2019) The relightables: volumetric performance capture of humans with realistic relighting. ACM Trans Graph 38(6):1–19. https://doi.org/10.1145/3355089.3356571

  4. Hilton A, Beresford D, Gentils T, Smith R, Sun W, Illingworth J (2000)Whole-body modelling of people from multi-view images to populate virtual worlds. Vis Comput 16(7):411–436. https://doi.org/10.1007/PL00013395

    Article  MATH  Google Scholar 

  5. Horváth I (2018) Evolution of teaching roles and tasks in VR/AR-based Education. 2018 9th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), 355–360. https://doi.org/10.1109/CogInfoCom.2018.8639907

  6. Jung S-H, Ko J-W, Heo S-H, Kyung B-P(2019) A Research for Functional Game based on AR Smartcare: Focusing on infant and child safety education. J Korea Game Soc 19(1):25–36. https://doi.org/10.7583/JKGS.2019.19.1.25

    Article  Google Scholar 

  7. Kening Z (2016) Panel: virtual reality and augmented reality for education. Symposium on Education - Talks, Siggraph Asia, pp 1–22016.https://doi.org/10.1145/2993363.3006041

  8. Lee L, Ho HJ (2018) Exploring Young Children’s Communication Development through the Soundbeam Trigger Modes in the ‘Holistic Music Educational Approach for Young Children’ Programme’. Malays J Music 7:1–19. https://doi.org/10.37134/MJM.VOL7.1.2018

    Article  Google Scholar 

  9. Lemos B, Corrêa AG, Nascimento MD, Lopes RDD (2017) Augmented reality musical app to support children’s musical education. Comput Sci Inf Technol. https://doi.org/10.13189/CSIT.2017.050401

    Article  Google Scholar 

  10. Mahayuddin ZR, Saif AFMS (2020) Augmented reality based AR Alphabets towards improved learning process in primary education system. J Crit Rev 7(19):514–521. https://doi.org/10.31838/jcr.07.19.66

    Article  Google Scholar 

  11. Makhkamova A, Exner JP, Greff T, Werth D (2020) Towards a taxonomy of virtual reality usage in education: a systematic review. In: Jung T, Tom Dieck M, Rauschnabel P (eds) Augmented Reality and Virtual Reality. Progress in IS. Springer, Cham. https://doi.org/10.1007/978-3-030-37869-1_23

  12. Maraffi C (2003) Maya character creation: modeling and animation controls. New Riders. ISBN 10: 0735713448 ISBN 13: 9780735713444

  13. Regenbrecht H, Ott C, Park N, Duncan S, Collins J (2021) Voxelvideos for entertainment, education, and training. IEEE Access 9:68185–68196. https://doi.org/10.1109/ACCESS.2021.3076488

    Article  Google Scholar 

  14. Shatunova O, Anisimova T, Sabirova F, Kalimullina O (2019) STEAM as an innovative educational technology. J Soc Stud Educ Res 10(2):131–144

    Google Scholar 

  15. Starck J, Hilton A (2007) Surface capture for performance-based animation. IEEE Comput Graph Appl 27(3):21–31. https://doi.org/10.1109/MCG.2007.68

    Article  Google Scholar 

  16. Wei CC (2015) Media form design for picture book based on attractiveness factors and children’s attention. [Doctoral dissertation, National Cheng Kung University, Tainan, Taiwan]. http://ir.lib.ncku.edu.tw/handle/987654321/155997

  17. Yannier N, Koedinger KR, Hudson SE (2015) Learning from mixed-reality games: is shaking a tablet as effective as physical observation? Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp 1045–1054. https://doi.org/10.1145/2702123.2702397

Download references

Acknowledgements

This project is supported by a grant from the Ministry of Culture, R.O.C. (Taiwan). Fruit Grandma is under the license of the Taiwan Public Television Service Foundation and IFKIDS Theatre Company. Helpful advice about children education and data analysis from Dr. Liza Lee, the Dean of the College of Humanities and Social Sciences and a Professor in the Department of Early Childhood Development and Education in Chaoyang University of Technology, is highly appreciated. Moreover, advice about revision in terms of technical writing and presentation from Dr. Chia-Hung Yeh, the Distinguished Professor of the Department of Electrical Engineering in National Taiwan Normal University, greatly improved the quality of the paper.

Funding

This work was supported by the Ministry of Culture, R.O.C. (Taiwan) [grant number 1083035025].

Author information

Authors and Affiliations

  1. National Taiwan Symphony Orchestra, Taichung, Taiwan

    Chin-Ling Ho

  2. Chaoyang University of Technology, Taichung, Taiwan

    Chin-Ling Ho

  3. Service Systems Technology Center, Industrial Technology Research Institute, Hsinchu, Taiwan

    Tsang-Gang Lin & Chan-Ru Chang

  4. Interplan International Corporation, Taipei, Taiwan

    Tsang-Gang Lin & Chan-Ru Chang

Authors
  1. Chin-Ling Ho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tsang-Gang Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chan-Ru Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chin-Ling Ho.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ho, CL., Lin, TG. & Chang, CR. Interactive multi-sensory and volumetric content integration for music education applications. Multimed Tools Appl 82, 4847–4862 (2023). https://doi.org/10.1007/s11042-022-12314-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12314-3

Keywords

Search

Navigation