Hearing the Moment with MetaEcho! From Physical to Virtual in Synchronized Sound Recording
Authors: 
Zheng Wei, Yuzheng Chen, Wai Tong, Xuan Zong, Huamin Qu, Xian Xu, Lik-Hang LeeAuthors Info & Claims
Zheng Wei, Yuzheng Chen, Wai Tong, Xuan Zong, Huamin Qu, Xian Xu, Lik-Hang LeeAuthors Info & ClaimsPages 6520 - 6529
Abstract
In film education, high expenses and limited space significantly challenge teaching synchronized sound recording (SSR). Traditional methods, which emphasize theory with limited practical experience, often fail to bridge the gap between theoretical understanding and practical application. As such, we introduce MetaEcho, an educational virtual reality leveraging the presence theory for teaching SSR. MetaEcho provides realistic simulations of various recording equipment and facilitates communication between learners and instructors, offering an immersive learning experience that closely mirrors actual practices. An evaluation with 24 students demonstrated that MetaEcho surpasses the traditional method in presence, collaboration, usability, realism, comprehensibility, and creativity. Three experts also commented on the benefits of MetaEcho and the opportunities for promoting SSR education in the metaverse era.
References
[1]
Olusola O Adesope, Dominic A Trevisan, and Narayankripa Sundararajan. 2017. Rethinking the use of tests: A meta-analysis of practice testing. Review of educational research 87, 3 (2017), 659--701.
[2]
Ammar Ahmad, Cyrille Migniot, and Albert Dipanda. 2019. Hand pose estimation and tracking in real and virtual interaction: A review. Image and Vision Computing 89 (2019), 35--49.
[3]
Stephen Andriano-Moore. 2018. The rise of the sound designer: Northern California film sound in the 1960s and 1970s. Historical Journal of Film, Radio and Television 38, 3 (2018), 536--554.
[4]
Alessandro Antonietti and Manuela Cantoia. 2000. To see a painting versus to walk in a painting: an experiment on sense-making through virtual reality. Computers & Education 34, 3--4 (2000), 213--223.
[5]
Bruce Bartlett and Jenny Bartlett. 2016. Practical Recording Techniques: The step-by-step approach to professional audio recording. Routledge.
[6]
Wendy Bolier, Wolfgang Hürst, Guido Van Bommel, Joost Bosman, and Harriët Bosman. 2018. Drawing in a virtual 3D space-introducing VR drawing in elementary school art education. In Proceedings of the 26th ACM international conference on Multimedia. 337--345.
[7]
David J Chatting, Josie S Galpin, and Judith S Donath. 2006. Presence and portrayal: video for casual home dialogues. In Proceedings of the 14th ACM international conference on Multimedia. 395--401.
[8]
Yu-Hsuan Chen, Chen-Wei Fu, Wei-Lun Huang, Ming-Cong Su, Hsin-Yu Huang, AndrewChen, and Tse-Yu Pan. 2023. SetterVision: Motion-based Tactical Training System for Volleyball Setters in Virtual Reality. In Proceedings of the 31st ACM International Conference on Multimedia. 9382--9384.
[9]
Donald Crafton. 1999. The talkies: American cinema's transition to sound, 1926--1931. Vol. 4. Univ of California Press.
[10]
Barney Dalgarno and Mark JW Lee. 2010. What are the learning affordances of 3-D virtual environments' British journal of educational technology 41, 1 (2010), 10--32.
[11]
Ralf Doerner and Robin Horst. 2022. Overcoming challenges when teaching hands-on courses about Virtual Reality and Augmented Reality: Methods, techniques and best practice. Graphics and Visual Computing 6 (2022), 200037.
[12]
Scott Eyman. 1997. The speed of sound: Hollywood and the talkie revolution 1926--1930. Simon and Schuster.
[13]
Laura Freina and Michela Ott. 2015. A literature review on immersive virtual reality in education: state of the art and perspectives. In The international scientific conference elearning and software for education, Vol. 1. 10--1007.
[14]
Shari Geller. 2021. Cultivating online therapeutic presence: Strengthening therapeutic relationships in teletherapy sessions. Counselling Psychology Quarterly 34, 3--4 (2021), 687--703.
[15]
Mariana-Daniela González-Zamar and Emilio Abad-Segura. 2020. Implications of virtual reality in arts education: Research analysis in the context of higher education. Education Sciences 10, 9 (2020), 225.
[16]
SIMON HORROCKS. 2018. SOUND RECORDING ON A BUDGET FOR FILM VIDEO. https://momofilmfest.com/budget-sound-recording-film-video/
[17]
Qiao Jin, Yu Liu, Svetlana Yarosh, Bo Han, and Feng Qian. 2022. How will vr enter university classrooms? multi-stakeholders investigation of vr in higher education. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1--17.
[18]
Sam Kavanagh, Andrew Luxton-Reilly, Burkhard Wuensche, and Beryl Plimmer. 2017. A systematic review of virtual reality in education. Themes in Science and Technology Education 10, 2 (2017), 85--119.
[19]
Sebastian Kernbach, Sabrina Bresciani, and Martin J Eppler. 2015. Slip-slidingaway: A review of the literature on the constraining qualities of PowerPoint. Business and Professional Communication Quarterly 78, 3 (2015), 292--313.
[20]
Nawel Khenak, Jeanne Vezien, and Patrick Bourdot. 2020. Spatial presence, performance, and behavior between real, remote, and virtual immersive environments. IEEE Transactions on Visualization and Computer Graphics 26, 12 (2020), 3467--3478.
[21]
Wanda Lazar. 1998. Sound for film: Audio education for filmmakers. Journal of Film and Video 50, 3 (1998), 54--61.
[22]
Elinda Ai-Lim Lee, Kok Wai Wong, and Chun Che Fung. 2010. How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Computers & Education 55, 4 (2010), 1424--1442.
[23]
Mariana Lopez. 2020. Gamifying the Learning Experience: Evaluating Teaching Methodologies in an Introductory Module to Sound for Film and Television. In Audio Education. Routledge, 81--99.
[24]
Xinyi Luo, YuyangWang, Lik-Hang Lee, Zihan Xing, Shan Jin, Boya Dong, Yuanyi Hu, Zeming Chen, Jing Yan, and Pan Hui. 2023. Using a virtual reality interview simulator to explore factors influencing people?s behavior. Virtual Reality 28 (2023), 1--20. https://api.semanticscholar.org/CorpusID:258685730
[25]
Patrick E McKnight and Julius Najab. 2010. Mann-Whitney U Test. The Corsini encyclopedia of psychology (2010), 1--1.
[26]
Tassos A Mikropoulos and Antonis Natsis. 2011. Educational virtual environments: A ten-year review of empirical research (1999--2009). Computers & education 56, 3 (2011), 769--780.
[27]
Miriam Mulders, Josef Buchner, and Michael Kerres. 2020. A framework for the use of immersive virtual reality in learning environments. International Journal of Emerging Technologies in Learning (iJET) 15, 24 (2020), 208--224.
[28]
Leo Murray. 2019. Sound design theory and practice: Working with sound. Routledge.
[29]
Michael Nebeling, Shwetha Rajaram, Liwei Wu, Yifei Cheng, and Jaylin Herskovitz. 2021. Xrstudio: A virtual production and live streaming system for immersive instructional experiences. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--12.
[30]
Alec Nisbett. 2003. Sound Studio: Audio Techniques for Radio, Television, Film and Recording. Routledge.
[31]
Thomas Ohanian and Natalie Phillips. 2013. Digital filmmaking: the changing art and craft of making motion pictures. Routledge.
[32]
George Papanastasiou, Athanasios Drigas, Charalabos Skianis, Miltiadis Lytras, and Effrosyni Papanastasiou. 2019. Virtual and augmented reality effects on K-12, higher and tertiary education students? twenty-first century skills. Virtual Reality 23, 4 (2019), 425--436.
[33]
Nikolaos Pellas, Stylianos Mystakidis, and Ioannis Kazanidis. 2021. Immersive Virtual Reality in K-12 and Higher Education: A systematic review of the last decade scientific literature. Virtual Reality 25, 3 (2021), 835--861.
[34]
Gustav Bøg Petersen, Giorgos Petkakis, and Guido Makransky. 2022. A study of how immersion and interactivity drive VR learning. Computers & Education 179 (2022), 104429.
[35]
Daniele Prina. 2017. Creative skill sets in sound jobs (1/3). https://www.sounde sign.info/2017/02/27/creative-skill-sets-sound-jobs/
[36]
Jaziar Radianti, Tim A Majchrzak, Jennifer Fromm, and Isabell Wohlgenannt. 2020. A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers & education 147 (2020), 103778.
[37]
Henri Rapp. 2020. Everything You Need to Know About Recording Production Sound for Film. https://nofilmschool.com/what-you-need-know-aboutrecording- production-sound
[38]
Jack Ratcliffe and Laurissa Tokarchuk. 2020. Presence, embodied interaction and motivation: distinct learning phenomena in an immersive virtual environment. In Proceedings of the 28th ACM International Conference on Multimedia. 3661--3668.
[39]
Katherine A Rawson and John Dunlosky. 2012. When is practice testing most effective for improving the durability and efficiency of student learning? Educational Psychology Review 24, 3 (2012), 419--435.
[40]
James A Russell and Lawrence MWard. 1982. Environmental psychology. Annual review of psychology 33, 1 (1982), 651--689.
[41]
Greg S Ruthenbeck and Karen J Reynolds. 2015. Virtual reality for medical training: the state-of-the-art. Journal of Simulation 9 (2015), 16--26.
[42]
Majed Samad, Elia Gatti, Anne Hermes, Hrvoje Benko, and Cesare Parise. 2019. Pseudo-haptic weight: Changing the perceived weight of virtual objects by manipulating control-display ratio. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--13.
[43]
Thomas Schubert, Frank Friedmann, and Holger Regenbrecht. 2001. The experience of presence: Factor analytic insights. Presence: Teleoperators & Virtual Environments 10, 3 (2001), 266--281.
[44]
Dong-Hee Shin. 2017. The role of affordance in the experience of virtual reality learning: Technological and affective affordances in virtual reality. Telematics and Informatics 34, 8 (2017), 1826--1836.
[45]
Guillermo Marcos Sommer, Johannes Broschewitz, Sabine Huppert, Christina Gesine Sommer, Nora Jahn, Boris Jansen-Winkeln, Ines Gockel, and Hans-Michael Hau. 2021. The role of virtual reality simulation in surgical training in the light of COVID-19 pandemic: Visual spatial ability as a predictor for improved surgical performance: a randomized trial. Medicine 100, 50 (2021), e27844.
[46]
Wai Tong, Meng Xia, and Huamin Qu. 2024. Exploring Stage Lighting Education in Metaverse. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. 1--6.
[47]
Stefan Weber, David Weibel, and Fred W Mast. 2021. How to get there when you are there already? Defining presence in virtual reality and the importance of perceived realism. Frontiers in psychology 12 (2021), 628298.
[48]
Zheng Wei, Xian Xu, Lik-Hang Lee, Wai Tong, Huamin Qu, and Pan Hui. 2023. Feeling Present! From Physical to Virtual Cinematography Lighting Education with Metashadow. In Proceedings of the 31st ACM International Conference on Multimedia. 1127--1136.
[49]
Min-Yu Wu, Pai-Wen Ting, Ya-Hui Tang, En-Te Chou, and Li-Chen Fu. 2020. Hand pose estimation in object-interaction based on deep learning for virtual reality applications. Journal of Visual Communication and Image Representation 70 (2020), 102802.
[50]
Hilary Wyatt and Tim Amyes. 2013. Audio post production for television and film: an introduction to technology and techniques. Routledge.
[51]
Ding Xingeng, Liu Jianxiang, et al. 2012. Advantages and disadvantages of PowerPoint in lectures to science students. IJ Education and Management Engineering 9, 1 (2012), 61--65.
[52]
Xian Xu, Wai Tong, Zheng Wei, Meng Xia, Lik-Hang Lee, and Huamin Qu. 2023. Cinematography in the Metaverse: Exploring the Lighting Education on a Soundstage. In 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 571--572.
Index Terms
- Hearing the Moment with MetaEcho! From Physical to Virtual in Synchronized Sound Recording
Recommendations
The presence of an avatar can reduce cybersickness in Virtual Reality
AbstractVirtual Reality (VR) applications are increasingly being utilized for research, healthcare, and education. Despite their benefits, many VR users report motion sickness-like sensations (cybersickness), such as headache, disorientation, or nausea. ...
Why virtual, why environments? Implementing virtual reality concepts in computer-assisted language learning
The research area of computer-assisted language learning (CALL) has recently shifted from seeing the computer as a tool or partner to seeing it as a virtual environment where learners can collaborate and interact with a wide variety of native speakers. ...
Presence in virtual environments as a function of visual and auditory cues
VRAIS '95: Proceedings of the Virtual Reality Annual International Symposium (VRAIS'95)The paper reports the results of two experiments each investigating the sense of presence within visual and auditory virtual environments. The variables for the studies included the presence or absence of head tracking, the presence or absence of ...
Comments
Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Hong Kong Polytechnic University?s Start-up Fund for New Recruits
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 185Total Downloads
- Downloads (Last 12 months)185
- Downloads (Last 6 weeks)141
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in