Ruofan Liu
Erwin Wu
Hideki Koike
ABSTRACT
Video-based lessons are becoming a popular way for distance piano education. However, limited by the fixed camera angle, a video is difficult to tell precise 3D hand posture, which is one of the most essential factors for learning piano. This paper presents a visualization system providing the intuitive discrepancy of hand postures in two piano performance videos. Through a motion capture system, the estimated 3D postures are visualized and discrepancies based on distinct metrics are displayed, integrated with modular functions assisting skill acquisition. A pilot study proves that the proposed visualization can be a supplementary means for only video-based lessons in terms of correcting hand postures and fingering.
Supplemental Material
- John Brooke. 2013. SUS: A Retrospective. J. Usability Studies 8, 2 (feb 2013), 29–40.Google Scholar
Digital Library
- Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. 2019. Temporal Cycle-Consistency Learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Allen L Edwards. 1951. Balanced latin-square designs in psychological research. The American journal of psychology 64, 4 (1951), 598–603.Google Scholar
- Adrian Freed and Rimas Avizienis. 2000. A New Music Keyboard with Continuous Key-position Sensing and High-speed Communication. (01 2000).Google Scholar
- Natsuki Hamanishi and Jun Rekimoto. 2019. TTT: Time Synchronization Method by Time Distortion for VR Training Including Rapidly Moving Objects. In 25th ACM Symposium on Virtual Reality Software and Technology (Parramatta, NSW, Australia) (VRST ’19). Association for Computing Machinery, New York, NY, USA, Article 95, 2 pages. https://doi.org/10.1145/3359996.3364745Google ScholarDigital Library
- Kevin Huang, Thad Starner, Ellen Do, Gil Weinberg, Daniel Kohlsdorf, Claas Ahlrichs, and Ruediger Leibrandt. 2010. Mobile Music Touch: Mobile Tactile Stimulation for Passive Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 791–800. https://doi.org/10.1145/1753326.1753443Google ScholarDigital Library
- Hitoshi Kawasaki, Sohei Wakisaka, Hiroto Saito, Atsushi Hiyama, and Masahiko Inami. 2022. A System for Augmenting Humans’ Ability to Learn Kendama Tricks through Virtual Reality Training. In Augmented Humans 2022 (Kashiwa, Chiba, Japan) (AHs 2022). Association for Computing Machinery, New York, NY, USA, 152–161. https://doi.org/10.1145/3519391.3519404Google ScholarDigital Library
- Ruofan Liu, Erwin Wu, Chen-Chieh Liao, Hayato Nishioka, Shinichi Furuya, and Hideki Koike. 2022. Synchronized Hand Difference Visualization for Piano Learning. In ACM SIGGRAPH 2022 Posters (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 4, 2 pages. https://doi.org/10.1145/3532719.3543196Google ScholarDigital Library
- Markus Löchtefeld, Antonio Krüger, and Hans Gellersen. 2016. DeceptiBike: Assessing the Perception of Speed Deception in a Virtual Reality Training Bike System. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (Gothenburg, Sweden) (NordiCHI ’16). Association for Computing Machinery, New York, NY, USA, Article 40, 10 pages. https://doi.org/10.1145/2971485.2971513Google ScholarDigital Library
- Megan Lorains, Kevin Ball, and Clare MacMahon. 2013. An above real time training intervention for sport decision making. Psychology of Sport and Exercise 14, 5 (2013), 670–674. https://doi.org/10.1016/j.psychsport.2013.05.005Google ScholarCross Ref
- Andrew P. McPherson and Youngmoo E. Kim. 2011. Multidimensional Gesture Sensing at the Piano Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 2789–2798. https://doi.org/10.1145/1978942.1979355Google ScholarDigital Library
- Robert A. Moog and Thomas L. Rhea. 1990. Evolution of the Keyboard Interface: The Bosendorfer 290 SE Recording Piano and the Moog Multiply-Touch-Sensitive Keyboards. Computer Music Journal 14 (1990), 52.Google ScholarCross Ref
- Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. 2020. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In European Conference on Computer Vision. Springer, 548–564.Google ScholarDigital Library
- Takayuki Nozawa, Erwin Wu, and Hideki Koike. 2019. VR Ski Coach: Indoor Ski Training System Visualizing Difference from Leading Skier. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 1341–1342. https://doi.org/10.1109/VR.2019.8797717Google ScholarCross Ref
- Linsey Raymaekers, Jo Vermeulen, Kris Luyten, and Karin Coninx. 2014. Game of Tones: Learning to Play Songs on a Piano Using Projected Instructions and Games. (04 2014). https://doi.org/10.1145/2559206.2574799Google ScholarDigital Library
- Katja Rogers, Amrei Röhlig, Matthias Weing, Jan Gugenheimer, Bastian Könings, Melina Klepsch, Florian Schaub, Enrico Rukzio, Tina Seufert, and Michael Weber. 2014. P.I.A.N.O.: Faster Piano Learning with Interactive Projection. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (Dresden, Germany) (ITS ’14). Association for Computing Machinery, New York, NY, USA, 149–158. https://doi.org/10.1145/2669485.2669514Google ScholarDigital Library
- Austin Jerald Stanbury, Ines Said, and Hyo Jeong Kang. 2021. HoloKeys: Interactive Piano Education Using Augmented Reality and IoT. Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (2021).Google ScholarDigital Library
- Nobuhiro Takahashi, Shinichi Furuya, and Hideki Koike. 2020. Soft Exoskeleton Glove with Human Anatomical Architecture: Production of Dexterous Finger Movements and Skillful Piano Performance. IEEE Transactions on Haptics 13, 4 (2020), 679–690. https://doi.org/10.1109/TOH.2020.2993445Google ScholarDigital Library
- Yoshinari Takegawa, Tsutomu Terada, and Masahiko Tsukamoto. 2012. A piano learning support system considering rhythm. (01 2012), 325–332.Google Scholar
- Xizi Wang. 2020. PPVR: Teaching and Learning to Play the Piano in Virtual Reality. Master’s thesis. School of Information, University of Michigan. https://hdl.handle.net/2027.42/162561Google Scholar
- Erwin Wu, Mitski Piekenbrock, Takuto Nakumura, and Hideki Koike. 2021. SPinPong - Virtual Reality Table Tennis Skill Acquisition using Visual, Haptic and Temporal Cues. IEEE Transactions on Visualization and Computer Graphics 27, 5 (2021), 2566–2576. https://doi.org/10.1109/TVCG.2021.3067761Google ScholarCross Ref
Index Terms
- PianoHandSync: An Alignment-based Hand Pose Discrepancy Visualization System for Piano Learning
Recommendations
Following the Master’s Hands: Capturing Piano Performances for Mixed Reality Piano Learning Applications
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing SystemsPiano learning applications in Mixed Reality (MR) are a promising substitute for physical instruction when a piano teacher is absent. Existing piano learning applications that use visual indicators to highlight the notes to be played on the keyboard or ...
Synchronized Hand Difference Visualization for Piano Learning
SIGGRAPH '22: ACM SIGGRAPH 2022 PostersWhen learning a dexterous skill such as playing the piano, people commonly watch videos of a teacher. However, this conventional way has some downsides such as limited information to be retrieved and less intuitive instructions. We propose a virtual ...
Exploiting eye-hand coordination to detect grasping movements
Human beings are very skillful at reaching for and grasping objects under multiple conditions, even when faced with an object's wide variety of positions, locations, structures and orientations. This natural ability, controlled by the human brain, is ...
Comments