ABSTRACT
The process of learning the piano for novices is usually difficult and time-consuming. Several approaches in augmented reality such as piano-roll visualizations have been explored but have not garnered enough success and adoption. These piano roll prototypes have introduced several features and modules that assist novices on aspects such in sight reading, timing and many others. However, improvisation, the act of allowing the piano user to incorporate their personal touch into their performance, and personalised learning have not been much explored in this domain. In this PhD, we are going to explore how we can encourage piano learners to improvise with the use of adaptive piano roll visualisations. Specifically, we are going to investigate how heuristics defined by experts and spatiotemporal models can be used to design visualisations that motivate and encourage learners based on their personalised learning patterns. Using these models and inputs, we will design and build a piano roll training system integrated with adaptive visualisations that serve as intervention helping learners. We will evaluate and compare these visualisations in various user studies where they get to play piano pieces and develop their improvisation skills. We intend to uncover whether these adaptive visualisations will be helpful in the overall training of piano learners. Additionally, we wish to explore whether these adaptive visualisations will allow us to discover affordances that can potentially improve piano learning in general.
- István Barakonyi and Dieter Schmalstieg. 2005. Augmented reality agents in the development pipeline of computer entertainment. In International Conference on Entertainment Computing. Springer, 345–356.Google ScholarDigital Library
- Jonathan Chow, Haoyang Feng, Robert Amor, and Burkhard C Wünsche. 2013. Music education using augmented reality with a head mounted display. In Proceedings of the Fourteenth Australasian User Interface Conference-Volume 139. 73–79.Google ScholarDigital Library
- Yawen Eunice Chyu. 2004. Teaching improvisation to piano students of elementary to intermediate levels. Ph.D. Dissertation. The Ohio State University.Google Scholar
- Peter Cope and Hugh Smith. 1997. Cultural context in musical instrument learning. British Journal of Music Education 14, 3 (1997), 283–289.Google ScholarCross Ref
- Andrea Creech. 2010. Learning a musical instrument: The case for parental support. Music Education Research 12, 1 (2010), 13–32.Google ScholarCross Ref
- Ryan Daniel. 2006. Exploring music instrument teaching and learning environments: Video analysis as a means of elucidating process and learning outcomes. Music Education Research 8, 2 (2006), 191–215.Google ScholarCross Ref
- Jordan Aiko Deja. 2021. Adaptive Visualisations Using Spatiotemporal and Heuristic Models to Support Piano Learning. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (Utrecht, Netherlands) (UMAP ’21). Association for Computing Machinery, New York, NY, USA, 286–290. https://doi.org/10.1145/3450613.3459656Google ScholarDigital Library
- Dominique Fober, Stéphane Letz, and Yann Orlarey. 2007. Vemus-feedback and groupware technologies for music instrument learning.Google Scholar
- Alix Goguey, Carl Gutwin, Zhe Chen, Pang Suwanaposee, and Andy Cockburn. 2021. Interaction Pace and User Preferences. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445772Google Scholar
- Marc Hassenzahl, Michael Burmester, and Franz Koller. 2003. AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In Mensch & computer 2003. Springer, 187–196.Google ScholarCross Ref
- Feng Huang, Yu Zhou, Yao Yu, Ziqiang Wang, and Sidan Du. 2011. Piano ar: A markerless augmented reality based piano teaching system. In 2011 Third International Conference on Intelligent Human-Machine Systems and Cybernetics, Vol. 2. IEEE, 47–52.Google ScholarDigital Library
- Jakob Karolus, Annika Kilian, Thomas Kosch, Albrecht Schmidt, and Paweł W Wozniak. 2020. Hit the Thumb Jack! Using Electromyography to Augment the Piano Keyboard. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 429–440.Google ScholarDigital Library
- Mohammed K Khalil, Fred Paas, Tristan E Johnson, and Andrew F Payer. 2005. Design of interactive and dynamic anatomical visualizations: the implication of cognitive load theory. The Anatomical Record Part B: The New Anatomist: An Official Publication of the American Association of Anatomists 286, 1(2005), 15–20.Google Scholar
- Sunjun Kim, Byungjoo Lee, and Antti Oulasvirta. 2018. Impact activation improves rapid button pressing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–8.Google ScholarDigital Library
- Melina Klepsch, Florian Schmitz, and Tina Seufert. 2017. Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load. Frontiers in psychology 8 (2017), 1997.Google Scholar
- Sébastien Lallé and Cristina Conati. 2019. The role of user differences in customization: a case study in personalization for infovis-based content. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 329–339.Google ScholarDigital Library
- Kevin C. Lam, Carl Gutwin, Madison Klarkowski, and Andy Cockburn. 2021. The Effects of System Interpretation Errors on Learning New Input Mechanisms. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445366Google Scholar
- Edward W Large and Caroline Palmer. 2002. Perceiving temporal regularity in music. Cognitive science 26, 1 (2002), 1–37.Google Scholar
- Byungjoo Lee. 2016. Temporal Pointing. http://www.leebyungjoo.com/Temporal-Pointing-1Google Scholar
- Byungjoo Lee, Qiao Deng, Eve Hoggan, and Antti Oulasvirta. 2017. Boxer: a multimodal collision technique for virtual objects. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. 252–260.Google ScholarDigital Library
- Byungjoo Lee and Antti Oulasvirta. 2016. Modelling error rates in temporal pointing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1857–1868.Google ScholarDigital Library
- Yi-Chi Liao, Sunjun Kim, Byungjoo Lee, and Antti Oulasvirta. 2020. Button Simulation and Design via FDVV Models. arXiv preprint arXiv:2001.04352(2020).Google Scholar
- Edward A Lippman. 1984. Progressive temporality in music. Journal of Musicology 3, 2 (1984), 121–141.Google ScholarCross Ref
- Eunji Park and Byungjoo Lee. 2020. An Intermittent Click Planning Model. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Jeff Pressing. 1998. Error correction processes in temporal pattern production. Journal of Mathematical Psychology 42, 1 (1998), 63–101.Google ScholarDigital Library
- Remy MJP Rikers, Pascal WM Van Gerven, and Henk G Schmidt. 2004. Cognitive load theory as a tool for expertise development. Instructional Science 32, 1-2 (2004), 173–182.Google ScholarCross Ref
- Katja Rogers, Amrei Röhlig, Matthias Weing, Jan Gugenheimer, Bastian Könings, Melina Klepsch, Florian Schaub, Enrico Rukzio, Tina Seufert, and Michael Weber. 2014. Piano: Faster piano learning with interactive projection. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces. 149–158.Google ScholarDigital Library
- Marc Ericson C Santos, Angie Chen, Takafumi Taketomi, Goshiro Yamamoto, Jun Miyazaki, and Hirokazu Kato. 2013. Augmented reality learning experiences: Survey of prototype design and evaluation. IEEE Transactions on learning technologies 7, 1 (2013), 38–56.Google ScholarCross Ref
- Dirk Vorberg and Hans-Henning Schulze. 2002. Linear phase-correction in synchronization: Predictions, parameter estimation, and simulations. Journal of Mathematical Psychology 46, 1 (2002), 56–87.Google ScholarDigital Library
- Matthias Weing, Amrei Röhlig, Katja Rogers, Jan Gugenheimer, Florian Schaub, Bastian Könings, Enrico Rukzio, and Michael Weber. 2013. PIANO: enhancing instrument learning via interactive projected augmentation. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. 75–78.Google ScholarDigital Library
- Alan M Wing and AB Kristofferson. 1973. The timing of interresponse intervals. Perception & Psychophysics 13, 3 (1973), 455–460.Google ScholarCross Ref
- Alan M Wing and Alfred B Kristofferson. 1973. Response delays and the timing of discrete motor responses. Perception & Psychophysics 14, 1 (1973), 5–12.Google ScholarCross Ref
- Beste F Yuksel, Kurt B Oleson, Lane Harrison, Evan M Peck, Daniel Afergan, Remco Chang, and Robert JK Jacob. 2016. Learn piano with BACh: An adaptive learning interface that adjusts task difficulty based on brain state. In Proceedings of the 2016 chi conference on human factors in computing systems. 5372–5384.Google ScholarDigital Library
Recommendations
Piano Learning and Improvisation through Adaptive Visualisation and Digital Augmentation
ISS '22: Companion Proceedings of the 2022 Conference on Interactive Surfaces and SpacesThe task of learning the piano has been a centuries-old challenge for novices, experts and technologists. Several innovations have been introduced to support proper posture, movement, and motivation, while sight-reading and improvisation remain the ...
Adaptive Visualisations Using Spatiotemporal and Heuristic Models to Support Piano Learning
UMAP '21: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and PersonalizationLearning the piano is hard and many approaches including piano-roll visualisations have been explored in order to support novices and seasoned learners in this process. However, existing piano roll prototypes have not considered the spatiotemporal ...
Piano&Dancer: Interaction Between a Dancer and an Acoustic Instrument
MOCO '17: Proceedings of the 4th International Conference on Movement ComputingPiano&Dancer is an interactive piece for a dancer and an electromechanical acoustic piano. The piece presents the dancer and the piano as two performers on stage whose bodily movements are mutually interdependent. This interdependence reveals a close ...
Comments