Abstract
Inspiring musicians and non-musician hobbyists alike can enjoy various benefits from learning music theory beyond its importance in performing music itself. Such reasons include developing their mathematical abilities, improving their reading comprehension, expanding their memory capabilities, better appreciating music that they listen to, and so on. However, current resources that are available for teaching music theory to students present their own inherent disadvantages. Specifically, traditional music theory classroom centers assume that students already have existing musical knowledge, existing self-study paper-based materials lack immediate feedback, while emerging educational apps either lack stylus-based interaction or intelligent feedback appropriate for novice students. In this paper, we introduce a stylus-driven intelligent tutoring system for music education instruction that aims to combine the benefits while addressing the limitations of existing instructional resources for teaching music theory. Our proposed system provides an accessible educational application with an intelligent sketch user interface that is designed for novice students interested in learning music theory through a series of interactive music composition lessons and quizzes. Following the completion of a student’s composed solution to a prompted music theory question, our system first leverages appropriate sketch and gesture recognition techniques to automatically understand the student’s input, and then generates feedback and assessment of the student’s input that emulates those from a music theory instructor. From our evaluations, we demonstrate that not only did our system automatically understood students’ composed solutions with reasonable accuracy, but also that novice students were able to successfully grasp introductory music theory concepts from a single session using our system.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anthony L, Wobbrock J (2012) \({\$}\)N-protractor: a fast and accurate multistroke recognizer. In: Proceedings of graphics interface 2012, GI ’12. Canadian information processing society, Mississauga, Ontario, Canada, pp 117–120
Baba T, Kikukawa Y, Yoshiike T, Suzuki T, Shoji R, Kushiyama K, Aoki M (2012) Gocen: a handwritten notational interface for musical performance and learning music. In: ACM SIGGRAPH 2012 emerging technologies SIGGRAPH ’12. ACM, New York, USA, pp 9–9
Brooke E (2013) SoundBrush, an app that lets you draw music, turns the untrained into musicians [Online News Article]. Accessed 12 July 2013
Cheema S, LaViola J Jr (2012) PhysicsBook: a sketch-based interface for animating physics diagrams. In: Proceedings of the 2012 ACM international conference on intelligent user interfaces IUI ’12. ACM, New York, USA, pp 51–60
Colletti C (2014) A comparative review of five music fundamentals textbooks. J Music Theory Pedag 28(1)
Cummmings D, Vides F, Hammond T (2012) I don’t believe my eyes!: geometric sketch recognition for a computer art tutorial. In: Proceedings of the international symposium on sketch-based interfaces and modeling, SBIM’12. Eurographics Association, Geneva, Switzerland, pp 97–106
Dahmen K, Hammond T (2008) Distinguishing between sketched scribble look alikes. In: Proceedings of the 23rd national conference on artificial intelligence AAAI ’08. AAAI, Palo Alto, California, USA, pp 1790–1791
Deihl N (1971) Computer-assisted instruction and instrumental music: implications for teaching and research. J Res Music Educ 19(3):299–306
Dixon D, Prasad M, Hammond T (2010) iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces. In: Proceedings of the SIGCHI conference on human factors in computing systems CHI ’10. ACM, New York, USA, pp 897–906
Fornés A, Dutta A, Gordo A, Llads J (2012) CVC-MUSCIMA: a ground truth of handwritten music score images for writer identification and staff removal. Int J Doc Anal Recognit 15(3):243–251
Forsberg A, Dieterich M, Zeleznik R (1998) The music notepad. In: Proceedings of the 11th annual ACM symposium on user interface software and technology UIST ’98. ACM, New York, USA, pp 203–210
Kara L, Stahovich T (2005) An image-based, trainable symbol recognizer for hand-drawn sketches. Comput Gr 29(4):501–517
LaViola J Jr (2007) Advances in mathematical sketching: moving toward the paradigm’s full potential. IEEE Comput Gr Appl 27(1):38–48
Lee K, Phon-Amnuaisuk S, Ting CY (2010) Handwritten music notation recognition using HMM—a non-gestural approach. In: 2010 international conference on information retrieval & knowledge management CIKM ’10. ICMS, Australasia, Melbourne, Victoria, Australia, pp 255–259
Leoné M, van Dijk B, van Beijnum BJ (2008) Design and trial evaluation of the user interface for musicReader. In: Proceedings of the 12th international conference on entertainment and media in the ubiquitous era, MindTrek ’08. ACM, New York, USA, pp 95–99
Li Y (2010) Protractor: a fast and accurate gesture recognizer. In: Proceedings of the SIGCHI conference on human factors in computing systems CHI ’10. ACM, New York, USA, pp 2169–2172
Miller M (2005) The complete idiot’s guide to music theory, 1st edn. Alpha Books, New York
Nelson R, Christensen C (2014) Foundations of music, enhanced, 1st edn. Cengage Learning, Stamford
Paulson B, Hammond T (2008) Paleosketch: accurate primitive sketch recognition and beautification. In: Proceedings of the 13th international conference on intelligent user interfaces IUI ’08. ACM, New York, USA, pp 1–10
Ramirez R, Hazan A, Mariner J, Maestre E (2006) A sequential covering evolutionary algorithm for expressive music performance. In: Proceedings of the 18th conference on innovative applications of artificial intelligence IAAI ’06. AAAI, Palo Alto, California, USA, pp 1830–1835
Rebelo A, Tkaczuk J, Sousa R, Cardoso J (2011) Metric learning for music symbol recognition. In: 2011 10th international conference on machine learning and applications and workshops ICMLA ’11. IEEE, New York, USA, pp 106–111
Rebelo A, Fujinaga I, Paszkiewicz F, Marcal A, Guedes C, Cardoso J (2012) Optical music recognition: state-of-the-art and open issues. Int J Multimed Inf Retr 1(3):173–190
Rogers M (2004) Teaching approaches in music theory: an overview of pedagogical philosophies, 2nd edn. Southern Illinois University Press, Carbondale
Taele P, Hammond T (2009) Hashigo: a next-generation sketch interactive system for japanese kanji. In: Proceedings of the twenty-first innovative applications artificial intelligence conference IAAI ’09. AAAI, Palo Alto, California, USA, pp 153–158
Taele P, Peschel J, Hammond T (2009) A sketch interactive approach to computer-assisted biology instruction. In: 2009 intelligent user interfaces workshop on sketch recognition IUISR’09
Taele P, Barreto L, Hammond T (2015) Maestoso: an intelligent educational sketching tool for learning music theory. In: Proceedings of the twenty-seventh innovative applications of artificial intelligence conference IAAI ’15. AAAI, Palo Alto, California, USA, pp 3999–4005
Tan S, Hu Z, Koh A, Felicia A, Zhao S (2010) Musicflow: an interactive music composition system. In: Proceedings of SPIE visual communications and image processing 2010, SPIE
The telegraph: The telegraph: Music helps children learn maths (2012) [Online Artcile]. 22 March 2012. Accessed 30 June 2015
Topolu O (2014) Critical thinking and music education. Procedia—Soc Behav Sci 116:2252–2256
van Sommers P (2009) Drawing and cognition: descriptive and experimental studies of graphic production processes, 1st edn. Cambridge University Press, Cambridge
Valentine SMF, Hammond T, Smith A (2011) A shape comparison technique for use in sketch-based tutoring systems. In: Proceedings of the 2011 intelligent user interfaces workshop on sketch recognition IUI SR ’11
Valentine S, Vides F, Lucchese G, Turner D, Kim Hh, Li W, Linsey J, Hammond T (2012) Mechanix: a sketch-based tutoring system for statics courses. In: Proceedings of the twenty-fourth innovative applications of artificial intelligence conference IAAI ’12. AAAI, Palo Alto, California, USA, pp 2253–2260
Valentine S, Lara-Garduno R, Linsey J, Hammond T (2015) Mechanix: a sketch based tutoring system that automatically corrects hand-sketched statics homework. In: Hammond T, Valentine S, Adler A (eds) The impact of pen and touch technology on education. Springer International Publishing, Heidelberg
Vatavu RD, Anthony L, Wobbrock J (2012) Gestures as point clouds: a \({\$}\)p recognizer for user interface prototypes. In: Proceedings of the ACM international conference on multimodal interfaces ICMI ’12. ACM, New York, USA, pp 273–280
Warschauer M (1997) Computer-mediated collaborative learning: theory and practice. Mod Lang J Spec Issue: Interact Collab Coop: Learn Lang Prep Lang Teach 81(4):470–481
Wobbrock J, Wilson A, Li Y (2007) Gestures without libraries, toolkits or training: a \({\$}\)1 recognizer for user interface prototypes. In: Proceedings of the 20th annual ACM symposium on user interface software and technology UIST ’07. ACM, New York, USA, pp 159–168
Wolin A, Eoff B, Hammond T (2008) Shortstraw: a simple and effective corner finder for polylines. In: Proceedings of the fifth eurographics conference on sketch-based interfaces and modeling SBIM ’08. Eurographics Association, Geneva, Switzerland, pp 33–40
Wu B, Zhong E, Horner A, Yang Q (2014) Music emotion recognition by multi-label multi-layer multi-instance multi-view learning. In: Proceedings of the ACM international conference on multimedia MM ’14. ACM, New York, USA, pp 117–126
Yamamoto Y, Uchiyama H, Kakehi Y (2011) onNote: playing printed music scores as a musical instrument. In: Proceedings of the 24th annual ACM symposium on user interface software and technology UIST ’11. ACM, New York, USA, pp 413–422
Yeary M (2014) Computer-assisted instruction and remedial music fundamentals. J Music Theory Pedag 28(1)
Zbikowski L, Long C (1994) Cooperative learning in the music theory classroom. J Music Theory Pedag 8(1):135–157
Acknowledgments
The authors thank members of the Sketch Recognition Lab and the WIPTTE community for their support and feedback throughout this project. L. Barreto is supported by the CRA-W Distributed Research Experience for Undergraduates Grant. This work is funded in part by the Microsoft Surface Hub Research Grant: Sketch and Gesture Recognition for Collaborative and Design Interfaces on the Surface Hub. We would also like to thank Texas A&M University and Vassar College for their support of this project.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Barreto, L., Taele, P., Hammond, T. (2016). A Stylus-Driven Intelligent Tutoring System for Music Education Instruction. In: Hammond, T., Valentine, S., Adler, A. (eds) Revolutionizing Education with Digital Ink. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-31193-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-31193-7_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-31191-3
Online ISBN: 978-3-319-31193-7
eBook Packages: Computer ScienceComputer Science (R0)