Skip to main content
SpringerLink
Log in

A Stylus-Driven Intelligent Tutoring System for Music Education Instruction

  • Chapter
  • First Online:
Revolutionizing Education with Digital Ink

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Inspiring musicians and non-musician hobbyists alike can enjoy various benefits from learning music theory beyond its importance in performing music itself. Such reasons include developing their mathematical abilities, improving their reading comprehension, expanding their memory capabilities, better appreciating music that they listen to, and so on. However, current resources that are available for teaching music theory to students present their own inherent disadvantages. Specifically, traditional music theory classroom centers assume that students already have existing musical knowledge, existing self-study paper-based materials lack immediate feedback, while emerging educational apps either lack stylus-based interaction or intelligent feedback appropriate for novice students. In this paper, we introduce a stylus-driven intelligent tutoring system for music education instruction that aims to combine the benefits while addressing the limitations of existing instructional resources for teaching music theory. Our proposed system provides an accessible educational application with an intelligent sketch user interface that is designed for novice students interested in learning music theory through a series of interactive music composition lessons and quizzes. Following the completion of a student’s composed solution to a prompted music theory question, our system first leverages appropriate sketch and gesture recognition techniques to automatically understand the student’s input, and then generates feedback and assessment of the student’s input that emulates those from a music theory instructor. From our evaluations, we demonstrate that not only did our system automatically understood students’ composed solutions with reasonable accuracy, but also that novice students were able to successfully grasp introductory music theory concepts from a single session using our system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.musictheory.net/products/tenuto.

  2. 2.

    http://www.musictheory.net/products/lessons.

  3. 3.

    http://apple.co/1mYs0nk.

  4. 4.

    https://www.coursera.org/course/musictheory.

  5. 5.

    http://www.finalemusic.com/.

  6. 6.

    http://www.sibelius.com/.

  7. 7.

    http://www.neuratron.com/notateme.html.

  8. 8.

    http://bit.ly/19Mq6z5.

References

  1. Anthony L, Wobbrock J (2012) \({\$}\)N-protractor: a fast and accurate multistroke recognizer. In: Proceedings of graphics interface 2012, GI ’12. Canadian information processing society, Mississauga, Ontario, Canada, pp 117–120

    Google Scholar 

  2. Baba T, Kikukawa Y, Yoshiike T, Suzuki T, Shoji R, Kushiyama K, Aoki M (2012) Gocen: a handwritten notational interface for musical performance and learning music. In: ACM SIGGRAPH 2012 emerging technologies SIGGRAPH ’12. ACM, New York, USA, pp 9–9

    Google Scholar 

  3. Brooke E (2013) SoundBrush, an app that lets you draw music, turns the untrained into musicians [Online News Article]. Accessed 12 July 2013

    Google Scholar 

  4. Cheema S, LaViola J Jr (2012) PhysicsBook: a sketch-based interface for animating physics diagrams. In: Proceedings of the 2012 ACM international conference on intelligent user interfaces IUI ’12. ACM, New York, USA, pp 51–60

    Google Scholar 

  5. Colletti C (2014) A comparative review of five music fundamentals textbooks. J Music Theory Pedag 28(1)

    Google Scholar 

  6. Cummmings D, Vides F, Hammond T (2012) I don’t believe my eyes!: geometric sketch recognition for a computer art tutorial. In: Proceedings of the international symposium on sketch-based interfaces and modeling, SBIM’12. Eurographics Association, Geneva, Switzerland, pp 97–106

    Google Scholar 

  7. Dahmen K, Hammond T (2008) Distinguishing between sketched scribble look alikes. In: Proceedings of the 23rd national conference on artificial intelligence AAAI ’08. AAAI, Palo Alto, California, USA, pp 1790–1791

    Google Scholar 

  8. Deihl N (1971) Computer-assisted instruction and instrumental music: implications for teaching and research. J Res Music Educ 19(3):299–306

    Article  Google Scholar 

  9. Dixon D, Prasad M, Hammond T (2010) iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces. In: Proceedings of the SIGCHI conference on human factors in computing systems CHI ’10. ACM, New York, USA, pp 897–906

    Google Scholar 

  10. Fornés A, Dutta A, Gordo A, Llads J (2012) CVC-MUSCIMA: a ground truth of handwritten music score images for writer identification and staff removal. Int J Doc Anal Recognit 15(3):243–251

    Article  Google Scholar 

  11. Forsberg A, Dieterich M, Zeleznik R (1998) The music notepad. In: Proceedings of the 11th annual ACM symposium on user interface software and technology UIST ’98. ACM, New York, USA, pp 203–210

    Google Scholar 

  12. Kara L, Stahovich T (2005) An image-based, trainable symbol recognizer for hand-drawn sketches. Comput Gr 29(4):501–517

    Article  Google Scholar 

  13. LaViola J Jr (2007) Advances in mathematical sketching: moving toward the paradigm’s full potential. IEEE Comput Gr Appl 27(1):38–48

    Article  Google Scholar 

  14. Lee K, Phon-Amnuaisuk S, Ting CY (2010) Handwritten music notation recognition using HMM—a non-gestural approach. In: 2010 international conference on information retrieval & knowledge management CIKM ’10. ICMS, Australasia, Melbourne, Victoria, Australia, pp 255–259

    Google Scholar 

  15. Leoné M, van Dijk B, van Beijnum BJ (2008) Design and trial evaluation of the user interface for musicReader. In: Proceedings of the 12th international conference on entertainment and media in the ubiquitous era, MindTrek ’08. ACM, New York, USA, pp 95–99

    Google Scholar 

  16. Li Y (2010) Protractor: a fast and accurate gesture recognizer. In: Proceedings of the SIGCHI conference on human factors in computing systems CHI ’10. ACM, New York, USA, pp 2169–2172

    Google Scholar 

  17. Miller M (2005) The complete idiot’s guide to music theory, 1st edn. Alpha Books, New York

    Google Scholar 

  18. Nelson R, Christensen C (2014) Foundations of music, enhanced, 1st edn. Cengage Learning, Stamford

    Google Scholar 

  19. Paulson B, Hammond T (2008) Paleosketch: accurate primitive sketch recognition and beautification. In: Proceedings of the 13th international conference on intelligent user interfaces IUI ’08. ACM, New York, USA, pp 1–10

    Google Scholar 

  20. Ramirez R, Hazan A, Mariner J, Maestre E (2006) A sequential covering evolutionary algorithm for expressive music performance. In: Proceedings of the 18th conference on innovative applications of artificial intelligence IAAI ’06. AAAI, Palo Alto, California, USA, pp 1830–1835

    Google Scholar 

  21. Rebelo A, Tkaczuk J, Sousa R, Cardoso J (2011) Metric learning for music symbol recognition. In: 2011 10th international conference on machine learning and applications and workshops ICMLA ’11. IEEE, New York, USA, pp 106–111

    Google Scholar 

  22. Rebelo A, Fujinaga I, Paszkiewicz F, Marcal A, Guedes C, Cardoso J (2012) Optical music recognition: state-of-the-art and open issues. Int J Multimed Inf Retr 1(3):173–190

    Article  Google Scholar 

  23. Rogers M (2004) Teaching approaches in music theory: an overview of pedagogical philosophies, 2nd edn. Southern Illinois University Press, Carbondale

    Google Scholar 

  24. Taele P, Hammond T (2009) Hashigo: a next-generation sketch interactive system for japanese kanji. In: Proceedings of the twenty-first innovative applications artificial intelligence conference IAAI ’09. AAAI, Palo Alto, California, USA, pp 153–158

    Google Scholar 

  25. Taele P, Peschel J, Hammond T (2009) A sketch interactive approach to computer-assisted biology instruction. In: 2009 intelligent user interfaces workshop on sketch recognition IUISR’09

    Google Scholar 

  26. Taele P, Barreto L, Hammond T (2015) Maestoso: an intelligent educational sketching tool for learning music theory. In: Proceedings of the twenty-seventh innovative applications of artificial intelligence conference IAAI ’15. AAAI, Palo Alto, California, USA, pp 3999–4005

    Google Scholar 

  27. Tan S, Hu Z, Koh A, Felicia A, Zhao S (2010) Musicflow: an interactive music composition system. In: Proceedings of SPIE visual communications and image processing 2010, SPIE

    Google Scholar 

  28. The telegraph: The telegraph: Music helps children learn maths (2012) [Online Artcile]. 22 March 2012. Accessed 30 June 2015

    Google Scholar 

  29. Topolu O (2014) Critical thinking and music education. Procedia—Soc Behav Sci 116:2252–2256

    Article  Google Scholar 

  30. van Sommers P (2009) Drawing and cognition: descriptive and experimental studies of graphic production processes, 1st edn. Cambridge University Press, Cambridge

    Google Scholar 

  31. Valentine SMF, Hammond T, Smith A (2011) A shape comparison technique for use in sketch-based tutoring systems. In: Proceedings of the 2011 intelligent user interfaces workshop on sketch recognition IUI SR ’11

    Google Scholar 

  32. Valentine S, Vides F, Lucchese G, Turner D, Kim Hh, Li W, Linsey J, Hammond T (2012) Mechanix: a sketch-based tutoring system for statics courses. In: Proceedings of the twenty-fourth innovative applications of artificial intelligence conference IAAI ’12. AAAI, Palo Alto, California, USA, pp 2253–2260

    Google Scholar 

  33. Valentine S, Lara-Garduno R, Linsey J, Hammond T (2015) Mechanix: a sketch based tutoring system that automatically corrects hand-sketched statics homework. In: Hammond T, Valentine S, Adler A (eds) The impact of pen and touch technology on education. Springer International Publishing, Heidelberg

    Google Scholar 

  34. Vatavu RD, Anthony L, Wobbrock J (2012) Gestures as point clouds: a \({\$}\)p recognizer for user interface prototypes. In: Proceedings of the ACM international conference on multimodal interfaces ICMI ’12. ACM, New York, USA, pp 273–280

    Google Scholar 

  35. Warschauer M (1997) Computer-mediated collaborative learning: theory and practice. Mod Lang J Spec Issue: Interact Collab Coop: Learn Lang Prep Lang Teach 81(4):470–481

    Google Scholar 

  36. Wobbrock J, Wilson A, Li Y (2007) Gestures without libraries, toolkits or training: a \({\$}\)1 recognizer for user interface prototypes. In: Proceedings of the 20th annual ACM symposium on user interface software and technology UIST ’07. ACM, New York, USA, pp 159–168

    Google Scholar 

  37. Wolin A, Eoff B, Hammond T (2008) Shortstraw: a simple and effective corner finder for polylines. In: Proceedings of the fifth eurographics conference on sketch-based interfaces and modeling SBIM ’08. Eurographics Association, Geneva, Switzerland, pp 33–40

    Google Scholar 

  38. Wu B, Zhong E, Horner A, Yang Q (2014) Music emotion recognition by multi-label multi-layer multi-instance multi-view learning. In: Proceedings of the ACM international conference on multimedia MM ’14. ACM, New York, USA, pp 117–126

    Google Scholar 

  39. Yamamoto Y, Uchiyama H, Kakehi Y (2011) onNote: playing printed music scores as a musical instrument. In: Proceedings of the 24th annual ACM symposium on user interface software and technology UIST ’11. ACM, New York, USA, pp 413–422

    Google Scholar 

  40. Yeary M (2014) Computer-assisted instruction and remedial music fundamentals. J Music Theory Pedag 28(1)

    Google Scholar 

  41. Zbikowski L, Long C (1994) Cooperative learning in the music theory classroom. J Music Theory Pedag 8(1):135–157

    Google Scholar 

Download references

Acknowledgments

The authors thank members of the Sketch Recognition Lab and the WIPTTE community for their support and feedback throughout this project. L. Barreto is supported by the CRA-W Distributed Research Experience for Undergraduates Grant. This work is funded in part by the Microsoft Surface Hub Research Grant: Sketch and Gesture Recognition for Collaborative and Design Interfaces on the Surface Hub. We would also like to thank Texas A&M University and Vassar College for their support of this project.

Author information

Authors and Affiliations

  1. Sketch Recognition Lab, Computer Science & Engineering Department, Texas A&M University, College Station, Texas, USA

    Laura Barreto, Paul Taele & Tracy Hammond

  2. Computer Science Department, Vassar College, Poughkeepsie, New York, USA

    Laura Barreto

Authors
  1. Laura Barreto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Taele
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tracy Hammond
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Laura Barreto or Paul Taele .

Editor information

Editors and Affiliations

  1. Texas A&M University, College Station, Texas, USA

    Tracy Hammond

  2. Dept of Computer Science & Engineering, Texas A&M University, College Station, Texas, USA

    Stephanie Valentine

  3. Raytheon BBN Technologies, Cambridge, Massachusetts, USA

    Aaron Adler

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Barreto, L., Taele, P., Hammond, T. (2016). A Stylus-Driven Intelligent Tutoring System for Music Education Instruction. In: Hammond, T., Valentine, S., Adler, A. (eds) Revolutionizing Education with Digital Ink. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-31193-7_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-31193-7_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-31191-3

  • Online ISBN: 978-3-319-31193-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation