Intelligent Music Interfaces: When Interactive Assistance and Adaptive Augmentation Meet Musical Instruments
Authors: 
Thomas Kosch, 
Andreas Weiß, 
Jordan Aiko Deja, Snehesh Shrestha, Matthias Hoppe, Andrii Matviienko, Karola MarkyAuthors Info & Claims
Thomas Kosch, Andreas Weiß, Jordan Aiko Deja, Snehesh Shrestha, Matthias Hoppe, Andrii Matviienko, Karola MarkyAuthors Info & ClaimsPages 327 - 330
Abstract
The interactive augmentation of musical instruments to foster self-expression and learning has a rich history. Over the past decades, incorporating interactive technologies into musical instruments has emerged as a research field requiring strong collaboration between disciplines. The workshop “Intelligent Music Interfaces” covers a wide range of musical research subjects and directions, including (a) current challenges in musical learning, (b) prototyping for improvements, (c) new means of musical expression, and (d) evaluation of the solutions.
References
[1]
Jordan Aiko Deja, Alexczar Dela Torre, Hans Joshua Lee, Jose Florencio Ciriaco IV, and Carlo Miguel Eroles. 2020. Vitune: A visualizer tool to allow the deaf and hard of hearing to see music with their eyes. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–8.
[2]
Jordan Aiko Deja, Bettina Eska, Snehesh Shrestha, Matthias Hoppe, Jakob Karolus, Thomas Kosch, Andrii Matviienko, Andreas Weiß, and Karola Marky. 2023. Intelligent Music Interfaces: When Interactive Assistance and Augmentation Meet Musical Instruments. In Proceedings of the Augmented Humans International Conference 2023. 379–383.
[3]
Jordan Aiko Deja, Sven Mayer, Klen Čopič Pucihar, and Matjaž Kljun. 2022. A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?Proceedings of the ACM on Human-Computer Interaction 6, ISS (2022), 226–253.
[4]
Jordan Aiko Deja, Sandi Štor, Ilonka Pucihar, Klen Čopič Pucihar, and Matjaž Kljun. 2024. Teach Me How to ImproVISe: Co-Designing an Augmented Piano Training System for Improvisation, In 8th Human-Computer Interaction Slovenia conference 2023. arXiv preprint arXiv:2402.02999.
[5]
Christiane Glatz, Jonas Ditz, Thomas Kosch, Albrecht Schmidt, Marie Lahmer, and Lewis L. Chuang. 2017. Reading the Mobile Brain: From Laboratory to Real-World Electroencephalography. In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia (Stuttgart, Germany) (MUM ’17). Association for Computing Machinery, New York, NY, USA, 573–579. https://doi.org/10.1145/3152832.3156560
[6]
Jiawen Han, George Chernyshov, Moe Sugawa, Dingding Zheng, Danny Hynds, Taichi Furukawa, Marcelo Padovani, Kouta Minamizawa, Karola Marky, Jamie A Ward, and Kai Kunze. 2022. Linking Audience Physiology to Choreography. ACM Trans. Comput.-Hum. Interact. 29 (aug 2022), 1–32. https://doi.org/10.1145/3557887 Just Accepted.
[7]
Jakob Karolus, Annika Kilian, Thomas Kosch, Albrecht Schmidt, and Paweł W. Wozniak. 2020. Hit the Thumb Jack! Using Electromyography to Augment the Piano Keyboard. In Proceedings of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS ’20). Association for Computing Machinery, New York, NY, USA, 429–440. https://doi.org/10.1145/3357236.3395500
[8]
Jakob Karolus, Hendrik Schuff, Thomas Kosch, Paweł W. Wozniak, and Albrecht Schmidt. 2018. EMGuitar: Assisting Guitar Playing with Electromyography. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 651–655. https://doi.org/10.1145/3196709.3196803
[9]
Joseph R. Keebler, Travis J. Wiltshire, Dustin C. Smith, and Stephen M. Fiore. 2013. Picking Up STEAM: Educational Implications for Teaching With an Augmented Reality Guitar Learning System. In Proceedings of the International Conference on Virtual, Augmented and Mixed Reality(VAMR). Springer, Cham, Switzerland, 170–178. https://doi.org/10.1007/978-3-642-39420-1_19
[10]
Chutisant Kerdvibulvech and Hideo Saito. 2007. Real-Time Guitar Chord Estimation by Stereo Cameras for Supporting Guitarists. In Proceedings of the 10th International Workshop on Advanced Image Technology(IWAIT). The Institute of Electronics, Information and Communication Engineers, Bangkok, Thailand, 256–261.
[11]
Annika Kilian, Jakob Karolus, Thomas Kosch, Albrecht Schmidt, and Paweł W. Paweł. 2021. EMPiano: Electromyographic Pitch Control on the Piano Keyboard. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 196, 4 pages. https://doi.org/10.1145/3411763.3451556
[12]
Markus Löchtefeld, Sven Gehring, Ralf Jung, and Antonio Krüger. 2011. GuitAR: Supporting Guitar Learning through Mobile Projection. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI EA ’11). Association for Computing Machinery, New York, NY, USA, 1447–1452. https://doi.org/10.1145/1979742.1979789
[13]
Andreas Löcken, Mark Colley, Andrii Matviienko, Kai Holländer, Debargha Dey, Azra Habibovic, Andrew L Kun, Susanne Boll, and Andreas Riener. 2021. WeCARe: Workshop on Inclusive Communication between Automated Vehicles and Vulnerable Road Users. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI ’20). Association for Computing Machinery, New York, NY, USA, Article 43, 5 pages. https://doi.org/10.1145/3406324.3424587
[14]
Andreas Löcken, Andrii Matviienko, Mark Colley, Debargha Dey, Azra Habibovic, Yee Mun Lee, and Andreas Riener. 2022. Accessible Automated Automotive Workshop Series (A3WS): International Perspective on Inclusive External Human-Machine Interfaces. In Adjunct Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Seoul, Republic of Korea) (AutomotiveUI ’22). Association for Computing Machinery, New York, NY, USA, 192–195. https://doi.org/10.1145/3544999.3551347
[15]
Karola Marky, Julian Fischer, Max Mühlhäuser, and Andrii Matviienko. 2021. Investigating Page Turning Methods for Sheet Music during Piano Play. In Adjunct Publication of the 23rd International Conference on Mobile Human-Computer Interaction (Toulouse & Virtual, France) (MobileHCI ’21). Association for Computing Machinery, New York, NY, USA, Article 18, 6 pages. https://doi.org/10.1145/3447527.3474863
[16]
Karola Marky, Annika Kilian, Andreas Weiß, Jakob Karolus, Matthias Hoppe, Pawel W. Wozniak, Max Mühlhäuser, and Thomas Kosch. 2022. Intelligent Music Interfaces: When Interactive Assistance and Augmentation Meet Musical Instruments. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 84, 4 pages. https://doi.org/10.1145/3491101.3503743
[17]
Karola Marky, Andreas Weiß, Andrii Matviienko, Florian Brandherm, Sebastian Wolf, Martin Schmitz, Florian Krell, Florian Müller, Max Mühlhäuser, and Thomas Kosch. 2021. Let’s Frets! Assisting Guitar Students During Practice via Capacitive Sensing. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 746, 12 pages. https://doi.org/10.1145/3411764.3445595
[18]
Karola Marky, Andreas Weiß, Florian Müller, Martin Schmitz, Max Mühlhäuser, and Thomas Kosch. 2021. Let’s Frets! Mastering Guitar Playing with Capacitive Sensing and Visual Guidance. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 169, 4 pages. https://doi.org/10.1145/3411763.3451536
[19]
Andrii Matviienko, Wilko Heuten, Alan Dix, and Susanne CJ Boll. 2021. Interactive Technology for Cycling — Ideate, Make – Remote, Together. In Adjunct Publication of the 23rd International Conference on Mobile Human-Computer Interaction (Toulouse & Virtual, France) (MobileHCI ’21). Association for Computing Machinery, New York, NY, USA, Article 29, 4 pages. https://doi.org/10.1145/3447527.3474870
[20]
Andrew P. McPherson, Adrian Gierakowski, and Adam M. Stark. 2013. The Space between the Notes: Adding Expressive Pitch Control to the Piano Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2195–2204. https://doi.org/10.1145/2470654.2481302
[21]
Florian Müller, Dirk Schnelle-Walka, Tobias Grosse-Puppendahl, Sebastian Günther, Markus Funk, Kris Luyten, Oliver Brdiczka, Niloofar Dezfuli, and Max Mühlhäuser. 2018. SmartObjects: Sixth Workshop on Interacting with Smart Objects. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI EA ’18). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3170427.3170606
[22]
Katja Rogers, Amrei Röhlig, Matthias Weing, Jan Gugenheimer, Bastian Könings, Melina Klepsch, Florian Schaub, Enrico Rukzio, Tina Seufert, and Michael Weber. 2014. P.I.A.N.O.: Faster Piano Learning with Interactive Projection. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (Dresden, Germany) (ITS ’14). Association for Computing Machinery, New York, NY, USA, 149–158. https://doi.org/10.1145/2669485.2669514
[23]
Hatice Sahin, Heiko Mueller, Shadan Sadeghian, Debargha Dey, Andreas Löcken, Andrii Matviienko, Mark Colley, Azra Habibovic, and Philipp Wintersberger. 2021. Workshop on Prosocial Behavior in Future Mixed Traffic. In 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Leeds, United Kingdom) (AutomotiveUI ’21 Adjunct). Association for Computing Machinery, New York, NY, USA, 167–170. https://doi.org/10.1145/3473682.3477438
[24]
Gian-Luca Savino, Tamara von Sawitzky, Andrii Matviienko, Miriam Sturdee, Paweł W. Woźniak, Markus Löchtefeld, Andrew L. Kun, Andreas Riener, and Jonna Häkkilä. 2021. Cycling@CHI: Towards a Research Agenda for HCI in the Bike Lane. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21). Association for Computing Machinery, New York, NY, USA, Article 107, 5 pages. https://doi.org/10.1145/3411763.3441316
[25]
Martin Schmitz, Sebastian Günther, Karola Marky, Florian Müller, Andrii Matviienko, Alexandra Voit, Roberts Marky, Max Mühlhäuser, and Thomas Kosch. 2022. Rethinking Smart Objects: The International Workshop on Interacting with Smart Objects in Interactive Spaces. In Companion Proceedings of the 2022 Conference on Interactive Surfaces and Spaces (Wellington, New Zealand) (ISS ’22). Association for Computing Machinery, New York, NY, USA, 64–67. https://doi.org/10.1145/3532104.3571470
[26]
Martin Schmitz, Dominik Schön, Henning Klagemann, and Thomas Kosch. 2024. 3DA: Assessing 3D-Printed Electrodes for Measuring Electrodermal Activity. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’24). ACM, New York, NY, USA. https://doi.org/10.1145/3613905.3650938
[27]
Snehesh Shrestha, Cornelia Fermüller, Tianyu Huang, Pyone Thant Win, Adam Zukerman, Chethan M Parameshwara, and Yiannis Aloimonos. 2022. AIMusicGuru: Music Assisted Human Pose Correction.
[28]
Snehesh Shrestha, William Sentosatio, Huiashu Peng, Cornelia Fermuller, and Yiannis Aloimonos. 2023. FEVA: Fast Event Video Annotation Tool.
[29]
John Edel Tamani, Jan Christian Blaise Cruz, Joshua Raphaelle Cruzada, Jolene Valenzuela, Kevin Gray Chan, and Jordan Aiko Deja. 2018. Building guitar strum models for an interactive air guitar prototype. In Proceedings of the 4th International Conference on Human-Computer Interaction and User Experience in Indonesia, CHIuXiD’18. 18–22.
[30]
Tamara von Sawitzky, Philipp Wintersberger, Andrii Matviienko, Andreas Löcken, Andreas Riener, and Florian Michahelles. 2021. Workshop on Intelligent Cyclist Support Systems and Applications. In Mensch und Computer 2021 - Workshopband, Carolin Wienrich, Philipp Wintersberger, and Benjamin Weyers (Eds.). Gesellschaft für Informatik e.V., Bonn, 1–3. https://doi.org/10.18420/muc2021-mci-ws12-122
Index Terms
- Intelligent Music Interfaces: When Interactive Assistance and Adaptive Augmentation Meet Musical Instruments
Recommendations
Intelligent Music Interfaces: When Interactive Assistance and Augmentation Meet Musical Instruments
AHs '23: Proceedings of the Augmented Humans International Conference 2023The interactive augmentation of musical instruments to foster self-expressiveness and learning has a rich history. Over the past decades, the incorporation of interactive technologies into musical instruments emerged into a new research field requiring ...
Intelligent Music Interfaces: When Interactive Assistance and Augmentation Meet Musical Instruments
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing SystemsThe interactive augmentation of musical instruments to foster self-expressiveness and learning has a rich history. Over the past decades, the incorporation of interactive technologies into musical instruments emerged into a new research field requiring ...
Post-DMI musical Instruments
AM '16: Proceedings of the Audio Mostly 2016What are the particularities of developing contemporary instruments? This text points out how the authors observed both a focal shift and persisting elements in the design of contemporary digital musical instruments (DMI's). Based on observations around ...
Comments
Information & Contributors
Information
Published In
April 2024
355 pages
ISBN:9798400709807
DOI:10.1145/3652920
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 May 2024
Check for updates
Author Tags
Qualifiers
- Extended-abstract
- Research
- Refereed limited
Conference
AHs 2024
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 44Total Downloads
- Downloads (Last 12 months)44
- Downloads (Last 6 weeks)7
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format