Skip to main content
SpringerLink
Log in

Mobile Devices as Musical Instruments - State of the Art and Future Prospects

  • Conference paper
  • First Online:
Music Technology with Swing (CMMR 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11265))

Included in the following conference series:

Abstract

Mobile music making has established itself as a maturing field of inquiry, scholarship, and artistic practice. While mobile devices were used before the advent of the iPhone, its introduction no doubt drastically accelerated the field. We take a look at the current state of the art of mobile devices as musical instruments, and explore future prospects for research in the field.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://reactable.com/mobile/.

  2. 2.

    https://www.ableton.com/en/link/.

References

  1. Agiwal, M., Roy, A., Saxena, N.: Next generation 5g wireless networks: a comprehensive survey. IEEE Commun. Surveys Tuts. 18(3), 1617–1655 (2016)

    Article  Google Scholar 

  2. Allison, J., Dell, C.: AuRal: a mobile interactive system for geo-locative audio synthesis. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)

    Google Scholar 

  3. Axford, E.C.: Music Apps for Musicians and Music Teachers. Rowman & Littlefield (2015)

    Google Scholar 

  4. Baldan, S., Götzen, A.D., Serafin, S.: Sonic tennis: a rhythmic interaction game for mobile devices. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  5. Barraclough, T., Carnegie, D., Kapur, A.: Musical instrument design process for mobile technology. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2015)

    Google Scholar 

  6. Brinkmann, P., et al.: Embedding pure data with libpd. In: Proceedings of the Pure Data Convention (2011)

    Google Scholar 

  7. Bryan, N.J., Herrera, J., Oh, J., Wang, G.: MoMu: a mobile music toolkit. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)

    Google Scholar 

  8. Bryan, N.J., Wang, G.: Two turntables and a mobile phone. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)

    Google Scholar 

  9. de Carvalho Junior, A.D., Lee, S.W., Essl, G.: Understanding cloud support for the audience participation concert performance of crowd in c[loud]. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2016)

    Google Scholar 

  10. Clément, A.R., Ribeiro, F., Penha, R.: Bridging the gap between performers and the audience using networked smartphones: the a.bel system. In: Proceedings of the International Conference on Live Interfaces (2016)

    Google Scholar 

  11. d’Alessandro, N., et al.: A digital mobile choir: joining two interfaces towards composing and performing collaborative mobile music. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)

    Google Scholar 

  12. Derbinsky, N., Essl, G.: Cognitive architecture in mobile music interactions. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)

    Google Scholar 

  13. Derbinsky, N., Essl, G.: Exploring reinforcement learning for mobile percussive collaboration. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)

    Google Scholar 

  14. Dubus, G., Hansen, K.F., Bresin, R.: An overview of sound and music applications for android available on the market. In: Proceedings of the 9th Sound and Music Computing Conference (2012)

    Google Scholar 

  15. Erkut, C., Jylhä, A., Serafin, S.: (and sound) of simpe: Showcasing outcomes of a mobile audio programming seminar. In: Workshop on Speech and Sound in Mobile and Pervasive Environments (2013)

    Google Scholar 

  16. Essl, G.: UrMus - an environment for mobile instrument design and performance. In: Proceedings of the International Computer Music Conference (2010)

    Google Scholar 

  17. Essl, G.: UrSound - live patching of audio and multimedia using a multi-rate normed single-stream data-flow engine. In: Proceedings of the International Computer Music Conference (2010)

    Google Scholar 

  18. Essl, G.: Automated ad hoc networking for mobile and hybrid music performance. In: Proceedings of the International Computer Music Conference (2011)

    Google Scholar 

  19. Essl, G., Rohs, M., Kratz, S.: Use the force (or something) - pressure and pressure-like input for mobile music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)

    Google Scholar 

  20. Essl, G., Rohs, M.: Interactivity for Mobile Music Making. Organ. Sound 14(2) (2009)

    Article  Google Scholar 

  21. Essl, G., Müller, A.: Designing mobile musical instruments and environments with urMus. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)

    Google Scholar 

  22. Essl, G., Rohs, M., Kratz, S.: Use the force (or something) - pressure and pressure - like input for mobile music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)

    Google Scholar 

  23. Fabiani, M., Dubus, G., Bresin, R.: MoodifierLive: interactive and collaborative expressive music performance on mobile devices. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)

    Google Scholar 

  24. Gaye, L., Holmquist, L.E., Behrendt, F., Tanaka, A.: Mobile music technology: report on an emerging community. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2006)

    Google Scholar 

  25. Gonzalez-Inostroza, M., Sylleros, A., Cádiz, R.F.: Understanding interaction in musical multi-touch surfaces. In: Proceedings of the International Computer Music Conference (2017)

    Google Scholar 

  26. Halpern, M., Zhu, Y., Reddi, V.J.: Mobile CPU’s rise to power: quantifying the impact of generational mobile CPU design trends on performance, energy, and user satisfaction. In: IEEE Symposium on High Performance Computer Architecture (2016)

    Google Scholar 

  27. Hamilton, R., Smith, J., Wang, G.: Social composition: musical data systems for expressive mobile music. Leonardo Music J. 21, 57–64 (2011)

    Article  Google Scholar 

  28. Hindle, A.: SWARMED: captive portals, mobile devices, and audience participation in multi-user music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  29. Hindle, A.: CloudOrch: a portable soundcard in the cloud. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)

    Google Scholar 

  30. Hödl, O., Kayali, F., Fitzpatrick, G.: Designing interactive audience participation using smart phones in a musical performance. In: Proceedings of the International Computer Music Conference (2012)

    Google Scholar 

  31. John, D.: Updating the classifications of mobile music projects. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  32. Keefe, P.O., Essl, G.: The visual in mobile music performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)

    Google Scholar 

  33. Kim, B., Yeo, W.S.: Interactive mobile music performance with digital compass. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)

    Google Scholar 

  34. Kim, H.S., Herrera, J., Wang, G.: Ping-pong: Musically discovering locations. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)

    Google Scholar 

  35. Kimmel, J.: Introduction to mobile and wearable displays. In: Chen, J., Cranton, W., Fihn, M. (eds.) Handbook of Visual Display Technology, pp. 1–12. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-642-35947-7_120-2

    Chapter  Google Scholar 

  36. Knapp, B., Bortz, B.: MobileMuse: integral music control goes mobile. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)

    Google Scholar 

  37. Kruge, N., Wang, G.: MadPad: a crowdsourcing system for audiovisual sampling. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)

    Google Scholar 

  38. Lazzarini, V., Yi, S., Timoney, J., Keller, D., Pimenta, M.S.: The mobile Csound platform. In: Proceedings of the International Computer Music Conference (2012)

    Google Scholar 

  39. Lee, S.W., de Carvalho Jr, A.D., Essl, G.: Crowd in c[loud]: audience participation music with online dating metaphor using cloud service. In: Proceedings of the Web Audio Conference (2016)

    Google Scholar 

  40. Lee, S.W., Essl, G.: Live coding the mobile music instrument. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  41. Lee, S.W., Essl, G.: Communication, control, and state sharing in collaborative live coding. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)

    Google Scholar 

  42. Lee, S.W., Essl, G., Mao, Z.M.: Distributing mobile music applications for audience participation using mobile ad-hoc network (MANET). In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)

    Google Scholar 

  43. Lee, S.W., Freeman, J.: echobo: audience participation using the mobile music instrument. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  44. Lee, S.W., Srinivasamurthy, A., Tronel, G., Shen, W., Freeman, J.: TOK!: a collaborative acoustic instrument using mobile phones. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)

    Google Scholar 

  45. Liu, D., van der Heide, E.: Interaction models for real-time participatory musical performance using mobile devices. In: Proceedings of the International Computer Music Conference (2017)

    Google Scholar 

  46. Martin, C.: Performing with a mobile computer system for vibraphone. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  47. Michon, R., Smith, J.O., Wright, M., Chafe, C., Granzow, J., Wang, G.: Passively augmenting mobile devices towards hybrid musical instrument design. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 19–24 (2017)

    Google Scholar 

  48. Michon, R., Smith, J.O.I., Wright, M., Chafe, C.: Augmenting the iPad: the BladeAxe. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2016)

    Google Scholar 

  49. Michon, R., Smith III, J., Orlarey, Y.: MobileFaust: a set of tools to make musical mobile applications with the faust programming language. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2015)

    Google Scholar 

  50. Miranda, E.R., Wanderley, M.M.: New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. A-R Editions (2006)

    Google Scholar 

  51. Oh, J., Wang, G.: Audience-participation techniques based on social mobile computing. In: Proceedings of the International Computer Music Conference (2011)

    Google Scholar 

  52. Ouzounian, G., Haworth, C., Bennett, P.: Speculative designs: towards a social music. In: Proceedings of the International Computer Music Conference (2017)

    Google Scholar 

  53. Park, S., Ban, S., Hong, D.R., Yeo, W.S.: Sound surfing network (SSN): mobile phone-based sound spatialization with audience collaboration. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  54. Park, T.H., Nieto, O.: Fortissimo: force-feedback for mobile devices. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  55. Pugliese, R., Tahiroglu, K., Goddard, C., Nesfield, J.: Augmenting human-human interaction in mobile group improvisation. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)

    Google Scholar 

  56. Reus, J.: Crackle: a dynamic mobile multitouch topology for exploratory sound interaction. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2011)

    Google Scholar 

  57. Robaszkiewicz, S., Schnell, N.: Soundworks a playground for artists and developers to create collaborative mobile web performances (2015)

    Google Scholar 

  58. Roberts, C.: Control: aoftware for end-user interface programming and interactive performance. In: Proceedings of the International Computer Music Conference (2011)

    Google Scholar 

  59. Roberts, C., Wakefield, G., Wright, M.: Mobile controls On-The-Fly: an abstraction for distributed nimes. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)

    Google Scholar 

  60. Salazar, S., Wang, G.: Auraglyph: Handwritten computer music composition and design. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)

    Google Scholar 

  61. Salazar, S., Wang, G.: miniAudicle for iPad: touchscreen-based music software programming. In: Proceedings of the International Computer Music Conference (2014)

    Google Scholar 

  62. Savage, N.S., Ali, S.R., Chavez, N.E.: Mmmmm: a multi-modal mobile music mixer. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)

    Google Scholar 

  63. Shaw, T., Piquemal, S., Bowers, J.: Fields: an exploration into the use of mobile devices as a medium for sound diffusion. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2015)

    Google Scholar 

  64. Strylowski, B., Allison, J., Guessford, J.: Pitch Canvas: Touchscreen based mobile music instrument. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)

    Google Scholar 

  65. Tanaka, A.: Mapping out instruments, affordances, and mobiles. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2010)

    Google Scholar 

  66. Tanaka, A., Parkinson, A., Settel, Z., Tahiroglu, K.: A survey and thematic analysis approach as input to the design of mobile music GUIs. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2012)

    Google Scholar 

  67. Tarakajian, S., Zicarelli, D., Clayton, J.: Mira: Liveness in iPad controllers for Max/Msp. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2013)

    Google Scholar 

  68. Taylor, B.: A history of the audience as a speaker array. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 481–486 (2017)

    Google Scholar 

  69. Taylor, B., et al.: Simplified expressive mobile development with nexusui, nexusup, and nexusdrop. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2014)

    Google Scholar 

  70. Vaillant, G.L., Villée, G., Dutoit, T.: Portable C++ framework for low-latency musical touch interaction with geometrical shapes. In: Proceedings of the International Computer Music Conference (2017)

    Google Scholar 

  71. Wang, G., et al.: Smule = sonic media: an intersection of the mobile, musical, and social. In: Proceedings of the International Computer Music Conference (2009)

    Google Scholar 

  72. Wang, G.: Ocarina: designing the iphone’s magic flute. Comput. Music. J. 38(2), 8–21 (2014)

    Article  Google Scholar 

  73. Wang, G.: Game design for expressive mobile music. In: Proceedings of the International Conference on New Interfaces for Musical Expression, vol. 16 (2016)

    Google Scholar 

  74. Wang, G., Salazar, S., Oh, J., Hamilton, R.: World stage: crowdsourcing paradigm for expressive social mobile music. J. New Music. Res. 44(2) (2015)

    Article  Google Scholar 

  75. Weinberg, G., Beck, A., Godfrey, M.: ZooZBeat: a gesture-based mobile music studio. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2009)

    Google Scholar 

  76. Weitzner, N., Freeman, J., Chen, Y.L., Garrett, S.: massMobile: towards a flexible framework for large-scale participatory collaborations in live performances. Organ. Sound 18(1), 30–42 (2013)

    Article  Google Scholar 

  77. Yang, Q.: Not all gestures are created equal: gesture and visual feedback in interaction spaces. Ph.D. thesis, University of Michigan (2015)

    Google Scholar 

  78. Yang, Q., Essl, G.: Representation-plurality in multi-touch mobile visual programming for music. In: Proceedings of the International Conference on New Interfaces for Musical Expression (2015)

    Google Scholar 

Download references

Acknowledgments

We are grateful for helpful comments by the reviewers, which helped improve the manuscript.

Author information

Authors and Affiliations

  1. University of Wisconsin – Milwaukee, P.O. Box 413, Milwaukee, WI, 53201, USA

    Georg Essl

  2. University of Michigan, 500 S. State Street, Ann Arbor, MI, 48109, USA

    Sang Won Lee

Authors
  1. Georg Essl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sang Won Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Georg Essl .

Editor information

Editors and Affiliations

  1. Laboratoire PRISM, AMU-CNRS, Marseille, France

    Mitsuko Aramaki

  2. INESC TEC, Porto, Portugal

    Matthew E. P. Davies

  3. Laboratoire PRISM, AMU-CNRS, Marseille, France

    Richard Kronland-Martinet

  4. Laboratoire PRISM, AMU-CNRS, Marseille, France

    Sølvi Ystad

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Essl, G., Lee, S.W. (2018). Mobile Devices as Musical Instruments - State of the Art and Future Prospects. In: Aramaki, M., Davies , M., Kronland-Martinet, R., Ystad, S. (eds) Music Technology with Swing. CMMR 2017. Lecture Notes in Computer Science(), vol 11265. Springer, Cham. https://doi.org/10.1007/978-3-030-01692-0_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-01692-0_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01691-3

  • Online ISBN: 978-3-030-01692-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation