Skip to main content
SpringerLink
Log in

Investigating performance ecologies using screen scores: a case study

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

The purpose of this paper is to deepen the understanding of interconnections among actors and artifacts in a music performance. To this end, this work presents a case study with four participants. Each participant created a piece using Puffin, a screen score system that allows for real-time generation of scores to be played by traditional instrumentalists. We collected field notes and interviews and formalized them with the support of ARCAA (Actors, Role, Context, Activities and Artifacts), a recently developed framework specifically developed for this purpose. By discussing the resulting ecology we highlight some possible design suggestions, including the importance of considering non-direct interaction with a digital system as fundamental for a good “usage” of a given musical system; the relevance of accounting for the entire ecology of artifacts; and that different conceptions of affordances should be used to understand different perspectives of an ecology (in our case, social and hidden affordances).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. We operated in Funchal, Madeira in Portugal.

  2. https://opensoundcontrol.stanford.edu/index.html

  3. https://processing.org/

  4. https://puredata.info/

  5. The interviews analyzed the current study are not publicly available due to privacy issues as our participants are underage but are available from the corresponding author on reasonable request.

References

  1. Gurevich M, Treviño J (2007) Expression and its discontents: toward an ecology of musical creation. In: Proceedings of the 7th international conference on new interfaces for musical expression, NIME ’07. Association for Computing Machinery, New York, USA, pp 106–111. https://doi.org/10.1145/1279740.1279759

  2. Waters S (2007) Performance ecosystems: Ecological approaches to musical interaction. Electroacoustic Music Studies Network, EMS, pp 1–20

    Google Scholar 

  3. Keller D, Lazzarini V (2017) Ecologically grounded creative practices in ubiquitous music. Organised Sound 22(1):61–72. https://doi.org/10.1017/S1355771816000340 Query ID="Q2" text="Please check year provided in references 3, 7, 8, 12, 20, 24, 25, 26, 28, 31, 33-45, 48, 52, 55, 56, 57,61, 67, 70 and 74 if captured correctly."

    Article  Google Scholar 

  4. Keller D (2000) Compositional processes from an ecological perspective. Leonardo Music Journal 10:55–60

    Article  Google Scholar 

  5. Masu R, Bettega M, Correia NN, Romão T, Morreale F (2019) ARCAA : A framework to analyse the artefact ecology in computer music performance. Association for Computing Machinery, New York, USA. https://doi.org/10.1145/3359852.3359870

    Article  Google Scholar 

  6. Jung H, Stolterman E, Ryan W, Thompson T, Siegel M (2008) Toward a framework for ecologies of artifacts: how are digital artifacts interconnected within a personal life? In: Proceedings of the 5th nordic conference on human-computer interaction: Building Bridges, NordiCHI ’08. Association for Computing Machinery, ??? pp. 201–210. https://doi.org/10.1145/1463160.1463182. Accessed 15 Aug 2020

  7. Gibson JJ (2014) The ecological approach to visual perception. Classic Edition, Psychology Press. https://doi.org/10.4324/9781315740218

    Article  Google Scholar 

  8. Magnusson T (2011) Algorithms as scores. Coding live music 21:19–23. https://doi.org/10.1162/LMJ_a_00056. Accessed 09 Sept 2021

  9. Tomás E, Kaltenbrunner M (2014) Tangible scores: Shaping the inherent instrument score. In: Proceedings of the international conference on new interfaces for musical expression,Goldsmiths, University of London, United Kingdom, pp 609–614. https://doi.org/10.5281/zenodo.1178953. http://www.nime.org/proceedings/2014/nime2014_352.pdf

  10. Masu R, Bala P, Ahmad M, Correia NN, Nisi V, Nunes N, Romão T (2020) Vr open scores: Scores as inspiration for vr scenarios. In: Michon R, Schroeder F (eds.) Proceedings of the international conference on new interfaces for musical expression, Birmingham City University, Birmingham, UK, pp 109–114. https://doi.org/10.5281/zenodo.4813262. https://www.nime.org/proceedings/2020/nime2020_paper21.pdf

  11. Hope C, Vickery L (2011) Screen scores. New media music manuscripts

  12. Vickery L (2012) The evolution of notational innovations from the mobile score to the screen score. Publisher: Cambridge University Press, 17(2):128–136. https://doi.org/10.1017/S1355771812000052. Accessed 07 Sept 2021

  13. Winkler GE (2004) The realtime-score: a missing-link in computer-music performance. Sound and Music Computing 4

  14. Masu R, Correia NN, Romao T (2021) Nime scores a systematic review of how scores have shaped performance ecologies in nime. In: Proceedings of the international conference on new interfaces for musical expression, Shanghai, China. https://doi.org/10.21428/92fbeb44.3ffad95a. https://nime.pubpub.org/pub/41cj1pyt

  15. Morreale F, McPherson A (2017) Design for longevity: Ongoing use of instruments from nime 2010-14. In: Proceedings of the international conference on new interfaces for musical expression, Aalborg University Copenhagen, Copenhagen, Denmark. pp 192–197. https://doi.org/10.5281/zenodo.1176218. http://www.nime.org/proceedings/2017/nime2017_paper0036.pdf

  16. Masu R, Correia NN, Romão T (2021) Technology-mediated musical connections. The ecology of a screen-score performance,Association for Computing Machinery, New York, USA, pp 109–116. https://doi.org/10.1145/3478384.3478395

  17. Nyman M (1999) Experimental music cage and beyond. Cambridge University Press, Cambridge

    Google Scholar 

  18. Lucier A (1998) Origins of a form: Acoustical exploration, science and incessancy. Leonardo Music Journal 8(1):5–11

    Article  MathSciNet  Google Scholar 

  19. Cook PR (2001) Principles for designing computer music controllers. In: Proceedings of the international conference on new interfaces for musical expression, Seattle, WA, pp 3–6 . https://doi.org/10.5281/zenodo.1176358. http://www.nime.org/proceedings/2001/nime2001_003.pdf

  20. Bown O, Eldridge A, McCormack J (2009) Understanding interaction in contemporary digital music from instruments to behavioural objects. Publisher Cambridge University Press 14(2):188–196. https://doi.org/10.1017/S1355771809000296. Accessed 08 Sept 2021

  21. Chadabe J (2002) The limitations of mapping as a structural descriptive in electronic instruments. In: Proceedings of the international conference on new interfaces for musical expression, Dublin, Ireland, pp 38–42 . https://doi.org/10.5281/zenodo.1176402. http://www.nime.org/proceedings/2002/nime2002_038.pdf

  22. Schnell N, Battier M (2002) Introducing composed instruments, technical and musicological implications. In: Proceedings of the international conference on new interfaces for musical expression, Dublin, Ireland, pp 156–160. https://doi.org/10.5281/zenodo.1176460. http://www.nime.org/proceedings/2002/nime2002_156.pdf

  23. Tanaka A (2000) Musical performance practice on sensor-based instruments. Trends in Gestural Control of Music 13(389–405):284

    Google Scholar 

  24. Dudas R (2010) “Comprovisation”: The various facets of composed improvisation within interactive performance systems 20:29–31. https://doi.org/10.1162/LMJ_a_00009. Accessed 08 Sept 2021

  25. Magnusson T (2009) Of epistemic tools musical instruments as cognitive extensions. Publisher: Cambridge University Press 14(2):168–176. https://doi.org/10.1017/S1355771809000272. Accessed 08 Sept 2021

  26. Varela FJ, Thompson E, Rosch E (2016) The embodied mind. MIT Press, Cognitive Science and Human Experience

    Google Scholar 

  27. Clark A, Chalmers D (1998) The extended mind. Analysis 58(1):7–19

    Article  Google Scholar 

  28. Magnusson T (2019) Sonic Writing: Technologies of Material, Symbolic, and Signal Inscriptions. Bloomsbury Publishing USA. Google-Books-ID: w93kDwAAQBAJ

  29. Magnusson T (2014) Improvising with the threnoscope: Integrating code, hardware, gui, network, and graphic scores. In: Proceedings of the international conference on new interfaces for musical expression, Goldsmiths, University of London, London, United Kingdom, pp 19–22. https://doi.org/10.5281/zenodo.1178857. http://www.nime.org/proceedings/2014/nime2014_276.pdf

  30. Magnusson T (2015) Code scores in live coding practice. In: Proceedings of the international conference for technologies for music notation and representation, Paris, p 5

  31. Withagen R, de Poel HJ, Araújo D, Pepping G-J (2012) Affordances can invite behavior: Reconsidering the relationship between affordances and agency 30(2):250–258. https://doi.org/10.1016/j.newideapsych.2011.12.003. Accessed 03 Aug 2021

  32. Heft H (1989) Affordances and the body: An intentional analysis of gibson’s ecological approach to visual perception. Journal for the theory of social behaviour 19(1):1–30

    Article  Google Scholar 

  33. Costall A (1995). Socializing Affordances. https://doi.org/10.1177/0959354395054001

    Article  Google Scholar 

  34. Norman DA (1999) Affordance conventions and design. 6(3):38–43. https://doi.org/10.1145/301153.301168. Accessed 26 July 2021

  35. Gaver WW (1991) Technology affordances. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’91, Association for Computing Machinery, pp 79–84. https://doi.org/10.1145/108844.108856. Accessed 26 July 2021

  36. Dourish P (2003) The appropriation of interactive technologies: Some lessons from placeless documents. 12(4):465–490. https://doi.org/10.1023/A:1026149119426. Accessed 25 May 2021

  37. Dix A (2007) Designing for appropriation. Publisher BCS Learning & Development. https://doi.org/10.14236/ewic/HCI2007.53. Accessed 05 Aug 2021

  38. Leong TW (2006) Designing for experiences: randomness as a resource. In: Proceedings of the 6th conference on designing interactive systems. DIS ’06, Association for Computing Machinery, pp 346–347. https://doi.org/10.1145/1142405.1142458. Accessed 26 Aug 2021

  39. Sengers P, Gaver B (2006) Staying open to interpretation: engaging multiple meanings in design and evaluation. In: Proceedings of the 6th conference on designing interactive systems. DIS ’06, Association for Computing Machinery. pp 99–108. https://doi.org/10.1145/1142405.1142422. Accessed 23 Aug 2021

  40. Höök K, Sengers P, Andersson G (2003) Sense and sensibility: evaluation and interactive art. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’03, Association for Computing Machinery, pp 241–248. https://doi.org/10.1145/642611.642654. Accessed 23 Aug 2021

  41. Gaver, WW, Bowers J, Boucher A, Gellerson H, Pennington S, Schmidt A, Steed A, Villars N, Walker B (2004) The drift table: designing for ludic engagement. In: CHI ’04 extended abstracts on human factors in computing systems. CHI EA ’04. Association for Computing Machinery, pp 885–900. https://doi.org/10.1145/985921.985947. Accessed 23 Aug 2021

  42. Gaver, WW, Beaver J, Benford S (2003) Ambiguity as a resource for design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’03, Association for Computing Machinery. pp 233–240. https://doi.org/10.1145/642611.642653. Accessed 25 May 2021

  43. Bell G, Blythe M, Sengers P (2005) Making by making strange. Defamiliarization and the design of domestic technologies 12(2):149–173. https://doi.org/10.1145/1067860.1067862. Accessed 23 Aug 2021

  44. Muth C, Hesslinger VM, Carbon C-C (2015) The appeal of challenge in the perception of art: How ambiguity, solvability of ambiguity, and the opportunity for insight affect appreciation. Publisher: Educational Publishing Foundation, Place: US, 9(3):206–216. https://doi.org/10.1037/a0038814

  45. Zappi V, Mcpherson A (2014) Design and use of a hackable digital instrument. Accessed 06 Aug 2021

  46. Zappi V, McPherson A (2014) Dimensionality and appropriation in digital musical instrument design. In: Proceedings of the international conference on new interfaces for musical expression, Goldsmiths, University of London, London, United Kingdom. pp 455–460. https://doi.org/10.5281/zenodo.1178993. http://www.nime.org/proceedings/2014/nime2014_409.pdf

  47. Gurevich M, Stapleton P, Marquez-Borbon A (2010) Style and constraint in electronic musical instruments. In: Proceedings of the international conference on new interfaces for musical expression, Sydney, Australia, pp 106–111. https://doi.org/10.5281/zenodo.1177785. http://www.nime.org/proceedings/2010/nime2010_106.pdf

  48. Bødker S (2015) Third-wave HCI, 10 years later—participation and sharing 22(5):24–31. https://doi.org/10.1145/2804405. Accessed 20 May 2021

  49. Bødker S, Klokmose C (2013) V. human-computer interaction 26(4):315–371. https://doi.org/10.1080/07370024.2011.626709

  50. Resmini A, Rosati L (2009) Information architecture for ubiquitous ecologies. In: Proceedings of the international conference on management of emergent digital ecosystems, pp 196–199

  51. Avila JPM, Greenhalgh C, Hazzard A, Benford S, Chamberlain A (2019) Encumbered interaction: A study of musicians preparing to perform. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–13. https://doi.org/10.1145/3290605.3300706

  52. Bettega M, Masu R, Tel M (2021) “It’s like a GPS community tool”: Tactics to foster digital commons through artifact ecology. In: Designing interactive systems conference 2021, DIS ’21, Association for Computing Machinery, pp 1710–1725. https://doi.org/10.1145/3461778.3462034. Accessed 28 Aug 2021

  53. Larsen-Ledet I, Korsgaard H, Bødker S (2020) Collaborative writing across multiple artifact ecologies. In: Proceedings of the 2020 CHI conference on human factors in computing systems, pp 1–14. https://doi.org/10.1145/3313831.3376422

  54. Bødker S, Lyle P, Saad-Sulonen J (2017) Untangling the mess of technological artifacts: investigating community artifact ecologies. In: Proceedings of the 8th international conference on communities and technologies, pp 246–255. https://doi.org/10.1145/3083671.3083675

  55. Magnusson T (2021) The migration of musical instruments: On the socio-technological conditions of musical evolution. Publisher: Routledge, 50(2):175–183. https://doi.org/10.1080/09298215.2021.1907420. Accessed 10 Sept 2021

  56. Tahiroğlu K, Magnusson T, Parkinson A, Garrelfs I, Tanaka A (2020) Digital musical instruments as probes: How computation changes the mode-of-being of musical instruments. Publisher: Cambridge University Press, 25(1):64–74. https://doi.org/10.1017/S1355771819000475. Accessed 08 Sept 2021 Query ID="Q3" text="Please check the last name of Author (Tahiroğlo K), if captured and presented correctly."

  57. Rodger M, Stapleton P, van Walstijn M, Ortiz M, Pardue LS What makes a good musical instrument? a matter of processes, ecologies and specificities. In: Michon R, Schroeder F (eds.) Proceedings of the international conference on new interfaces for musical expression, Birmingham City University, pp 405–410. ISSN: 2220-4806. https://doi.org/10.5281/zenodo.4813438. https://www.nime.org/proceedings/2020/nime2020_paper79.pdf

  58. Lazzarini V, Keller D, Otero N, Turchet L (2020) The ecologies of ubiquitous music. In: Ubiquitous Music Ecologies, Routledge, ??? pp 1–22

  59. Melbye AP (2021) Resistance, mastery, agency: Improvising with the feedback-actuated augmented bass. Organised Sound 26(1):19–30. https://doi.org/10.1017/S1355771821000029

    Article  Google Scholar 

  60. Stapleton P, Davis T (2021) Ambiguous devices: Improvisation, agency, touch and feedthrough in distributed music performance. Organised Sound 26(1):52–64. https://doi.org/10.1017/S1355771821000054

    Article  Google Scholar 

  61. Waters S (2021) The entanglements which make instruments musical: Rediscovering sociality. Publisher: Routledge, 50(2):133–146. https://doi.org/10.1080/09298215.2021.1899247. Accessed 10 Sept 2021

  62. Lucas A, Harrison J, Schroeder F, Ortiz M (2021) Cross-pollinating ecological perspectives in admi design and evaluation. In: International conference on new interfaces for musical expression. PubPub

  63. Turchet L, Fischione C, Essl G, Keller D, Barthet M (2018) Internet of musical things: Vision and challenges. Ieee access 6:61994–62017

    Article  Google Scholar 

  64. Masu R, Correia NN, Jurgens S, Feitsch J, Romão T (2020) Designing interactive sonic artefacts for dance performance: An ecological approach, Association for Computing Machinery, New York, USA, pp 122–129. https://doi.org/10.1145/3411109.3412297

  65. Felice MC, Alaoui SF, Mackay WE (2021) Studying choreographic collaboration in the wild. In: Designing Interactive Systems Conference 2021. DIS ’21, Association for Computing Machinery, New York, USA, pp 2039–2051. https://doi.org/10.1145/3461778.3462063

  66. Fdili Alaoui S (2019) Making an interactive dance piece: Tensions in integrating technology in art. In: Proceedings of the 2019 on designing interactive systems conference. DIS ’19, Association for Computing Machinery, New York, USA pp 1195–1208. https://doi.org/10.1145/3322276.3322289

  67. Dourish P (2001) Where the Action Is: The foundations of embodied interaction. MIT Press

  68. Benford S, Greenhalgh C, Crabtree A, Flintham M, Walker B, Marshall J, Koleva B, Rennick Egglestone S, Giannachi G, Adams M, Tandavanitj N, Row Farr J (2013) Performance-led research in the wild. ACM Trans. Comput.-Hum. Interact. 20(3). https://doi.org/10.1145/2491500.2491502

  69. Neustaedter C, Sengers P (2012) Autobiographical design in hci research: Designing and learning through use-it-yourself. In: Proceedings of the designing interactive systems conference. DIS ’12, Association for Computing Machinery, New York, USA, pp 514–523. https://doi.org/10.1145/2317956.2318034

  70. Hook J, McCarthy J, Wright P, Olivier P (2013) Waves: exploring idiographic design for live performance. In: Proceedings of the SIGCHI conference on human factors in computing systems, Association for Computing Machinery, pp 2969–2978. https://doi.org/10.1145/2470654.2481412. Accessed 07 Sept 2021

  71. Turchet L (2018) Smart mandolin: Autobiographical design, implementation, use cases, and lessons learned. In: Proceedings of the audio mostly 2018 on sound in immersion and emotion. AM’18. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3243274.3243280

  72. Braun V, Clarke V (2006) Using thematic analysis in psychology. Qualitative research in psychology 3(2):77–101. https://doi.org/10.1191/1478088706qp063oa

  73. Bødker S, Dindler C, Iversen OS, Smith RC (2022) What is participatory design? In: Participatory design, Springer, ??? pp 5–13

  74. Lyle P, Korsgaard H, Bødker S (2020) What is in an ecology? a review of artifact, communicative, device and information ecologies. In: Proceedings of the 11th nordic conference on human-computer interaction: Shaping Experiences, Shaping Society. NordiCHI ’20, Association for Computing Machinery, pp 1–14. https://doi.org/10.1145/3419249.3420185. Accessed 28 Aug 2021

  75. Šmorgun I, Lamas D (2015) Exploring the use of the human-artifact model for studying ubiquitous interactions. In: Proceedings of the mulitimedia, interaction, design and innnovation. MIDI ’15, Association for Computing Machinery, ??? pp 1–7. https://doi.org/10.1145/2814464.2814484. Accessed 15 Aug 2020

  76. Bogers S, van Kollenburg J, Deckers E, Frens J, Hummels C (2018) A situated exploration of designing for personal health ecosystems through data-enabled design. In: Proceedings of the 2018 designing interactive systems conference. DIS ’18, Association for Computing Machinery, ??? pp 109–120. https://doi.org/10.1145/3196709.3196769. Accessed 15 Aug 2020

  77. Fdili Alaoui S (2019) Making an interactive dance piece: Tensions in integrating technology in art. In: Proceedings of the 2019 on designing interactive systems conference, pp 1195–1208

  78. Bakker S (2013) Design for peripheral interaction. Eindhoven University of Technology 184

  79. Bakker S, van den Hoven E, Eggen B (2015) Peripheral interaction: characteristics and considerations. Personal and ubiquitous computing 19(1):239–254

    Article  Google Scholar 

Download references

Acknowledgements

We wish to acknowledge the four participants in the study. We also want to thank the reviewers for their useful and valuable insights to this paper. The first and second author acknowledges ARDITI - Agencia Regional para o Desenvolvimento e Tecnologia under the scope of the Project M1420-09-5369-FSE-000002 - PhD Studentship. We also acknowledges the support of LARSyS to this research (Projeto - UIDB/50009/2020). This work was also co-funded by FCT/MCTES NOVA LINCS PEst UID/CEC/04516/2019.

Author information

Authors and Affiliations

  1. Institute of Music Science and Engineering, King Mongkut’s Institute of Technology Ladkrabang, Bangkok, Thailand

    Raul Masu

  2. Department of Informatics, FCT, Universidade Nova de Lisboa, Lisbon, Portugal

    Raul Masu & Teresa Romão

  3. ITI/LARSyS, Lisbon, Portugal

    Raul Masu & Nuno N. Correia

  4. NOVA LINCS, Lisbon, Portugal

    Raul Masu & Teresa Romão

  5. FCSH, Universidade Nova de Lisboa, Lisbon, Portugal

    Mela Bettega

  6. Tallinn University, Tallinn, Estonia

    Nuno N. Correia

Authors
  1. Raul Masu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mela Bettega
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nuno N. Correia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Teresa Romão
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Raul Masu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

Supplementary File1 (PDF 54.4 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Masu, R., Bettega, M., Correia, N.N. et al. Investigating performance ecologies using screen scores: a case study. Pers Ubiquit Comput 27, 1887–1907 (2023). https://doi.org/10.1007/s00779-023-01719-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-023-01719-y

Keywords

Search

Navigation