research-article

Boiling Mind: Amplifying the Audience-Performer Connection through Sonification and Visualization of Heart and Electrodermal Activities

Authors:

Taichi Furukawa,

George Chernyshov,

Marcelo Padovani,

Dingding Zheng,

Kouta MinamizawaAuthors Info & Claims

TEI '21: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction

Article No.: 34, Pages 1 - 10

https://doi.org/10.1145/3430524.3440653

Published: 14 February 2021 Publication History

TEI '21: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction

Boiling Mind: Amplifying the Audience-Performer Connection through Sonification and Visualization of Heart and Electrodermal Activities

Pages 1 - 10

Abstract
Supplementary Material
References

Abstract

In stage performances, an invisible wall in front of the stage often weakens the connections between the audience and performers. To amplify this performative connection, we present the concept ”Boiling Mind”. Our design concept is based on streaming sensor data related to heart and electrodermal activities from audience members and integrating this data into staging elements, such as visual projections, music, and lighting. Thus, the internal states of the audience directly influence the staging. Artists can have a more direct perception of the inner reactions of audience members and can create physical expressions in response to them. In this paper, we present the wearable sensing system as well as design considerations of mapping heart and electrodermal activity to changes in the staging elements. We evaluated our design and setup over three live performances.

Supplementary Material

MP4 File (3430524.3440653.mp4)

Supplementary video

Download
739.23 MB

References

[1]

John L Andreassi. 2010. Psychophysiology: Human behavior and physiological response. Psychology Press.

[2]

John L Andreassi. 2013. Psychophysiology: Human behavior & physiological response. Psychology Press.

[3]

Philip Auslander. 2008. Liveness: Performance in a mediatized culture. Routledge.

[4]

Steve Benford, Chris Greenhalgh, Andy Crabtree, Martin Flintham, Brendan Walker, Joe Marshall, Boriana Koleva, Stefan Rennick Egglestone, Gabriella Giannachi, Matt Adams, Nick Tandavanitj, and Ju Row Farr. 2013. Performance-led research in the wild. ACM Transactions on Computer-Human Interaction (TOCHI) 20, 3(2013), 1–22. https://doi.org/10.1145/2491500.2491502

Digital Library

[5]

Miranda Boorsma. 2006. A strategic logic for arts marketing: Integrating customer value and artistic objectives. International journal of cultural policy 12, 1 (2006), 73–92.

[6]

Wolfram Boucsein. 2012. Electrodermal activity: Springer Science & Business Media. Broek, EL vd, Schut, MH, Westerink, JHDM, Herk, J. v., & Tuinenbreijer, K (2012).

[7]

Astrid Breel. 2015. Audience agency in participatory performance: a methodology for examining aesthetic experience. Participations: Journal of Audience & Reception Studies 12, 1(2015), 368–387.

[8]

Susan Broadhurst. 2002. Blue Bloodshot Flowers: interaction, reaction and performance. Digital Creativity 13, 3 (2002), 157–163. https://doi.org/10.1076/digc.13.3.157.7340

[9]

Jos F Brosschot, Eduard Van Dijk, and Julian F Thayer. 2007. Daily worry is related to low heart rate variability during waking and the subsequent nocturnal sleep period. International journal of psychophysiology 63, 1 (2007), 39–47.

[10]

Courtney Brown. 2019. Machine Tango: An Interactive Tango Dance Performance. 565–569. https://doi.org/10.1145/3294109.3301263

Digital Library

[11]

Teresa Cerratto-Pargman, Chiara Rossitto, and Louise Barkhuus. 2014. Understanding Audience Participation in an Interactive Theater Performance. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (Helsinki, Finland) (NordiCHI ’14). Association for Computing Machinery, New York, NY, USA, 608–617. https://doi.org/10.1145/2639189.2641213

Digital Library

[12]

Kwang-Ho Choi, Junbeom Kim, O Sang Kwon, Min Ji Kim, Yeon Hee Ryu, and Ji-Eun Park. 2017. Is heart rate variability (HRV) an adequate tool for evaluating human emotions?–A focus on the use of the International Affective Picture System (IAPS). Psychiatry Research 251(2017), 192–196.

[13]

Greg Corness, Kristin Carlson, and Thecla Schiphorst. 2011. Audience empathy: a phenomenological method for mediated performance. In Proceedings of the 8th ACM conference on Creativity and cognition. 127–136. https://doi.org/10.1145/2069618.2069641

Digital Library

[14]

Stephen Davies. 1997. John Cage’s 4’ 33 ”: Is it music ?Australasian Journal of Philosophy 75, 4 (1997), 448–462.

[15]

E. Delaherche, M. Chetouani, A. Mahdhaoui, C. Saint-Georges, S. Viaux, and D. Cohen. 2012. Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines. IEEE Transactions on Affective Computing 3, 3 (2012), 349–365.

Digital Library

[16]

Steve Dixon and Barry Smith. 2007. Digital performance: a history of new media in theater, dance, performance art, and installation. MIT press.

[17]

Barbara Ehrenreich. 2007. Dancing in the streets: A history of collective joy. Macmillan.

[18]

Byeongwon Ha. 2015. A Pioneer of Interactive Art: Nam June Paik as Musique Concrète Composing Researcher. In ISEA (Vancouver, International Symposium on Electronic Art, 2015).

[19]

Rob Haskins. 2013. John Cage. Reaktion Books.

[20]

Mariam Hassib, Stefan Schneegass, Philipp Eiglsperger, Niels Henze, Albrecht Schmidt, and Florian Alt. 2017. EngageMeter: A system for implicit audience engagement sensing using electroencephalography. In Proceedings of the 2017 Chi conference on human factors in computing systems. 5114–5119. https://doi.org/10.1145/3025453.3025669

Digital Library

[21]

Michael Hemphill. 1996. A note on adults’ color–emotion associations. The Journal of genetic psychology 157, 3 (1996), 275–280.

[22]

Kristina Höök. 2009. Affective loop experiences: designing for interactional embodiment. Royal Society 364, 1535, Article B (12 December 2009). https://doi.org/10.1098/rstb.2009.0202

[23]

Keith W Jacobs and Frank E Hustmyer Jr. 1974. Effects of four psychological primary colors on GSR, heart rate and respiration rate. Perceptual and motor skills 38, 3 (1974), 763–766.

[24]

Corinne Jola. 2016. The magic connection: Dancer-audience interaction. Zwischenleiblichkeit und Bewegtes Verstehen–Intercorporeity, movement and tacit knowledge(2016), 269–287.

[25]

Peter Jönsson. 2007. Respiratory sinus arrhythmia as a function of state anxiety in healthy individuals. International journal of psychophysiology 63, 1 (2007), 48–54.

[26]

Carl Gustav Jung. 1918. Studies in word-association. Heinemann.

[27]

Keina Konno, Richi Owaki, Yoshito Onishi, Akiko Takeshita, Ryo Kanda, Naoko Shiomi, Kyle McDonald, Satoru Higa, Motoi Shimizu, Yosuke Sakai, Yasuaki Kakehi, Kazuhiro Jo, Yoko Ando, Kazunao Abe, and Takayuki Ito. 2016. Dividual Plays Experimental Lab-An installation derived from Dividual Plays. TEI ’16, February 14 - 17, 2016, Eindhoven, Netherlands (2016). https://doi.org/10.1145/2839462.2856346

Digital Library

[28]

Richard Kostelanetz. 2003. Conversing with cage. Routledge.

[29]

Rikard Küller, Byron Mikellides, and Jan Janssens. 2009. Color, arousal, and performance—A comparison of three experiments. Color Research and Application 34, 2 (2009), 141–152. https://doi.org/10.1002/col.20476

[30]

Peter J Lang. 1995. The emotion probe: studies of motivation and attention.American psychologist 50, 5 (1995), 372.

[31]

Peter J Lang, Mark K Greenwald, Margaret M Bradley, and Alfons O Hamm. 1993. Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology 30, 3 (1993), 261–273.

[32]

Celine Latulipe, Erin A. Carroll, and Danielle Lottridge. 2011. Love, Hate, Arousal and Engagement: Exploring Audience Responses to Performing Arts. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1978942.1979210

Digital Library

[33]

Jane Lessiter, Jonathan Freeman, Edmund Keogh, and Jules Davidoff. 2001. A cross-media presence questionnaire: The ITC-Sense of Presence Inventory. Presence: Teleoperators & Virtual Environments 10, 3(2001), 282–297.

Digital Library

[34]

Christopher Lindinger, Martina Mara, Klaus Obermaier, Roland Aigner, Roland Haring, and Veronika Pauser. 2013. The (St)Age of Participation: audience involvement in interactive performances. Digital Creativity 24, 2 (2013), 119–129. https://doi.org/10.1080/14626268.2013.808966

[35]

Banu Manav. 2007. Color-emotion associations and color preferences: A case study for residences. Color Research and Application 32, 2 (2007), 144–150. https://doi.org/10.1002/col.20294

[36]

Gewnhi Park and Julian F Thayer. 2014. From the heart to the mind: cardiac vagal tone modulates top-down and bottom-up visual perception and attention to emotional stimuli. Frontiers in psychology 5 (2014), 278.

[37]

TW Picton, I Martin, and PH Venables. 1980. Techniques in psychophysiology. (1980).

[38]

Jennifer Radbourne, Katya Johanson, Hilary Glow, and Tabitha White. 2009. The Audience Experience: Measuring Quality in the Performing Arts. International Journal of Arts Management 11 (01 2009).

[39]

Richard Ramchurn, Sarah Martindale, Max L Wilson, and Steve Benford. 2019. From Director’s Cut to User’s Cut: to Watch a Brain-Controlled Film is to Edit it. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–14. https://doi.org/10.1145/3290605.3300378

Digital Library

[40]

Danilo Rodrigues, Emily Grenader, Fernando Nos, Marcel Dall’Agnol, Troels Hansen, and Nadir Weibel. 2013. MotionDraw: a tool for enhancing art and performance using kinect. In CHI ’13. https://doi.org/10.1145/2468356.2468570

Digital Library

[41]

Asreen Rostami, Donald McMillan, Elena Márquez Segura, Chiara Rossito, and Louise Barkhuus. 2017. Bio-sensed and embodied participation in interactive performance. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction. 197–208.

Digital Library

[42]

Fred Shaffer and JP Ginsberg. 2017. An overview of heart rate variability metrics and norms. Frontiers in public health 5 (2017), 258.

[43]

Catherine Stevens, Renee Glass, Emery Schubert, Johnson Chen, and Heather Winskel. 2007. Methods for measuring audience reactions. In Proceedings of the inaugeral International Conference on Music Communication Science. Citeseer, 155.

[44]

Julian F Thayer and Esther Sternberg. 2006. Beyond heart rate variability: vagal regulation of allostatic systems. Annals of the New York Academy of Sciences 1088, 1 (2006), 361–372.

[45]

Christian Tronstad, Gaute E Gjein, Sverre Grimnes, Ørjan G Martinsen, Anne-Lene Krogstad, and Erik Fosse. 2008. Electrical measurement of sweat activity. Physiological measurement 29, 6 (2008), S407.

[46]

Marieke van Dooren, Joris H Janssen, 2012. Emotional sweating across the body: Comparing 16 different skin conductance measurement locations. Physiology & behavior 106, 2 (2012), 298–304.

[47]

Francisco J Varela, Evan Thompson, and Eleanor Rosch. 2016. The embodied mind: Cognitive science and human experience. MIT press.

[48]

Jamie A Ward, Daniel Richardson, Guido Orgs, Kelly Hunter, and Antonia Hamilton. 2018. Sensing Interpersonal Synchrony between Actors and Autistic Children in Theatre Using Wrist-Worn Accelerometers. Proceedings of the 2018 ACM International Symposium on Wearable Computers, 148–155. https://doi.org/10.1145/3267242.3267263

Digital Library

[49]

Lois B Wexner. 1954. The degree to which colors (hues) are associated with mood-tones. Journal of applied psychology 38, 6 (1954), 432.

Cited By

Mõttus MCazin MTikka P(2024)Standardising Scales for Arousal Metrics of Electrodermal Activity: A Case Study in Experience DesignUser Experience - A Multidimensional Analysis of Human-Artefact Interactions [Working Title]10.5772/intechopen.1007760Online publication date: 14-Nov-2024
https://doi.org/10.5772/intechopen.1007760
Barbareschi GZhou SRyoichi AKawaguchi MArmstrong MOgino MAoiki SOhta ETaguchi HKamiyama YMuto MYoshifuji KMinamizawa K(2024)Brain Body Jockey project: Transcending Bodily Limitations in Live Performance via Human AugmentationProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675621(1-14)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675621
Correia NPrimett W(2024)Best Practices for Technology-Mediated Audience Interaction in Dance PerformancesProceedings of the 9th International Conference on Movement and Computing10.1145/3658852.3659083(1-6)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3658852.3659083
Show More Cited By

Index Terms

Boiling Mind: Amplifying the Audience-Performer Connection through Sonification and Visualization of Heart and Electrodermal Activities
1. Applied computing
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Boiling Mind - A Dataset of Physiological Signals during an Exploratory Dance Performance
AHs '21: Proceedings of the Augmented Humans International Conference 2021

The relationship between audience and performers is crucial to what makes live events so special. The aim of this work is to develop a new approach amplifying the link between audiences and performers. Specifically, we explore the use of wearable ...
Studying and designing emotions in live interactions with the audience
Abstract
In the last decade, a number of projects have investigated the use of digital technologies to increase the engagement of the audience in live interactive performance. In this paper, we explore the relation between interactive performance and ...
Activities-based virtual reality experience for better audience engagement
Abstract
The benefits of virtual reality have definitely expanded beyond the gaming industry to a variety of other businesses. From virtual conferences to virtual exhibitions, the MICE (Meetings, Incentives, Conferences, and Exhibitions) ...
Highlights
- Activities in the virtual reality experience improve audience engagement.
- ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

TEI '21: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction

February 2021

908 pages

ISBN:9781450382137

DOI:10.1145/3430524

Editor:
Raphael Wimmer
University of Regensburg, Germany
,
General Chairs:
Martin Kaltenbrunner
University of Art and Design Linz, Austria
,
Martin Murer
University of Salzburg, Austria
,
Program Chairs:
Katrin Wolf
Beuth University of Applied Sciences, Berlin, Germany
,
Ian Oakley
UNIST, Ulsan, Republic of Korea

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 February 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

JST CREST

Conference

TEI '21

TEI '21: Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction

February 14 - 17, 2021

Salzburg, Austria

Acceptance Rates

TEI '21 Paper Acceptance Rate 40 of 136 submissions, 29%;

Overall Acceptance Rate 393 of 1,367 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
677
Total Downloads

Downloads (Last 12 months)90
Downloads (Last 6 weeks)7

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mõttus MCazin MTikka P(2024)Standardising Scales for Arousal Metrics of Electrodermal Activity: A Case Study in Experience DesignUser Experience - A Multidimensional Analysis of Human-Artefact Interactions [Working Title]10.5772/intechopen.1007760Online publication date: 14-Nov-2024
https://doi.org/10.5772/intechopen.1007760
Barbareschi GZhou SRyoichi AKawaguchi MArmstrong MOgino MAoiki SOhta ETaguchi HKamiyama YMuto MYoshifuji KMinamizawa K(2024)Brain Body Jockey project: Transcending Bodily Limitations in Live Performance via Human AugmentationProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675621(1-14)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675621
Correia NPrimett W(2024)Best Practices for Technology-Mediated Audience Interaction in Dance PerformancesProceedings of the 9th International Conference on Movement and Computing10.1145/3658852.3659083(1-6)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3658852.3659083
Liu MBello HZhou BLukowicz PKarolus J(2024)iFace: Hand-Over-Face Gesture Recognition Leveraging Impedance SensingProceedings of the Augmented Humans International Conference 202410.1145/3652920.3652923(131-137)Online publication date: 4-Apr-2024
https://dl.acm.org/doi/10.1145/3652920.3652923
Hynds DChernyshov GZheng DUyama ALi JMatsumoto KPogorzhelskiy MKunze KWard JMinamizawa KCiolfi LHogan TDöring TJenkins Tvan Dijk JHuron SLi ZCoyle DSigner B(2024)Innermost Echoes: Integrating Real-Time Physiology into Live Music PerformancesProceedings of the Eighteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3623509.3633356(1-12)Online publication date: 11-Feb-2024
https://dl.acm.org/doi/10.1145/3623509.3633356
Sparrow LGalwey CLoveridge BGlasser SOsborne MKelly R(2024)Heart and Soul: The Ethics of Biometric Capture in Immersive Artistic PerformanceProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642309(1-23)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642309
Li XShen XKamiyama YHynds DHorie AWakisaka SMinamizawa K(2024)“It’s Like Being on Stage”: Staging an Improvisational Haptic-Installed Contemporary Dance PerformanceHaptics: Understanding Touch; Technology and Systems; Applications and Interaction10.1007/978-3-031-70058-3_41(507-518)Online publication date: 30-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-70058-3_41
Xie YBarbareschi GKunze KInakage M(2023)Exploring Digital Embodiment in Wheelchair Dance with Generative AIProceedings of the 13th International Conference on the Internet of Things10.1145/3627050.3631576(224-227)Online publication date: 7-Nov-2023
https://dl.acm.org/doi/10.1145/3627050.3631576
Hynds DZheng DZhang YMa HRagozin KChernyshov GWard JSaito TKunze KMinamizawa KWard JMcGill MMarky K(2023)Phantom Undulations: Remote Physiological Sensing in Abstract Installation WorksProceedings of the Augmented Humans International Conference 202310.1145/3582700.3583710(367-370)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3583710
Meng XHe YKunze KWard JMcGill MMarky K(2023)Towards Enhancing a Recorded Concert Experience in Virtual Reality by Visualizing the Physiological Data of the AudienceProceedings of the Augmented Humans International Conference 202310.1145/3582700.3583709(330-333)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3583709
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten