Skip to main content
SpringerLink
Log in

Exploring Effective Relationships Between Visual-Audio Channels in Data Visualization

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

In recent years, there has been a growing trend towards taking advantage of audio--visual representations. Previous research has aimed at improving users’ performance and engagement with these representations. The attainment of these benefits primarily depends on the effectiveness of audio--visual relationships used to represent the data. However, the visualization field yet lacks an empirical study that guides the effective relationships. Given the compatibility effect between visual and auditory channels, this research presents the effectiveness of four audio channels (timbre, pitch, loudness, and tempo) with six visual channels (spatial position, color, position, length, angle, and area). In six experiments, one per visual channel, we observed how each audio channel, when used with a visual channel, impacted users’ ability to perform the differentiation or similarity task accurately. Each experiment provided the ranking of audio channels along a visual channel. Central to our experiments was the evaluation at two stages, and accordingly, we identified the effectiveness. Our results showed that timbre, with spatial position and color, aided in more accurate target identification than the three other audio channels. With position and length, pitch allowed a more accurate judgment of the magnitude of data than loudness and tempo but was less accurate than the other two channels along angle and area. Overall, our experiments showed that the choice of representation methods and tasks had impacted the effectiveness of audio channels.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://www.audacityteam.org/.

References

  • Adeli M, Rouat J, Molotchnikoff S (2014) Audiovisual correspondence between musical timbre and visual shapes. Front Hum Neurosci 8:352

    Article  Google Scholar 

  • Batch A, Ji Y, Fan M, Zhao J, Elmqvist N (2023) uxSense: Supporting user experience analysis with visualization and computer vision. IEEE Trans Vis Comput Graph, To appear

  • Blanca Mena MJ, Alarcón Postigo R, Arnau Gras J, Bono Cabré R, Bendayan R (2017) Non-normal data: is anova still a valid option? Psicothema 29(4):552–557

    Google Scholar 

  • Blazhenkova O, Kumar MM (2018) Angular versus curved shapes: correspondences and emotional processing. Perception 47(1):67–89

    Article  Google Scholar 

  • Borkin M, Gajos K, Peters A, Mitsouras D, Melchionna S, Rybicki F, Feldman C, Pfister H (2011) Evaluation of artery visualizations for heart disease diagnosis. IEEE Trans Vis Comput Graph 17(12):2479–2488

    Article  Google Scholar 

  • Bouchara T, Katz BF, Jacquemin C, Guastavino C (2010) Audio-visual renderings for multimedia navigation. In: Proc. of International Conference on Auditory Display, pp 245–252

  • Brewster SA, Clarke CV (2005) The design and evaluation of a sonically enhanced tool palette. ACM Trans Appl Percept 2(4):455–461

    Article  Google Scholar 

  • Bruls M, Huizing K, Wijk JJV (2000) Squarified treemaps. In: Proc. of Eurographics Conference on Visualization, pp 33–42

  • Cleveland WS, McGILL R (1984) Graphical perception: Theory, experimentation, and application to the development of graphical methods. J Am Stat Assoc 79(387):531–554

    Article  Google Scholar 

  • Daudé S, Nigay L (2003) Design process for auditory interfaces. In: Proc. of International Conference on Auditory Display, pp 176–179

  • Demiralp Ç, Bernstein MS, Heer J (2014) Learning perceptual kernels for visualization design. IEEE Trans Vis Comput Graph 20(12):1933–1942

    Article  Google Scholar 

  • Demiralp Ç, Scheidegger CE, Kindlmann GL, Laidlaw DH, Heer J (2014) Visual embedding: a model for visualization. IEEE Comput Graph Appl 34(1):10–15

    Article  Google Scholar 

  • Du M, Chou JK, Ma C, Chandrasegaran S, Ma KL (2018) Exploring the role of sound in augmenting visualization to enhance user engagement. In: Proc. of IEEE Pacific Visualization Symposium, pp 225– 229

  • Dubus G, Bresin R (2013) A systematic review of mapping strategies for the sonification of physical quantities. PloS ONE 8(12):e82491

    Article  Google Scholar 

  • Enge K, Rind A, Iber M, H¨oldrich R, Aigner W (2022) Towards multimodal exploratory data analysis: Soniscope as a prototypical implementation. In: Proc. of Eurographics Conference on Visualization-Short Papers, pp 67–71

  • Ernst MO (2007) Learning to integrate arbitrary signals from vision and touch. J Vis 7(5):1–14

    Article  Google Scholar 

  • Evans KK, Treisman A (2010) Natural cross-modal mappings between visual and auditory features. J Vis 10(1):6

    Article  Google Scholar 

  • Ferguson J, Brewster SA (2018) Investigating perceptual congruence between data and display dimensions in sonification. In: Proc. of ACM CHI Conference on Human Factors in Computing Systems, pp 1–9

  • Flowers JH (2005) Thirteen years of reflection on auditory graphing: Promises, pitfalls, and potential new directions. In:Proc. of International Conference on Auditory Display, pp 406–409

  • Franklin KM, Roberts JC (2003) Pie chart sonification. In: Proc. of International Conference on Information Visualisation, pp 4–9

  • Ghosh S, Winston L, Panchal N, Kimura-Thollander P, Hotnog J, Cheong D, Reyes G, Abowd GD (2018) Notifivr: exploring interruptions and notifications in virtual reality. IEEE Trans Vis Comput Graph 24(4):1447–1456

    Article  Google Scholar 

  • Giovannangeli L, Bourqui R, Giot R, Auber D (2022) Color and shape efficiency for outlier detection from automated to user evaluation. Vis Inform 6(2):25–40

    Article  Google Scholar 

  • Gogolou A, Tsandilas T, Bezerianos P, Bezerianos A (2019) Comparing similarity perception in time series visualizations. IEEE Trans Vis Comput Graph 25(1):523–533

    Article  Google Scholar 

  • Han YC, Surve P (2019) Eyes: Iris sonification and interactive biometric art. In: Proc. of ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp 1–4

  • Hansen B, Baltaxe-Admony LB, Kurniawan S, Forbes AG (2019) Exploring sonic parameter mapping for network data structures. In: Proc. of International Conference on Auditory Display, pp 67–74

  • Harada S, Wobbrock JO, Landay JA (2011) Voice games: investigation into the use of non-speech voice input for making computer games more accessible. In: Proc. of IFIP International Conference on Human Computer Interaction, pp 11–29

  • Harding C, Kakadiaris IA, Casey JF, Loftin RB (2002) A multi-sensory system for the investigation of geoscientific data. Elsevier Comput Graph 26(2):259–269

    Article  MATH  Google Scholar 

  • Harrison L, Yang F, Franconeri S, Chang R (2014) Ranking visualizations of correlation using weber’s law. IEEE Trans Vis Comput Graph 20(12):1943–1952

    Article  Google Scholar 

  • Heer J, Bostock M (2010) Crowdsourcing graphical perception: Using mechanical turk to assess visualization design. In: Proc. of ACM CHI Conference on Human Factors in Computing Systems Conference on Human Factors in Computing Systems, pp 203–212

  • Hermann T, Hunt A, Neuhoff JG (2011) The sonification handbook. Logos Verlag Berlin, Germany

    Google Scholar 

  • Hogan T, Hinrichs U, Hornecker E (2017) The visual and beyond: Characterizing experiences with auditory, haptic and visual data representations. In: Proc. of ACM Conference on Designing Interactive Systems, pp 797–809

  • Itoh T, Nakabayashi A, Hagita M (2023) Multidimensional data visualization applying a variety-oriented scatterplot selection technique. J Vis 26(1):199–210

    Article  Google Scholar 

  • Janata P, Childs E (2004) Marketbuzz: Sonification of real-time financial data. In: Proc. of International Conference on Auditory Display

  • Jin Z, Cao N, Shi Y, Wu W, Wu Y (2021) EcoLens: visual analysis of ecological regions in urban contexts using traffic data. J Vis 24(2):349–364

    Article  Google Scholar 

  • Jin Z, Wang X, Cheng F, Sun C, Liu Q, Qu H (2023) ShortcutLens: A visual analytics approach for exploring shortcuts in natural language understanding dataset. IEEE Trans Vis Comput Graph, To appear

  • Khulusi R, Kusnick J, Meinecke C, Gillmann C, Focht J, Jänicke S (2020) A survey on visualizations for musical data. Comput Graph Forum 39:82–110

    Article  Google Scholar 

  • Kim HY (2013) Statistical notes for clinical researchers: assessing normal distribution (2) using skewness and kurtosis. Restor Dent Endod 38(1):52–54

    Article  Google Scholar 

  • Kim YJ, Kumaran R, Sayyad E, Milner A, Bullock T, Giesbrecht B, H¨ollerer T (2022) Investigating search among physical and virtual objects under different lighting conditions. IEEE Trans. Vis. Comput. Graph 28(11):3788–3798

  • Kim K, Billinghurst M, Bruder G, Duh HBL, Welch GF (2018) Revisiting trends in augmented reality research: a review of the 2nd decade of ISMAR (2008–2017). IEEE Trans. Vis. Comput. Graph 24(11):2947–2962

    Article  Google Scholar 

  • Knoeferle KM, Knoeferle P, Velasco C, Spence C (2016) Multisensory brand search: how the meaning of sounds guides consumers’ visual attention. J. Exp. Psychol 22(2):196

    Google Scholar 

  • Kong HK, Zhu W, Liu Z, Karahalios K (2019) Understanding visual cues in visualizations accompanied by audio narrations. In: Proc. of ACM CHI Conference on Human Factors in Computing Systems, pp 1–13

  • Krygier JB (1994) Sound and geographic visualization. Modern Cartography Series 2:149–166

  • Kwok TC, Kiefer P, Schinazi VR, Adams B, Raubal M (2019) Gaze-guided narratives: adapting audio guide content to gaze in virtual and real environments. In: Proc. of ACM CHI Conference on Human Factors in Computing Systems, pp 1–12

  • Lan J, Wang J, Shu X, Zhou Z, Zhang H, Wu Y (2022) RallyComparator: visual comparison of the multivariate and spatial stroke sequence in table tennis rally. J Vis 25(1):1–16

    Google Scholar 

  • Lee Y, Lee CH, Cho JD (2021) 3d sound coding color for the visually impaired. Electronics 10(9):1037

    Article  Google Scholar 

  • Limberger D, Scheibel W, D¨ollner J, Trapp M (2023) Visual variables and configuration of software maps. J Vis 26(1):249–274

    Google Scholar 

  • Lipscomb SD, Kim EM (2004) Perceived match between visual parameters and auditory correlates. In: Proc. of International Conference on Music Perception and Cognition, pp 72–75

  • Mackinlay J (1986) Automating the design of graphical presentations of relational information. ACM Trans Graph 5(2):110–141

    Article  Google Scholar 

  • Mansoor H, Gerych W, Alajaji A, Buquicchio L, Chandrasekaran K, Agu E, Rundensteiner E, Rodriguez AI (2023) INPHOVIS: Interactive visual analytics for smartphone-based digital phenotyping. Vis Inform, To appear

  • McCormack J, Roberts JC, Bach B, Freitas CDS, Itoh T, Hurter C, Marriott K (2018) Multisensory immersive analytics. In: Immersive analytics, Springer, pp 57–94

  • Metatla O, Correia NN, Martin F, Bryan-Kinns N, Stockman T (2016) Tap the ShapeTones: Exploring the effects of crossmodal congruence in an audio-visual interface. In: Proc. of ACM CHI Conference on Human Factors in Computing Systems, pp 1055–1066

  • Munzner T (2014) Visualization analysis and design. CRC Press, Boca Raton, FL

    Book  Google Scholar 

  • Ness RS, Reimer P, Krell N, Odowichuck G, Schloss WA, Tzanetakis G (2010) Sonophenology: a tangible interface for sonification of geo-spatial phenological data at multiple time-scales. In: Proc. of International Conference on Auditory Display, pp 335–341

  • Nees MA, Walker BN (2011) Auditory displays for in-vehicle technologies. Rev Hum Factors Ergon 7(1):58–99

    Article  Google Scholar 

  • Nesbitt KV, Barrass S (2002) Evaluation of a multimodal sonification and visualization of depth of market stock data. In: Proc. of International Conference on Auditory Display, pp 1–6

  • Neuhoff JG, Wayand J, Kramer G (2002) Pitch and loudness interact in auditory displays: Can the data get lost in the map? J Exp Psychol Appl 8(1):17–25

    Google Scholar 

  • Ning H, Zheng X, Yuan Y, Lu X (2021) Audio description from image by modal translation network. Neurocomputing 423:124–134

    Article  Google Scholar 

  • Ondov B, Jardine N, Elmqvist N, Franconeri S (2019) Face to face: evaluating visual comparison. IEEE Trans Vis Comput Graph 25(1):861–871

    Article  Google Scholar 

  • Papachristodoulou P, Betella A, Manzolli J (2015) Augmenting the navigation of complex data sets using sonification: A case study with brainx 3. In: Proc. of IEEE VR Workshop: Sonic Interaction in Virtual Environments, pp 1–6

  • Parise C, Spence C (2013) Audiovisual cross-modal correspondences in the general population. The Oxford handbook of synaesthesia 790:815

  • Ren Z, Yeh H, Klatzky R, Lin MC (2013) Auditory perception of geometry-invariant material properties. IEEE Trans Vis Comput Graph 19(4):557–566

    Article  Google Scholar 

  • Rind A, Iber M, Aigner W (2018) Bridging the gap between sonification and visualization. In: Proc. of AVI Workshop on Multimodal Interaction for Data Visualization

  • Rogińska A, Friedman K, Mohanraj H (2013) Exploring sonification for augmenting brain scan data. In: Proc. of International Conference on Auditory Display, pp 95–105

  • Rönnberg N (2019) Musical sonification supports visual discrimination of color intensity. Behav Inform Technol 38(10):1028–1037

    Article  Google Scholar 

  • Roodaki H, Navab N, Eslami A, Stapleton C, Navab N (2017) Sonifeye: Sonification of visual information using physical modeling sound synthesis. IEEE Trans Vis Comput Graph 23(11):2366–2371

    Article  Google Scholar 

  • Rouben A, Terveen L (2007) Speech and non-speech audio: Navigational information and cognitive load. In: Proc. of International Conference on Auditory Display, pp 468–475

  • Rubab S, Tang J, Wu Y (2021) Examining interaction techniques in data visualization authoring tools from the perspective of goals and human cognition: a survey. J Vis 24(2):397–418

    Article  Google Scholar 

  • Saket B, Endert A, Demiralp C (2019) Task-based effectiveness of basic visualizations. IEEE Trans Vis Comput Graph 25(7):2505–2512

    Article  Google Scholar 

  • Sanabria D, Soto-Faraco S, Spence C (2004) Exploring the role of visual perceptual grouping on the audiovisual integration of motion. Neuroreport 15(18):2745–2749

    Google Scholar 

  • Sawe N, Chafe C, Treviño J (2020) Using data sonification to overcome science literacy, numeracy, and visualization barriers in science communication. Front comm 5:46

    Article  Google Scholar 

  • Sawilowsky SS (2009) New effect size rules of thumb. J Mod Appl Stat Methods 8(2):597–599

    Article  MathSciNet  Google Scholar 

  • Schito J, Fabrikant SI (2018) Exploring maps by sounds: using parameter mapping sonification to make digital elevation models audible. Int J Geogr Inf Sci 32(5):874–906

    Article  Google Scholar 

  • Shenkar O, Weiss PL, Algom D (2005) Auditory representation of visual stimuli: Mapping versus association. In: Proc. of International Conference on Auditory Display, pp 273–275

  • Skau D, Kosara R (2016) Arcs, angles, or areas: individual data encodings in pie and donut charts. Comput Graph Forum 35(3):121–130

    Article  Google Scholar 

  • Smith DR, Walker BN (2005) Effects of auditory context cues and training on performance of a point estimation sonification task. Appl Cogn Psychol 19(8):1065–1087

    Article  Google Scholar 

  • Spence C (2007) Audiovisual multisensory integration. Acoust Sci Technol 28(2):61–70

    Article  Google Scholar 

  • Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73(4):971–995

    Article  Google Scholar 

  • Spence C (2020) Simple and complex crossmodal correspondences involving audition. Acoust Sci Technol 41(1):6–12

    Article  Google Scholar 

  • Su C, Yang C, Chen Y, Wang F, Wang F, Wu Y, Zhang X (2021) Natural multimodal interaction in immersive flow visualization. Vis Inform 5(4):56–66

    Article  Google Scholar 

  • Sun X, Li X, Ji L, Han F, Wang H, Liu Y, Chen Y, Lou Z, Li Z (2018) An extended research of crossmodal correspondence between color and sound in psychology and cognitive ergonomics. PeerJ 6:e4443

    Article  Google Scholar 

  • Tang T, Rubab S, Lai J, Cui W, Yu L, Wu Y (2019) iStoryline: effective convergence to hand-drawn storylines. IEEE Trans Vis Comput Graph 25(1):769–778

    Article  Google Scholar 

  • Tang Z, Bryan NJ, Li D, Langlois TR, Manocha D (2020) Scene-aware audio rendering via deep acoustic analysis. IEEE Trans Vis Comput Graph 26(5):1991–2001

    Article  Google Scholar 

  • Tsiros A (2014) Evaluating the perceived similarity between audio-visual features using corpus-based concatenative synthesis. In: Proc. of International Conference on New Interfaces for Musical Expression, pp 421–426

  • Tsuchiya T, Freeman J, Lerner LW (2016) Data-driven live coding with datatomusic api

  • Turnage KD, Bonebright TL, Buhman DC, Flowers JH (1996) The effects of task demands on the equivalence of visual and auditory representations of periodic numerical data. Behav res meth instrum comput 28(2):270–274

    Article  Google Scholar 

  • Wang J, Cai X, Su J, Liao Y, Wu Y (2022a) What makes a scatterplot hard to comprehend: data size and pattern salience matter. J Vis 25(1):59–75

  • Wang L, Sun G, Wang Y, Ma J, Zhao X, Liang R (2022b) AFExplorer: Visual analysis and interactive selection of audio features. Vis Inform 6(1):47–55

    Article  Google Scholar 

  • Walker BN (2007) Consistency of magnitude estimations with conceptual data dimensions used for sonification. Appl Cogn Psychol 21(5):579–599

    Article  Google Scholar 

  • Wan CH, Chuang SP, Lee HY (2019) Towards audio to scene image synthesis using generative adversarial network. In: Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing, pp 496–500

  • Wang Y, Chen X, Ge T, Bao C, Sedlmair M, Fu CW, Deussen O, Chen B (2019) Optimizing color assignment for perception of class separability in multiclass scatterplots. IEEE Trans Vis Comput Graph 25(1):820–829

    Article  Google Scholar 

  • Wen X, Wang M, Richardt C, Chen ZY, Hu SM (2020) Photorealistic audio-driven video portraits. IEEE Trans Vis Comput Graph 26(12):3457–3466

    Article  Google Scholar 

  • Werner L, Fay RR, Popper AN (2011) Human auditory development, vol 42. Springer, Newyork

    Google Scholar 

  • Wersényi G, Nagy H, Csapó A (2015) Evaluation of reaction times to sound stimuli on mobile devices. In: Proc. of International Conference on Auditory Display, pp 268–272

  • Wilson SR (1982) Sound and exploratory data analysis. In: COMPSTAT symposium, Springer, pp 447–450

  • Xia H, Araujo B, Grossman T, Wigdor D (2016) Object-oriented drawing. In: Proc. of ACM CHI Conference on Human Factors in Computing Systems, pp 4610–4621

  • Yang J, Hermann T (2018) Interactive mode explorer sonification enhances exploratory cluster analysis. AES: J Audio Eng Soc 66(9):703–711

    Article  Google Scholar 

  • Yeung ES (1980) Pattern recognition by audio representation of multivariate analytical data. Anal Chem 52(7):1120–1123

    Article  Google Scholar 

  • Zacks J, Tversky B (1999) Bars and lines: a study of graphic communication. Memory Cogn 27(6):1073–1079

    Article  Google Scholar 

  • Zhou Y, Meng X, Wu Y, Tang T, Wang Y, Wu Y (2022) An intelligent approach to automatically discovering visual insights. J Vis, To appear

  • Zhao Y, Jiang J, Chen Y, Liu R, Yang Y, Xue X, Chen S (2022) Metaverse: Perspectives from graphics, interactions and visualization. Vis Inform 6(1):56–67

    Article  Google Scholar 

  • Ziemer T, Schultheis H (2018) A psychoacoustic auditory display for navigation. In: Proc. of International Conference on Auditory Display, pp 136–144

  • Zhao H, Plaisant C, Shneiderman B, Duraiswami R (2004) Sonification of geo-referenced data for auditory information seeking: Design principle and pilot study. In: Proc. of International Conference on Auditory Display, pp 1–8

Download references

Acknowledgements

The work was supported by NSFC (61761136020), NSFC-Zhejiang Joint Fund for the Integration of Industrialization and Information (U1609217), Zhejiang Provincial Natural Science Foundation (LR18F020001) and the 100 Talents Program of Zhejiang University. This project was also partially funded by Microsoft Research Asia.

Author information

Authors and Affiliations

  1. State Key Lab of CAD &CG, Zhejiang University, Hangzhou, China

    Sadia Rubab, Junxiu Tang & Yingcai Wu

  2. Department of Computing, Xi’an Jiaotong-Liverpool University, Suzhou, China

    Lingyun Yu

Authors
  1. Sadia Rubab
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lingyun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junxiu Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yingcai Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yingcai Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file 1 (PDF 5 kb)

Supplementary file 2 (MP4 79184 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rubab, S., Yu, L., Tang, J. et al. Exploring Effective Relationships Between Visual-Audio Channels in Data Visualization. J Vis 26, 937–956 (2023). https://doi.org/10.1007/s12650-023-00909-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-023-00909-3

Keywords

Search

Navigation