Abstract
Subjective Video Quality Assessment (VQA) is typically based on video stimuli without sound, for studying pure visual quality perception. This silent approach to VQA does not accurately represent the typical multisensory everyday use of video content. Previous studies have highlighted that audio plays a role in shaping subjective video quality because audio provides cues for viewers to interpret the visual content. The literature on subjective VQA agrees that not only the presence, but also the quality of audio influences perceived video quality, particularly in scenes with high visual complexity. Studies of the visual saliency of multimodal stimuli show that audio influences attention allocation through semantic and spatial cues, or semantic congruency. This study combines traditional subjective VQA methodology with psychophysics measures of perception such as eye-tracking and facial expression recognition. Its aim is to investigate how audio affects both video quality scores and the unconscious components of viewers’ perception during the assessment task. Findings show (i) lower levels of visual quality scores and engagement for silent video, (ii) the influence of audio impairment on quality scores and visual attention spatial allocation, (iii) the influence of audio characteristic on visual attention, emotions, and engagement, but not on quality scores.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
ITU-R BT: Methodologies for the subjective assessment of the quality of television images, document recommendation ITU-R BT. 500-14 (10/2019). ITU, Geneva (2020)
Akhtar, Z., Falk, T.H.: Audio-visual multimedia quality assessment: a comprehensive survey. IEEE Access 5, 21090–21117 (2017)
Cisco, U.: Cisco annual internet report (2018–2023) white paper 10(1), 1–35 (2020). Cisco, San Jose
Klein, R.M.: Perceptual-motor expectancies interact with covert visual orienting under conditions of endogenous but not exogenous control. Can. J. Exp. Psychol./Revue Canadienne de psychologie expérimentale 48(2), 167 (1994)
Driver, J., Spence, C.: Attention and the crossmodal construction of space. Trends Cogn. Sci. 2(7), 254–262 (1998)
Van der Burg, E., Talsma, D., Olivers, C.N., Hickey, C., Theeuwes, J.: Early multisensory interactions affect the competition among multiple visual objects. Neuroimage 55(3), 1208–1218 (2011)
Chen, Y., Nguyen, T.V., Kankanhalli, M., Yuan, J., Yan, S., Wang, M.: Audio matters in visual attention. IEEE Trans. Circ. Syst. Video Technol. 24(11), 1992–2003 (2014)
Vroomen, J., Gelder, B.D.: Sound enhances visual perception: cross-modal effects of auditory organization on vision. J. Exp. Psychol. Hum. Percept. Perform. 26(5), 1583 (2000)
Lee, J.S., De Simone, F., Ebrahimi, T.: Influence of audio-visual attention on perceived quality of standard definition multimedia content. In: 2009 International Workshop on Quality of Multimedia Experience, pp. 13–18. IEEE (2009)
Becerra Martinez, H., Hines, A., Farias, M.C.: Perceptual quality of audio-visual content with common video and audio degradations. Appl. Sci. 11(13), 5813 (2021)
Beerends, J.G., De Caluwe, F.E.: The influence of video quality on perceived audio quality and vice versa. J. Audio Eng. Soc. 47(5), 355–362 (1999)
Min, X., Zhai, G., Gao, Z., Hu, C., Yang, X.: Sound influences visual attention discriminately in videos. In: 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX), pp. 153–158. IEEE (2014)
Ansani, A., Marini, M., D’Errico, F., Poggi, I.: How soundtracks shape what we see: analyzing the influence of music on visual scenes through self-assessment, eye tracking, and pupillometry. Front. Psychol. 11, 2242 (2020)
Cunningham, S., McGregor, I.: Subjective evaluation of music compressed with the ACER codec compared to AAC, MP3, and uncompressed PCM. Int. J. Digit. Multimed. Broadcast. 2019, 1–17 (2019)
Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978), pp. 1125–1134
Mirkovic, M., Vrgovic, P., Culibrk, D., Stefanovic, D., Anderla, A.: Evaluating the role of content in subjective video quality assessment. Sci. World J. 2014, 1–9 (2014)
Msakni, H.G., Youssef, H.: Impact of user emotion and video content on video Quality of Experience. In: Proceedings 5th ISCA/DEGA Workshop on Perceptual Quality of Systems (PQS 2016), pp. 97–101 (2016)
Schmidt, S., Zadtootaghaj, S., Wang, S., Möller, S.: Towards the influence of audio quality on gaming quality of experience. In: 2021 13th International Conference on Quality of Multimedia Experience (QoMEX), pp. 169–174. IEEE (2021)
Cheek, J.M., Smith, L.R.: Music training and mathematics achievement. Adolescence 34(136), 759–761 (1999). https://doi.org/10.1177/105971239900700311
Zhang, G., Wang, W., Qu, J., Li, H., Song, X., Wang, Q.: Perceptual influence of auditory pitch on motion speed. J. Vis. 21(10), 11 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mele, M.L., Millar, D., Colabrese, S. (2023). The Role of Audio in Visual Perception of Quality. In: Kurosu, M., et al. HCI International 2023 – Late Breaking Papers. HCII 2023. Lecture Notes in Computer Science, vol 14054. Springer, Cham. https://doi.org/10.1007/978-3-031-48038-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-48038-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-48037-9
Online ISBN: 978-3-031-48038-6
eBook Packages: Computer ScienceComputer Science (R0)