skip to main content
10.1145/3573381.3596470acmconferencesArticle/Chapter ViewAbstractPublication PagesimxConference Proceedingsconference-collections
Work in Progress

Subjective Test Environments: A Multifaceted Examination of Their Impact on Test Results

Published: 29 August 2023 Publication History

Abstract

Quality of Experience (QoE) in video streaming scenarios is significantly affected by the viewing environment and display device. Understanding and measuring the impact of these settings on QoE can help develop viewing environment-aware metrics and improve the efficiency of video streaming services. In this ongoing work, we conducted a subjective study in both laboratory and home settings using the same content and design to measure QoE in Degradation Category Rating (DCR). We first analyzed subject inconsistency and confidence intervals of the Mean Opinion Scores (MOS) between the two settings. We then used statistical models such as ANOVA and t-test to analyze the differences in subjective tests on video quality between the two viewing environments. Additionally, we employed the Eliminated-By-Aspects (EBA) model to quantify the influence of different settings on the measured QoE. We conclude with several research questions that could be further explored to better understand the impact of the viewing environment on QoE.

References

[1]
Ronald Aylmer Fisher. 1992. Statistical methods for research workers. Springer.
[2]
ITU. 2008. Subjective video quality assessment methods for multimedia applications.
[3]
ITU-R. 2019. Methodology for the Subjective Assessment of the Quality of Television Pictures. ITU-R Recommendation BT.500-14.
[4]
ITU-R. 2021. Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment. ITU-R Recommendation Recommendation P.913.
[5]
ITU-R. 2022. Subjective video quality assessment methods for multimedia applications. ITU-R Recommendation Recommendation P.910.
[6]
ITU-R BT.1769. 2006. Parameter values for an expanded hierarchy of LSDI image formats for production and international programme exchange. Int’l Telecommunication Union (2006).
[7]
ITU-R BT.2013-1. 2013. A reference viewing environment for evaluation of HDTV program material or completed programmes. Int’l Telecommunication Union (2013).
[8]
Satu Jumisko-Pyykkö and Miska M. Hannuksela. 2008. Does Context Matter in Quality Evaluation of Mobile Television?(MobileHCI ’08). Association for Computing Machinery, New York, NY, USA, 63–72. https://doi.org/10.1145/1409240.1409248
[9]
Jing Li, Lukáš Krasula, Patrick Le Callet, Zhi Li, and Yoann Baveye. 2018. Quantifying the Influence of Devices on Quality of Experience for Video Streaming. In 2018 Picture Coding Symposium (PCS). 308–312. https://doi.org/10.1109/PCS.2018.8456304
[10]
Suiyi Ling, Yoann Baveye, Deepthi Nandakumar, Sriram Sethuraman, and Patrick Le Callet. 2020. Towards better quality assessment of high-quality videos. In Proceedings of the 1st Workshop on Quality of Experience (QoE) in Visual Multimedia Applications. 3–9.
[11]
Margaret H. Pinson, Marcus Barkowsky, and Patrick Le Callet. 2013. Selecting scenes for 2D and 3D subjective video quality tests. EURASIP Journal on Image and Video Processing 2013, 1 (Aug. 2013), 50–61.
[12]
Amos Tversky. 1972. Elimination by Aspects: A Theory of Choice. Psychological Review 79, 4 (1972), 281–299. https://doi.org/10.1037/h0032955
[13]
Haiqiang Wang, Ioannis Katsavounidis, Jiantong Zhou, Jeonghoon Park, Shawmin Lei, Xin Zhou, Man-On Pun, Xin Jin, Ronggang Wang, Xu Wang, 2017. VideoSet: A large-scale compressed video quality dataset based on JND measurement. Journal of Visual Communication and Image Representation 46 (2017), 292–302.
[14]
Florian Wickelmaier and Christian Schmid. 2004. A Matlab function to estimate choice model parameters from paired-comparison data. Behavior Research Methods, Instruments, & Computers 36 (2004), 29–40.
[15]
S William. 1908. The probable error of a mean. Biometrika 6, 1 (1908), 1–25.
[16]
Jingwen Zhu, Ali Ak, Patrick Le Callet, Sriram Sethuraman, and Kumar Rahul. 2023. ZREC : robust recovery of mean and percentile opinion scores. (March 2023). https://hal.science/hal-04017583 working paper or preprint.
[17]
Jingwen Zhu, Suiyi Ling, Yoann Baveye, and Patrick Le Callet. 2022. A framework to map vmaf with the probability of just noticeable difference between video encoding recipes. In 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP). IEEE, 1–5.

Index Terms

  1. Subjective Test Environments: A Multifaceted Examination of Their Impact on Test Results

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      IMX '23: Proceedings of the 2023 ACM International Conference on Interactive Media Experiences
      June 2023
      465 pages
      ISBN:9798400700286
      DOI:10.1145/3573381
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 29 August 2023

      Check for updates

      Author Tags

      1. HD
      2. Subjective test
      3. Video Quality Assessment

      Qualifiers

      • Work in progress
      • Research
      • Refereed limited

      Conference

      IMX '23

      Acceptance Rates

      Overall Acceptance Rate 69 of 245 submissions, 28%

      Upcoming Conference

      IMX '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 39
        Total Downloads
      • Downloads (Last 12 months)12
      • Downloads (Last 6 weeks)4
      Reflects downloads up to 17 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media