Abstract
Subtitles are present in almost all TV programs and films in Taiwan. Are Taiwanese more dependent on subtitles to appreciate the content of the film compared to people of other nationality? What happens if subtitles are removed or replaced by unfamiliar languages? In this research, we use Tobii EyeX to collect eye movement data from 45 native-speakers while they watch different films, and propose appropriate indicators to analyze their viewing behavior. To facilitate subsequent data analysis, certain areas of interest (AOI), such as the caption region and human face, are automatically detected using techniques including Canny edge detector and Faster R-CNN.
Experimental results indicate that auditory language is the most critical factor. Subjects in Group #1 (English, Chinese, English and Chinese) have a higher tendency to focus on the face area. Subjects in Group #2 (Chinese, English, Chinese and English) appear to read the subtitles more often. The initial behavior seems to determine the viewing pattern subsequently. For subjects in Group #2, preference for caption is clearly observed than those in Group #1. This habitual preference continues in follow-up movies, resulting in an immersion phenomenon. We also observe that when unfamiliar texts appear, the subjects exhibit ‘escaping’ behavior by avoiding the text region. It is worth noting that the video at the beginning of Group #2 is the native language of the testee, and the result demonstrates that the subject develops preferences toward viewing subtitles. Therefore, we can partially confirm that Taiwanese people have a certain degree of dependence on subtitles.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
National Communications Commission. Radio and Television Act (2020). Accessed 1 Dec 2020. https://law.moj.gov.tw/Eng/LawClass/LawAll.aspx?PCode=P0050001
Maddieson, I.: Tone. In: Dryer, M.S., Haspelmath, M. (eds.) The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig (2013)
Peng, C.-F., Liao, W.-H.: Evaluation of interactive data visualization tools based on gaze and mouse tracking. In: 2016 IEEE International Symposium on Multimedia (ISM), pp. 431–434. IEEE (2016)
Liao, W.-H., Chang, C.-W., Wu, Y.-C.: Classification of reading patterns based on gaze information. In: 2017 IEEE International Symposium on Multimedia (ISM), pp. 595–600. IEEE (2017)
Tobii Technology AB. An introduction to Tobii EyeX (2020). Accessed 1 Dec 2020. https://developer.tobii.com/an-introduction-to-the-tobii-eyex-sdk/
O’Bryan, K.G.: Eye movements as an index of television viewing strategies (1975)
Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 6, 679–698 (1986)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
LLC. TED Conferences. TED: Ideas worth spreading (2020). Accessed 1 Dec 2020. https://www.ted.com/
Chi, T.: Rapid prototyping Google Glass (2013). Accessed 1 Dec 2020. https://ed.ted.com/lessons/rapid-prototyping-google-glass-tom-chi
Dekofsky, J.: Is math discovered or invented? (2014). Accessed 1 Dec 2020. https://www.ted.com/talks/jeff_dekofsky_is_math_discovered_or_invented
Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)
Ordikhani-Seyedlar, M.: What happens in your brain when you pay attention? (2017). Accessed 1 Dec 2020. https://www.ted.com/talks/mehdi_ordikhani_seyedlar_what_happens_in_your_brain_when_you_pay_attention/
Cofer, A.: A plan to recycle the unrecyclable (2016). Accessed 1 Dec 2020. https://www.ted.com/talks/ashton_cofer_a_plan_to_recycle_the_unrecyclable
Harada, C.: How I teach kids to love science? (2015). Accessed 1 Dec 2020. https://www.ted.com/talks/cesar_harada_how_i_teach_kids_to_love_science
Bu, L.: How books can open your mind? (2013). Accessed 1 Dec 2020. https://www.ted.com/talks/lisa_bu_how_books_can_open_your_mind
Peleg, D.: Forget shopping. Soon you’ll download your new clothes (2015). Accessed 1 Dec 2020. https://www.ted.com/talks/danit_peleg_forget_shopping_soon_you_ll_download_your_new_clothes
Mullins, A.: My 12 pairs of legs (2009). Accessed 1 Dec 2020. https://www.ted.com/talks/aimee_mullins_my_12_pairs_of_legs
Harvey, D.: Protecting twitter users (sometimes from themselves) (2014). Accessed 1 Dec 2020. https://www.ted.com/talks/del_harvey_protecting_twitter_users_sometimes_from_themselves
OGAMA. Open gaze and mouse analyzer (2016). Accessed 4 Feb 2016. http://www.ogama.net/sites/default/files/pdf/OGAMA-DescriptionV25.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Liao, WH., Chen, CJ., Wu, YC. (2021). Investigating Viewer’s Reliance on Captions Based on Gaze Information. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2021. Lecture Notes in Computer Science(), vol 12797. Springer, Cham. https://doi.org/10.1007/978-3-030-77772-2_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-77772-2_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77771-5
Online ISBN: 978-3-030-77772-2
eBook Packages: Computer ScienceComputer Science (R0)