Abstract
This study employed an eye tracking technology to explore university students’ visual attention and learning performance while learning Japanese using an interactive manga-based e-book. The developed e-book consisted of 8 pages accompanied by 13 annotations with both text and graphical formats. The subjects consisted of 60 students whose eye movements were tracked and recorded by the eye tracking system. These students came from the applied foreign language department in a northern university in Taiwan, of which 30 were assigned to high prior knowledge (PK) group and the other 30 were assigned to low PK group. Eye tracking measurements, including total contact time, number of fixations, latency of first fixation, and number of clicks on the defined regions of interest of the two groups were compared to indicate their visual attention. The results revealed that overall students spent more time on reading text and annotation than graphic information. The high PK students showed longer fixation durations on the texts, while the low PK students showed longer fixation durations on the graphics and annotations. Meanwhile, the low PK students used more clicks to look up underlined annotations whenever they didn’t know words or phrases on the e-book. In addition, with respect to the latency of the first fixation, the graphic captured the attention faster than the text because of the size and its appeal to the students. Further analysis of saccade paths indicated that the low PK students showed more inter-scanning transitions not only between the text dialog and the annotation zone but also within annotation zone. Finally, the results of reading comprehension pretest and posttest found that there was a significant difference in learning outcomes between each PK group.
Similar content being viewed by others
References
Adams J (2001) A critical study of comics. The International Journal of Art & Design Education 20(2):133–143
Alexander PA, Jetton TL (2000) Learning from text: a multidimensional and developmental perspective. In: Kamil ML, Mosenthal PB, Pearson PD, Barr R (eds) Handbook of reading research, vol 3. Erlbaum, Mahwah, pp 285–310
Black RW (2005) Access and affiliation: the literacy and composition practices of English-language learners in an on-line fanfiction community. Journal of Adolescent & Adult Literacy 49(2):118–128
Bryce M, Davis J, Barber C (2008) The cultural biographies and social lives of manga: lessons from the mangaverse. Journal of Media Arts Culture Collection 5(2). Retrieved from) http://scan.net.au/scan/journal/display.php?journal_id=114
Calvo MG, Lang PJ (2004) Gaze patterns when looking at emotional pictures: motivationally biased attention. Motiv Emot 28(3):221–243
Canham M, Hegarty M (2010) Effects of knowledge and display design on comprehension of complex graphics. Learn Instr 20(2):155–166
Cary S (2004) Going graphic: Comics at work in the multilingual classroom. Heinemann, Portsmouth
Chen YC, Hwang RH, Wang CY (2012) Development and evaluation of a web 2.0 annotation system as a learning tool in an e-learning environment. Comput Educ 58(4):1094–1105
Chi MTH, VanLehn KA (2012) Seeing deep structure from the interactions of surface features. Educ Psychol 47(3):177–188
Chou IC (2014) Investigating EFL students’ E-book reading attitudes in first and second language. US-China Foreign. Language 12(1):64–74
Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, New Jersey
Coiro J (2011) Predicting reading comprehension on the internet: contributions of offline reading skills, online reading skills, and prior knowledge. J Lit Res 43(4):352–392
Cook M, Carter GN, Wiebe E (2008) The interpretation of cellular transport graphics by students with low and high prior knowledge. Int J Sci Educ 30(2):241–263
Doty DE, Popplewell SR, Byers GO (2001) Interactive CD-ROM storybooks young reader’s reading comprehension. J Res Comput Educ 3(4):374–384
Epelboim J, Suppes P (1997) Eye movements during geometrical problem solving. Paper presented at the 19th Annual Conference of the Cognitive Science Society
Grant ER, Spivey MJ (2003) Eye movements and problem solving guiding attention guides thought. Psychol Sci 14(5):462–466
Greene JA, Costa LJ, Robertson J, Pan Y, Deekens VM (2010) Exploring relations among college students’ prior knowledge, implicit theories of intelligence, and self-regulated learning in a hypermedia environment. Comput Educ 55:1027–1043
Hegarty M, Mayer RE, Green C (1992) Comprehension of arithmetic word problems: evidence from students’ eye fixations. J Educ Psychol 84(1):76–84
Hegarty M, Mayer RE, Monk CA (1995) Comprehension of arithmetic word problems: a comparison of successful and unsuccessful problem solvers. J Educ Psychol 87(1):18–32
Hewig J, Trippe RH, Hecht H, Straube T, Miltner WHR (2008) Gender differences for specific body regions when looking at men and women. J Nonverbal Behav 32(2):67–78
Higgins NC, Cocks P (1999) The effects of animation cues on vocabulary development. Journal of Reading Psychology 20:1–10
Ho HN, Tsai MJ, Wang CY, Tsai CC (2014) Prior knowledge and online inquiry-based science reading: evidence from eye tracking. Int J Sci Math Educ 12:525–554
Huang TH (2014) A study on the effects of interactive e-books on Taiwan high/vocational school students’ reading comprehension. Social Science Research Network (SSRN), Retrieved from SSRN https://ssrn.com/abstract=2462830
Humphrey K, Underwood G (2009) Domain knowledge moderates the influence of visual saliency in scene recognition. Br J Psychol 100(2):377–398
Hwang WY, Hsu GL (2011) The effects of pre-reading and haring mechanisms on learning with the use of annotations. Turkish Online Journal of Educational Technology 10(2):234–249
Hwang WY, Wang CY, Sharples M (2007) A study of multimedia annotation of web-based materials. Comput Educ 48(4):680–699
Hwang WY, Chen NS, Shadiev R, Li JS (2011a) Effects of reviewing annotations and homework solutions on math learning achievement. Br J Educ Technol 42(6):1016–1028
Hwang WY, Shadiev R, Huang SM (2011b) A study of a multimedia web annotation system and its effect on the EFL writing and speaking performance of junior high school students. ReCALL 23:160–180
Hwang WY, Liu YF, Chen HR, Huang JW, Li JY (2015) Role of parents and annotation sharing in children’s learning behavior and achievement using E-readers. Educational Technology & Society 18(1):292–307
Hyönä, J. Jr., Lorch, R.F., & Kaakinen, J.K. (2002). Individual differences in reading to summarize expository text: evidence from eye fixation patterns. J Educ Psychol, 94, 44–55
Jarodzka H, Scheiter K, Gerjets P, van Gog T (2010) In the eyes of the beholder: how experts and novices interpret dynamic stimuli. Learn Instr 20(2):146–154
Jarodzka H, Holmqvist K, Gruber H (2017) Eye tracking in educational science: theoretical frameworks and research agendas. J Eye Mov Res 10(1):1–18
Just MA, Carpenter PA (1980) A theory of reading: from eye fixations to comprehension. Psychol Rev 87:329–354
Just MA, Carpenter PA (1985) Cognitive coordinate systems: Accounts of mental rotation and individual differences in spatial ability. Psychol Review 92(2):137-172
Khurana S (2005) So you want to be a superhero? How the art of making comics in an afterschool setting can develop young people‘s creativity, literacy and identity. Afterschool Matters 4:1–9
Knoblich G, Ohlsson S, Raney GE (2001) An eye movement study of insight problem solving. Mem Cogn 29(7):1000–1009
Korat O, Shamir A (2004) Do Hebrew electronic books differs from Dutch electronic books? A replication of a Dutch content analysis. J Comput Assist Learn 20(4):257–268
Kristof R, Satran A (1995) Interactivity by design: creating & communicating with new media. Adobe Press, Mountain View
Lai ML, Tsai MJ, Yang FY, Hsu CY, Liu TC, Lee SWY, Lee MH, Chiou GL, Liang JC, Tsai CC (2013) A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review 10:90–115
Lin IY (2009) The effect of E-books on EFL learners’ reading attitude. Master thesis, National Taiwan Normal University, Taiwan. Retrieved from http://ndltd.ncl.edu.tw/cgi-bin/gs32/gsweb.cgi?o=dnclcde&s=id=%22097NTNU5238009%22.&searchmode=basic
Lin YT, Wu CC, Hou TY, Lin YC, Yang FY, Chang CH (2016) Tracking students’ cognitive processes during program debugging-an eye-movement approach. IEEE Trans Educ 59(3):175–186
Liu HC, Andre T, Greenbowe T (2008) The impact of learner’s prior knowledge on their use of chemistry computer simulations: a case study. J Sci Educ Technol 17:466–482
Liversedge S, Paterson K, Pickering M (1998) Eye movements and measures of reading time. In: Underwood G (ed) Eye guidance in reading and scene perception. Elsevier, Oxford, pp 55–75
Mason L, Pluchino P, Tornatora MC, Ariasi N (2013) An eye-tracking study of learning from science text with concrete and abstract illustrations. J Exp Educ 81(3):356–384
Mayer RE (1997) Multimedia learning: are we asking the right question? Educ Psychol 32(1):1–19
Mayer RE (2005) The Cambridge handbook of multimedia learning. Cambridge University Press, New York
Mayer RE (2008) Applying the science of learning: evidence-based principles for the design of multimedia instruction. Am Psychol 63(8):760–769
Mayer RE (2009) Multimedia learning, 2nd edn. Cambridge University Press, New York
Mayer RE (2011) Applying the science of learning. Pearson, Upper Saddle River
Mayer RE, Sims VK (1994) For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. J Educ Psychol 86:389–401
Mintzes J, Wandersee JH, Novak JD (eds) (2000) Assessing science understanding (pp. 304–341). Academic Press, San Diego
Moist KM, Bartholomew M (2007) When pigs fly: anime, auteurism, and Miyazaki’s Porco Rosso. SAGE Publications, London
Murakami S, Bryce M (2009) Manga as an educational medium. The International Journal of the Humanities 7(10):45–55
Ogawa E (2013) Educational manga: an effective medium for english learning. In: Sonda N, Krause A (eds) Paper presented at JALT2012 conference proceedings. Tokyo, JALT
Paivio A (1986) Mental representations: a dual coding approach. Oxford University Press, Oxford
Paivio A (2006) Mind and its evolution: A dual coding theoretical interpretation. Lawrence Erlbaum Associates, Inc, Mahwah
Pan TW, Hsu MC, Tsai MJ (2013) Effect of graphic design on E-book reading: a pilot eye-tracking study. In: Tan SC et al (eds) Paper presented at the workshop proceedings of the 21st international conference on computers in education, Indonesia
Pass F, van Merriënboer JJG (1994) Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. J Educ Psychol 86(1):122-133
Paas F, Renkl A, Sweller J (2003) Cognitive load theory and instructional design: recent developments. Educ Psychol 38:1-4
Peverly ST, Vekaria PC, Reddington LA, Sumowski JF, Johnson KR, Ramsay CM (2013) The relationship of handwriting speed, working memory, language comprehension and outlines to lecture note-taking and test-taking among college students. Appl Cogn Psychol 27(1):115–126
Plass JL, Moreno R, Brunken R (eds) (2010) Cognitive load theory. Cambridge University Press, New York
Potter MC, Wyble B, Hagmann CE, McCourt ES (2014) Detecting meaning in RSVP at 13 ms per picture. Atten Percept Psychophys 76:270–279
Rawlins GJE (1993) Publishing over the new decade. J Am Soc Inf Sci 44(8):474–479
Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124(3):372–422
Rayner K (2009) Eye movements and attention in reading, scene perception, and visual search. Q J Exp Psychol 62(8):1457–1506
Rayner K, Rotello CM, Stewart AJ, Keir J, Duffy SA (2001) Integrating text and pictorial information: eye movements when looking at print advertisements. J Exp Psychol Appl 7:219–226
Rayner K, Smith T, Malcolm GL, Henderson JM (2009) Eye movements and visual encoding during scene perception. Psychol Sci 20(1):6–10
Rayner K, Slattery TJ, Bélanger NN (2010) Eye movements, the perceptual span, and reading speed. Psychon Bull Rev 17(6):834–839
Reingold EM, Sheridan H (2011) Eye movements and visual expertise in chess and medicine. In: Liversedge SP, Gilchrist ID, Everling S (eds) Oxford handbook on eye movements. Oxford University Press, Oxford, pp 528–550
Riley P (2003) Drawing the threads together. In: Little D, Ridley J, Ushioda E (eds) Learner autonomy in the foreign language classroom: teacher, learner, curriculum and assessment. Dublin, Authentik, pp 237–252
Sadoski M, Willson VL (2006) Effects of a theoretically-based large scale reading intervention in a multicultural urban school district. Am Educ Res J 43:137–154
Sanders MS, McCormick EJ (1987) Human factors in engineering and design. McGraw-Hill, New York
She HC, Chen YZ (2009) The impact of multimedia effect on science learning: evidence from eye movements. Comput Educ 53(4):1297–1307
Shimada A, Okubo F, Yin CJ, Oi M, Kojima K, Yamada M, Ogata H (2015) Analysis of preview behavior in E-book system. In: Ogata H et al (eds) Paper presented at the 23rd international conference on computers in education (ICCE 2015), Hangzhou, China
Smith, S.D., Caruso, J.B., & Kim, J. (2010). The ECAR study of undergraduate students and information technology. ECAR Research Study, 6, 1–120. Retrieved from http://library.educause.edu/~/media/files/library/2010/10/ers1006w-pdf.pdf
Sweller J (1988) Cognitive load during problem solving: effects on learning. Cogn Sci 12(2):257–285
Thompson RA, Zamboanga BL (2003) Prior knowledge and its relevance to student achievement in introduction to psychology. Teach Psychol 30:96–101
Tsai JL, Yen MH, Wang CA (2005) Eye movement recording and the application in research of reading Chinese. Research in Applied Psychology 28:91–104
Tsai MJ, Hou HT, Lai ML, Liu WY, Yang FY (2011) Visual attention for solving multiple-choice science problem: an eye-tracking analysis. Comput Educ 58:375–385
Tsai MJ, Huang LJ, Hou HT, Hsu CY, Chiou GL (2016) Visual behavior, flow and achievement in game-based learning. Comput Educ 98:115–129
Van Gog T, Scheiter K (2010) Eye tracking as a tool to study and enhance multimedia learning. Learn Instr 20(2):95–99
Van Gog T, Kester L, Nievelstein F, Giesbers B, Paas F (2009) Uncovering cognitive processes: different techniques that can contribute to cognitive load research and instruction. Comput Hum Behav 25(2):325–331
Wolfe, P. (2010). Brain matters: Translating research into classroom practice (2nd Ed.) Alexandria: Association for Supervision & curriculum development
Wu AH, Hsu PF, Chiu HJ, Tsai MJ (2014) Visual behavior and cognitive load on E-book vocabulary learning. In: Liu CC et al (eds) Paper presented at the workshop proceedings of the 22nd international conference on computers in education, Japan
Yang FY, Chang CY, Chien WR, Chien YT, Tseng YH (2013) Tracking learners’ visual attention during a multimedia presentation in a real classroom. Comput Educ 62:208–220
Yin CJ, Okubo F, Shimada A, Oi M, Hirokawa S, Ogata H (2015) Identifying and analyzing the learning behaviors of students using e-books. In: Ogata H et al (eds) Paper presented at the workshop proceedings of the 23rd international conference on computers in education (ICCE 2015), Hangzhou, China
Zheng HT, Wang Z, Ma N, Chen J, Xiao X, Sangaiah AK (2017) Weakly-supervised image captioning based on rich contextual information. Multimedia Tools and Applications:1–17
Acknowledgements
The author is grateful to the Department of International Cooperation and Science Education as well as the Ministry of Science and Technology of the Republic of China for their financial support to carry out this work, under Grant number: MOST 104-2511-S-149-001-. The author also wish to thank the Aim for the Top University (ATU) project of National Taiwan Normal University (NTNU), sponsored by the Ministry of Education of the Republic of China.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
No conflicts of interest have been declared.
Rights and permissions
About this article
Cite this article
Wang, CC., Hung, J.C., Chen, SN. et al. Tracking students’ visual attention on manga-based interactive e-book while reading: an eye-movement approach. Multimed Tools Appl 78, 4813–4834 (2019). https://doi.org/10.1007/s11042-018-5754-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-018-5754-6