Abstract
Metacognitive judgment of one’s online search process (calibration) is an important element of searching efficiency. This study investigated Chinese university students’ calibration during online information search. Fourteen students s were asked to search answers for three assigned tasks. Immediately after the search, all participants attended individual interviews about their search behavior. Based on eye-tracking data of the actual search process, three profiles of searchers were identified: surface explorers, deep explorers, and smooth searchers. Eye-tracking and retrospective interview data were analyzed to examine the participants’ level of calibration. Results showed that the participants were not able to calibrate well on their searching process, with a mean score of 3.64 out of 10.0 (SD = 1.91), suggesting a mediocre level of calibration capacity. A moderate level of positive relationship (r = .42, p > .05) was also found between calibration scores and search performance scores, implying a general pattern wherein the high calibration group tended to perform better than the low calibration group. The findings called for integrating metacognitive judgment as a component of improving computer information literacy which, accordingly, could be used to enhance students’ ability to think critically about seeking and evaluating information.


Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
Data will be made available upon reasonable request.
References
Abdekhoda, M., Dehnad, A., & Yousefi, M. (2016). Effectiveness of training intervention to improve medical student’s information literacy skills. Korean Journal of Medical Education, 28(4), 391–395. https://doi.org/10.3946/kjme.2016.44
Abuhamdieh, A. H., & Harder, J. T. (2015). Systematizing web search through a Meta-Cognitive, Systems-Based, Information Structuring Model (McSIS). Contemporary Issues in Education Research, 8(1), 33–44
Alexander, P. A. (2013). Calibration: What is it and why it matters? An introduction to the special issue on calibrating calibration. Learning and Instruction, 24, 1–3. https://doi.org/10.1016/j.learninstruc.2012.10.003
Avargil, S., Lavi, R., & Dori, Y. J. (2018). Students’ metacognition and metacognitive strategies in science education. In Y. J. Dori, Z. R. Mevarech & D. R. Baker (Eds.), Cognition, metacognition, and culture in STEM education (pp. 33–64). Springer. https://doi.org/10.1007/978-3-319-66659-4
Azevedo, R., Moos, D. C., Johnson, A. M., & Chauncey, A. D. (2010). Measuring cognitive and metacognitive regulatory processes during hypermedia learning: Issues and challenges. Educational Psychologist, 45(4), 210–223. https://doi.org/10.1080/00461520.2010.515934
Azzopardi, L., Thomas, P., & Craswell, N. (2018, June). Measuring the utility of search engine result pages: An information foraging based measure. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 605–614). ACM. https://doi.org/10.1145/3209978.3210027
Binali, T., Tsai, C. C., & Chang, H. Y. (2021). University students’ profiles of online learning and their relation to online metacognitive regulation and internet-specific epistemic justification. Computers & Education, 175, 104315. https://doi.org/10.1016/j.compedu.2021.104315
Birjandi, P., Mirhassani, A., & Abbasian, G. (2006). Setting-based metacognitive strategy use. Journal of Faculty of Letters and Humanities, 49(198), 39–87
Bowler, L. (2007). Methods for revealing the metacognitive knowledge of adolescent information seekers during the information search process. In Proceedings of the 36th Annual Conference of the Canadian Association for Information Science (CAIS), McGill University, Montreal, Quebec. https://doi.org/10.29173/cais206
Bowler, L. (2010). A taxonomy of adolescent metacognitive knowledge during the information search process. Library & Information Science Research, 32(1), 27–42. https://doi.org/10.1016/j.lisr.2009.09.005
Catrysse, L., Gijbels, D., Donche, V., De Maeyer, S., Lesterhuis, M., & Van den Bossche, P. (2018). How are learning strategies reflected in the eyes? Combining results from self-reports and eye-tracking. British Journal of Educational Psychology, 88(1), 118–137. https://doi.org/10.1111/bjep.12181
Chen, P., & Rossi, P. D. (2013). Utilising calibration accuracy information with adolescents to improve academic learning and performance. In H. Bembenutty, T. J. Cleary, & A. Kitsantas (Eds.), Applications of self-regulated learning across diverse disciplines (pp. 263–298). Information Age Publishing
Cho, B. Y., Afflerbach, P., & Han, H. (2018). Strategic processing in accessing, comprehending, and using multiple sources online. In J. L. G. Braasch, I. Bråten, & M. T. McCrudden (Eds.), Handbook of multiple source use (pp. 133–150). Routledge
Civilcharran, S., Hughes, M., & Maharaj, M. S. (2015). Uncovering Web search tactics in South African higher education. South African Journal of Information Management, 17(1), 1–8. https://doi.org/10.4102/sajim.v17i1.644
Connor, C. M., Day, S. L., Zargar, E., Wood, T. S., Taylor, K. S., Jones, M. R., & Hwang, J. K. (2019). Building word knowledge, learning strategies, and metacognition with the Word-Knowledge e-Book. Computers & Education, 128, 284–311. https://doi.org/10.1016/j.compedu.2018.09.016
Crescenzi, A. (2016). Metacognitive knowledge and metacognitive regulation in time-constrained in information search. In Proceedings of ACM SIGIR Conference (pp. 1–5), Pisa, Italy
Dahlen, S. P., Haeger, H., Hanson, K., & Montellano, M. (2020). Almost in the wild: Student search behaviors when librarians aren’t looking. The Journal of Academic Librarianship, 46, 102096. https://doi.org/10.1016/j.acalib.2019.102096
Davis, D., Chen, G., Jivet, I., Hauff, C., & Houben, G. J. (2016). Encouraging metacognition & self-regulation in MOOCs through increased learner feedback. In Proceedings of LAL@ LAK (pp. 17–22). Edinburgh, Scotland
Dever, D. A., & Azevedo, R. (2019, June). Examining gaze behaviors and metacognitive judgments of informational text within game-based learning environments. In Proceedings of International Conference on Artificial Intelligence in Education (pp. 121–132). Springer
DiGiacomo, G., & Chen, P. P. (2016). Enhancing self-regulatory skills through an intervention embedded in a middle school mathematics curriculum. Psychology in the Schools, 53(6), 601–616. https://doi.org/10.1002/pits.21929
Dinsmore, D. L., & Parkinson, M. M. (2013). What are confidence judgments made of? Students’ explanations for their confidence ratings and what that means for calibration. Learning and Instruction, 24, 4–14. https://doi.org/10.1016/j.learninstruc.2012.06.001
Dunlosky, J., & Thiede, K. W. (2013). Four cornerstones of calibration research: Why understanding students’ judgments can improve their achievement. Learning and Instruction, 24, 58–61. https://doi.org/10.1016/j.learninstruc.2012.05.002
Durodolu, O. O., & Ngoaketsi, J. M. (2019). Multilateral relationship between information literacy, self-concept and metacognitive ability. In N. Feza (Ed.), Metacognition in Learning (pp. 107–122). IntechOpen. https://doi.org/10.5772/intechopen.78892
Emory, B., & Luo, T. (2022). Metacognitive training and online community college students’ learning calibration and performance. Community College Journal of Research and Practice, 46(4), 240–256. https://doi.org/10.1080/10668926.2020.1841042
Evans, G. J. (2018). Windmills of your mind: Metacognition and lifelong learning. In Proceedings of the Canadian Engineering Education Association (CEEA) (pp. 1–5). Vancouver, Canada. https://doi.org/10.24908/pceea.v0i0.12985
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906
Follmer, D. J., Patchan, M., & Spitznogle, R. (2022). Supporting college learners’ study time calibration: Relations to course achievement and self-regulated learning skills. Journal of College Reading and Learning. https://doi.org/10.1080/10790195.2022.2033646. Ahead-of-print
Forzani, E. (2018). How well can students evaluate online science information? Contributions of prior knowledge, gender, socioeconomic status, and offline reading ability. Reading Research Quarterly, 53(4), 385–390. https://doi.org/10.1002/rrq.218
Greene, J. A., Copeland, D. Z., Deekens, V. M., & Yu, S. B. (2018). Beyond knowledge: Examining digital literacy’s role in the acquisition of understanding in science. Computers and Education, 117, 141–159. https://doi.org/10.1016/j.compedu.2017.10.003
Hacker, D. J. (1998). Definitions and empirical foundations. In D. J. Hacker, J. Dunlosky & A. C. Grasser (Eds.), Metacognition in educational theory and practice (pp. 1–23). Lawrence Erlbaum Associates. https://doi.org/10.4324/9781410602350
Hahnel, C., Goldhammer, F., Kröhne, U., & Naumann, J. (2018). The role of reading skills in the evaluation of online information gathered from search engine environments. Computers in Human Behavior, 78, 223–234. https://doi.org/10.1016/j.chb.2017.10.004
Händel, M., de Bruin, A. B., & Dresel, M. (2020). Individual differences in local and global metacognitive judgments. Metacognition and Learning, 15(1), 51–75. https://doi.org/10.1007/s11409-020-09220-0
Händel, M., & Dresel, M. (2018). Confidence in performance judgment accuracy: The unskilled and unaware effect revisited. Metacognition and Learning, 13(3), 265–285. https://doi.org/10.1007/s11409-018-9185-6
Hautala, J., Kiili, C., Kammerer, Y., Loberg, O., Hokkanen, S., & Leppänen, P. H. (2018). Sixth graders’ evaluation strategies when reading Internet search results: An eye-tracking study. Behaviour & Information Technology, 37(8), 761–773. https://doi.org/10.1080/0144929X.2018.1477992
Huang, K., Law, V., & Lee, S. J. (2019). The role of learners’ epistemic beliefs in an online community of inquiry. British Journal of Educational Technology, 50(4), 1882–1895. https://doi.org/10.1111/bjet.12684
Ibieta, A., Hinostroza, J. E., & Labbé, C. (2019). Improving students’ information problem-solving skills on the Web through explicit instruction and the use of customized search software. Journal of Research on Technology in Education, 51(3), 217–238. https://doi.org/10.1080/15391523.2019.1576559
Karatas, K., & Arpaci, I. (2021). The role of self-directed learning, metacognition, and 21st century skills predicting the readiness for online learning. Contemporary Educational Technology, 13(3), ep300. https://doi.org/10.30935/cedtech/10786
Koriat, A. (2007). Metacognition and consciousness. In P. Zelazo, M. Moscovitch, & E. Thompson (Eds.), The Cambridge handbook of consciousness (pp. 289–325). Cambridge University Press. https://doi.org/10.1017/CBO9780511816789
Kuhn, D., & Dean, D. (2004). Metacognition: A bridge between cognitive psychology and educational practice. Theory into Practice, 43(4), 268–273. https://doi.org/10.1207/s15430421tip4304_4
Lazonder, A. W. (2003). Principles for designing web searching instruction. Education and Information Technologies, 8(2), 179–193. https://doi.org/10.1023/A:1024510531605
Lingel, K., Lenhart, J., & Schneider, W. (2019). Metacognition in mathematics: Do different metacognitive monitoring measures make a difference? Zdm Mathematics Education, 51(4), 587–600. https://doi.org/10.1007/s11858-019-01062-8
Macias, W., Lee, M., & Cunningham, N. (2018). Inside the mind of the online health information searcher using think-aloud protocol. Health Communication, 33(12), 1482–1493. https://doi.org/10.1080/10410236.2017.1372040
Martinez, M. E. (2006). What is metacognition? Phi Delta Kappan, 87(9), 696–699. https://doi.org/10.1177/003172170608700916
Mason, L., Boldrin, A., & Ariasi, N. (2010). Searching the Web to learn about a controversial topic: Are students epistemically active? Instructional Science, 38(6), 607–633. https://doi.org/10.1007/s11251-008-9089-y
Matsuda, T., Ushigome, R., Sonoda, M., Eto, M., Satoh, H., Hanada, T. … Katoh, D. (2019). Web search skill evaluation from eye and mouse momentum. In Proceedings of International Conference on Intelligent Human Systems Integration (pp. 884–888). Springer, Cham. https://doi.org/10.1007/978-3-030-11051-2_135
Miles, M. B., Huberman, A. M., & Saldaña, J. (2018). Qualitative data analysis: A methods sourcebook. Sage
Molina, A. I., Navarro, Ó., Ortega, M., & Lacruz, M. (2018). Evaluating multimedia learning materials in primary education using eye tracking. Computer Standards & Interfaces, 59, 45–60. https://doi.org/10.1016/j.csi.2018.02.004
Mudrick, N. V., Azevedo, R., & Taub, M. (2019). Integrating metacognitive judgments and eye movements using sequential pattern mining to understand processes underlying multimedia learning. Computers in Human Behavior, 96, 223–234. https://doi.org/10.1016/j.chb.2018.06.028
Pesout, O., & Nietfeld, J. L. (2021). How creative am I?: Examining judgments and predictors of creative performance. Thinking Skills and Creativity, 40, Article 100836. https://doi.org/10.1016/j.tsc.2021.100836
Reisoğlu, İ., Çebi, A., & Bahçekapılı, T. (2019). Online information searching behaviours: Examining the impact of task complexity, information searching experience, and cognitive style. Interactive Learning Environments. Ahead-of-print. https://doi.org/10.1080/10494820.2019.1662456
Reynders, G., Lantz, J., Ruder, S. M., Stanford, C. L., & Cole, R. S. (2020). Rubrics to assess critical thinking and information processing in undergraduate STEM courses. International Journal of STEM Education, 7(1), 1–15. https://doi.org/10.1186/s40594-020-00208-5
Riemer, M., Kubik, V., & Wolbers, T. (2019). The effect of feedback on temporal error monitoring and timing behavior. Behavioural Brain Research, 369, 111929. https://doi.org/10.1016/j.bbr.2019.111929
Roetzel, P. G. (2019). Information overload in the information age: A review of the literature from business administration, business psychology, and related disciplines with a bibliometric approach and framework development. Business Research, 12(2), 479–522. https://doi.org/10.1007/s40685-018-0069-z
Satori, M. (2022). Effects of field independence, metacognitive knowledge, and vocabulary knowledge on second language listening comprehension. International Journal of Listening, 1–14. https://doi.org/10.1080/10904018.2022.2056037
Sbaffi, L., & Rowley, J. (2017). Trust and credibility in web-based health information: A review and agenda for future research. Journal of Medical Internet Research, 19(6), 1–17. https://doi.org/10.2196/jmir.7579
Schindler, M., & Lilienthal, A. J. (2019). Students’ creative process in mathematics: Insights from eye-tracking-stimulated recall interview on students’ work on multiple solution tasks. International Journal of Science and Mathematics Education, 18, 1565–1586. https://doi.org/10.1007/s10763-019-10033-0
Schraw, G., Crippen, K. J., & Hartley, K. (2006). Promoting self-regulation in science education: metacognition as part of a broader perspective on learning. Research in Science Education, 36(1–2), 111–139. https://doi.org/10.1007/s11165-005-3917-8
Soto, C., de Blume, A. P. G., Jacovina, M., McNamara, D., Benson, N., & Riffo, B. (2019). Reading comprehension and metacognition: The importance of inferential skills. Cogent Education, 6(1), 1565067. https://doi.org/10.1080/2331186X.2019.1565067
Ståhl, T. (2019). Epistemic beliefs and Googling. Frontline Learning Research, 7(3), 27–63. https://doi.org/10.14786/flr.v7i3.417
Stolp, S., & Zabrucky, K. M. (2017). Contributions of metacognitive and self-regulated learning theories to investigations of calibration of comprehension. International Electronic Journal of Environmental Education, 2, 7–31
Stoten, D. W. (2019). Metacognition, calibration, and self-regulated learning: An exploratory study of undergraduates in a business school. Learning: Research and Practice, 5(1), 24–47
Tabachuk, N. P., Ledovskikh, I. A., Shulika, N. A., Karpova, I. V., Kazinets, V. A., & Polichka, A. E. (2018). Information competency and creative initiative of personality and their manifestation in activity. Journal of Social Studies Education Research, 9(1), 168–186
Taub, M., & Azevedo, R. (2019). How does prior knowledge influence eye fixations and sequences of cognitive and metacognitive SRL processes during learning with an intelligent tutoring system? International Journal of Artificial Intelligence in Education, 29(1), 1–28. https://doi.org/10.1007/s40593-018-0165-4
Thiede, K., Wright, K. L., Hagenah, S., & Wenner, J. (2019). Drawings as diagnostic cues for metacomprehension judgment. In N. N. Feza (Ed.), Metacognition in learning (pp.65–79). IntechOpen. https://doi.org/10.5772/intechopen.78892
Țîru, C. M. (2021). Using students’ reflection in the university educational process–a qualitative approach. Educatia 21 Journal, 21, 49–61. https://doi.org/10.24193/ed21.2021.21.05
Tsai, M. J. (2009). Online information searching strategy inventory (OISSI): A quick version and a complete version. Computers & Education, 53(2), 473–483. https://doi.org/10.1016/j.compedu.2009.03.006
Tsai, M. J., Liang, J. C., Hou, H. T., & Tsai, C. C. (2012). University students’ online information searching strategies in different search contexts. Australasian Journal of Educational Technology, 28(5), 881–895. https://doi.org/10.14742/ajet.822
Tsai, Y. H., Lin, C. H., Hong, J. C., & Tai, K. H. (2018). The effects of metacognition on online learning interest and continuance to learn with MOOCs. Computers & Education, 121, 18–29. https://doi.org/10.1016/j.compedu.2018.02.011
Tsai, M. J., & Tsai, C. C. (2003). Information searching strategies in web-based science learning: The role of Internet self-efficacy. Innovations in Education and Teaching International, 40, 43–50. https://doi.org/10.1080/1355800032000038822
Tu, Y. W., Shih, M., & Tsai, C. C. (2008). Eighth graders’ web searching strategies and outcomes: The role of task types, web experiences and epistemological beliefs. Computers & Education, 51(3), 1142–1153. https://doi.org/10.1016/j.compedu.2007.11.003
Tuluk, G., & Kepceoglu, I. (2019). Pre-service teachers’ web pedagogical content knowledge and online information searching strategies. International Journal of Evaluation and Research in Education, 8(2), 229–236. https://doi.org/10.11591/ijere.v8i2.18771
von Hoyer, J. F., Kimmerle, J., & Holtz, P. (2022). Acquisition of false certainty: Learners increase their confidence in the correctness of incorrect answers after online information search. Journal of Computer Assisted Learning. Ahead-of-print. https://doi.org/10.1111/jcal.12657
van Loon, M. H., de Bruin, A. B., van Gog, T., & van Merriënboer, J. J. (2013). Activation of inaccurate prior knowledge affects primary-school students’ metacognitive judgments and calibration. Learning and Instruction, 24, 15–25. https://doi.org/10.1016/j.learninstruc.2012.08.005
Veenman, M. V., Van Hout-Wolters, B. H., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1(1), 3–14. https://doi.org/10.1007/s11409-006-6893-0
Weber, H., Becker, D., & Hillmert, S. (2019). Information-seeking behaviour and academic success in higher education: Which search strategies matter for grade differences among university students and how does this relevance differ by field of study? Higher Education, 77(4), 657–678. https://doi.org/10.1007/s10734-018-0296-4
Weber, H., Hillmert, S., & Rott, K. J. (2018). Can digital information literacy among undergraduates be improved? Evidence from an experimental study. Teaching in Higher Education, 23(8), 909–926. https://doi.org/10.1080/13562517.2018.1449740
Whitebread, D., Coltman, P., Pasternak, D. P., Sangster, C., Grau, V., Bingham, S. … Demetriou, D. (2009). The development of two observational tools for assessing metacognition and self-regulated learning in young children. Metacognition and Learning, 4(1), 63–85. https://doi.org/10.1007/s11409-008-9033-1
Winne, P. H., & Jamieson-Noel, D. (2002). Exploring students’ calibration of self reports about study tactics and achievement. Contemporary Educational Psychology, 27(4), 551–572. https://doi.org/10.1016/S0361-476X(02)00006-1
Wu, J. Y., & Peng, Y. C. (2017). The modality effect on reading literacy: Perspectives from students’ online reading habits, cognitive and metacognitive strategies, and web navigation skills across regions. Interactive Learning Environments, 25(7), 859–876. https://doi.org/10.1080/10494820.2016.1224251
Zhai, X., Chu, X., Meng, N., Wang, M., Spector, M., Tsai, C. C., & Liu, H. (2022). The Effect of multi-mode stimuli of feedforward and eye tracking on metacognition—An Exploratory study using digital dictionaries. Educational Technology & Society, 25(1), 213–227
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413-419.https://doi.org/10.1016/j.learninstruc.2012.03.004
Zhou, M. (2017). Do online search processes vary by task complexity? An eye-tracking study. International Journal of Social Science and Humanity, 7(11), 698-701. https://doi.org/10.18178/ijssh.2017.7.11.910
Zhou, M., & Lam, K. K. L. (2019). Metacognitive scaffolding for online information search in K-12 and higher education settings: A systematic review. Educational Technology Research and Development, 67(6), 1353-1384. https://doi.org/10.1007/s11423-019-09646-7
Zion, M., Adler, I., & Mevarech, Z. (2015). The effect of individual and social metacognitive support on students’ metacognitive performances in an online discussion. Journal of Educational Computing Research, 52(1), 50–87. https://doi.org/10.1177/0735633114568855
Funding
This research was funded by the Multi-Year Research Grant in University of Macau (Ref.: MYRG2018-00084-FED).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The author has no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhou, M. Students’ metacognitive judgments in online search: a calibration study. Educ Inf Technol 28, 2619–2638 (2023). https://doi.org/10.1007/s10639-022-11217-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-022-11217-y