Measuring digital literacy across three age cohorts: Exploring test dimensionality and performance differences
Introduction
Digital literacy (DL) has become an essential capacity to successfully master daily tasks and routines in the 21st century as technology is ubiquitous in our daily lives and permeates all sectors of society (Siddiq, Gochyyev, & Wilson, 2017; van Laar, van Deursen, van Dijk, & de Haan, 2017; Voogt & Roblin, 2012). Although various definitions of DL have been proposed, these definitions converge around the retrieval and processing of information via new technologies, as well as communication and the production of content using information and communication technologies (ICT; Siddiq, Hatlevik, Olsen, Throndsen, & Scherer, 2016). Different terminologies have been used to refer to DL in the literature, such as computer literacy, information literacy, or computer and information literacy (Bawden, 2008), but DL is the term preferred here as it refers to uses of all kinds of digital devices and digital environments.
There are continuing debates about the components (or dimensions) of DL and its measurement. While most DL frameworks define DL as a multi-dimensional construct, several empirical studies reported only a general DL score (e.g., Gebhardt & Schulz, 2015; Katz, 2007). In addition, little is known about a child's developmental progression of DL over time or how students across age groups compare in terms of their DL performance. Research on primary school students' DL is relatively rare as most studies have focused on older students and adults (Siddiq et al., 2016). However, in view of the importance for schools to be able to provide age-appropriate DL education in the curriculum, an understanding of both its dimensional structure and measurement, as well as performance at different ages is needed. Therefore, the present study aimed to address gaps in existing research through the development of an assessment suitable for measuring DL at primary and secondary school levels.
Section snippets
Background
In this section, we review the research literature that underpinned our study: theoretical and empirical studies related to the definition and dimensionality of DL, DL performance of children in different age groups, and methodologies in assessing DL.
Method
This research adopted a cross-cohort panel design to examine performance differences among students in three different age cohorts, including one cohort of primary school students (Primary 3) and two cohorts of secondary students (Secondary 1 and Secondary 3) in Hong Kong. The current analysis reports on the development of the DL assessment instrument and cross-cohort differences in students’ DL performance using data from the first wave that was conducted in the 2018/2019 school year.
Dimensionality of DL
First, the analysis examined whether or not DL, as measured in the developed assessment, reflected the five-dimensional structure theorized in DigComp. The AIC and BIC were 105,483 and 106,378 for the 2PLM, respectively, and 105,460 and 106,411 for the M2PLM, respectively. These results suggested that an argument can be made for the M2PLM (based on the AICs) or the 2PLM (based on the BICs). Upon further inspection of the M2PLM, it revealed that the latent correlations among the five
Tension between the theoretical conceptualization of DL and empirical assessment results (RQ1)
This study aimed to develop a reliable DL performance assessment for three age cohorts. The assessment that was developed measured a broader range of DL aspects than most previous studies which made it particularly suitable for examining the dimensionality of DL. In particular, a goal of this research was to examine whether the instrument measured the five components that were conceptualized in the assessment framework, or if DL should be conceived of as a unidimensional literacy as several
Conclusion
This study has developed an age-appropriate assessment instrument measuring DL across three cohorts of primary and secondary school students. IRT analyses showed that the measure was essentially unidimensional, of high reliability, and comparable across students of different gender. This result may be considered as evidence for the construct validity of the assessment. The results also demonstrate that there is a tension between the theoretical conceptualizations of DL as a multidimensional
CRediT authorship contribution statement
Kuan-Yu Jin: Data curation, Investigation, Formal analysis, Writing - original draft, Visualization. Frank Reichert: Conceptualization, Data curation, Writing - original draft, Writing - review & editing, Supervision, Visualization. Louie P. Cagasan: Data curation, Investigation. Jimmy de la Torre: Conceptualization, Methodology, Writing - review & editing, Supervision, Funding acquisition. Nancy Law: Conceptualization, Methodology, Writing - review & editing, Supervision, Project
Acknowledgment
We would like to thank all the schools, teachers and students who have given their time to participate in our study.Funding: This work was supported by the Research Grants Council of the Hong Kong Special Administrative Region, China [Project No. T44-707/16-N].Declarations of interest: None.
References (57)
Reliability as a measurement design effect
Studies In Educational Evaluation
(2005)- et al.
Direct measures of digital information processing and communication skills in primary education: Using item response theory for the development and validation of an ICT competence scale
Computers & Education
(2014) - et al.
Assessment of 21st century ICT skills in Chile: Test design and results from high school level students
Computers & Education
(2012) - et al.
Moving beyond the study of gender differences: An analysis of measurement invariance and differential item functioning of an ICT literacy scale
Computers & Education
(2017) - et al.
The relation between 21st-century skills and digital skills: A systematic literature review
Computers in Human Behavior
(2017) - et al.
Developing and validating of a perceived ICT literacy scale for junior secondary school students: Pedagogical and educational contributions
Computers & Education
(2014) - et al.
Factorial invariance across gender of a perceived ICT literacy scale
Learning and Individual Differences
(2015) - et al.
Longitudinal assessment of digital literacy in children: Findings from a large Dutch single-school study
Computers & Education
(2020) - et al.
The role of ICT self-efficacy as a mediator between students' ICT use and their achievement in a computer and information literacy test
Computer Education
(2016) - et al.
Learning in digital networks—ICT literacy: A novel assessment of students' 21st century skills
Computers & Education
(2017)
Taking a future perspective by learning from the past – a systematic review of assessment instruments that aim to measure primary and secondary school students' ICT literacy
Educational Research Review
Is there a gender gap? A meta-analysis of the gender differences in students' ICT literacy
Educational Research Review
Gender differences in adolescents' out-of-school literacy practices: A multifaceted approach
Computers & Education
Information theory and an extension of the maximum likelihood principle
NAP sample assessment ICT literacy: Years 6 & 10
Origins and concepts of digital literacy
Some latent trait models and their use in inferring an examinee's ability
Children's thinking: Cognitive development and individual differences
DigComp 2.1: The Digital Competence Framework for Citizens with eight proficiency levels and examples of use (No. JRC106281)
Testing patient-reported outcome measurement equivalence in multinational clinical trials: An exemplar using the 12-item Multiple Sclerosis Walking Scale
Multiple Sclerosis Journal – Experimental Translational Clinical
Information literacy for Hong Kong students. Hong Kong: The government of the Hong Kong SAR
Changes over time in digital literacy
CyberPsychology and Behavior
Digcomp: A framework for developing and understanding digital competence in europe
eLearning Papers
International large-scale computer-based studies on information technology literacy in education
International computer and information literacy study 2018: Assessment framework
Preparing for life in a digital world
IEA international computer and information literacy study 2018 technical report
Preparing for life in a digital age
Cited by (62)
Digital literacy, the use of collaborative technologies, and perceived social proximity in a hybrid work environment: Technology as a social binder
2024, Computers in Human Behavior ReportsDigital competence test for learning in schools: Development of items and scales
2023, Computers and EducationExploring the role of intrinsic motivation in ChatGPT adoption to support active learning: An extension of the technology acceptance model
2023, Computers and Education: Artificial IntelligenceNavigating the digital world: development of an evidence-based digital literacy program and assessment tool for youth
2024, Smart Learning EnvironmentsDigital Literacy and Associated Factors in Older Adults Living in Urban South Korea: A Qualitative Study
2024, CIN - Computers Informatics Nursing