Elsevier

Computers & Education

Volume 157, November 2020, 103968
Computers & Education

Measuring digital literacy across three age cohorts: Exploring test dimensionality and performance differences

https://doi.org/10.1016/j.compedu.2020.103968Get rights and content

Highlights

  • Digital literacy (DL) has been conceptualized as a multidimensional construct.

  • A comprehensive DL assessment using the DigComp framework was developed.

  • Empirically, DL can be conceived of as a general, unidimensional literacy.

  • Secondary students show better DL performance than primary students.

  • Gender differences in DL performance exist only among secondary students.

Abstract

Digital literacy (DL) is an important capacity for students’ learning in a rapidly changing world. However, tension exists between the theoretical conceptualizations of DL as a multidimensional construct and empirical studies reporting unidimensional DL scores. Also, little is known about how DL may vary among different age cohorts, and whether and at which age do performance gaps emerge with respect to gender. The focus of this research is to develop a test appropriate for measuring DL performance at different ages and a comprehensive DL assessment framework has been adopted for this purpose. Using data from three age cohorts of students (one from primary schools and two from secondary schools), the dimensionality of DL and performance differences are examined. Comparisons of unidimensional and multidimensional item response models suggest the measured DL to be a unidimensional construct. The results also show that secondary school students obtained higher levels of DL compared to primary school students. A gender gap in DL is found among secondary school students. There is also a need for further research to understand through longitudinal studies the emergence of the gender gap in DL performance.

Introduction

Digital literacy (DL) has become an essential capacity to successfully master daily tasks and routines in the 21st century as technology is ubiquitous in our daily lives and permeates all sectors of society (Siddiq, Gochyyev, & Wilson, 2017; van Laar, van Deursen, van Dijk, & de Haan, 2017; Voogt & Roblin, 2012). Although various definitions of DL have been proposed, these definitions converge around the retrieval and processing of information via new technologies, as well as communication and the production of content using information and communication technologies (ICT; Siddiq, Hatlevik, Olsen, Throndsen, & Scherer, 2016). Different terminologies have been used to refer to DL in the literature, such as computer literacy, information literacy, or computer and information literacy (Bawden, 2008), but DL is the term preferred here as it refers to uses of all kinds of digital devices and digital environments.

There are continuing debates about the components (or dimensions) of DL and its measurement. While most DL frameworks define DL as a multi-dimensional construct, several empirical studies reported only a general DL score (e.g., Gebhardt & Schulz, 2015; Katz, 2007). In addition, little is known about a child's developmental progression of DL over time or how students across age groups compare in terms of their DL performance. Research on primary school students' DL is relatively rare as most studies have focused on older students and adults (Siddiq et al., 2016). However, in view of the importance for schools to be able to provide age-appropriate DL education in the curriculum, an understanding of both its dimensional structure and measurement, as well as performance at different ages is needed. Therefore, the present study aimed to address gaps in existing research through the development of an assessment suitable for measuring DL at primary and secondary school levels.

Section snippets

Background

In this section, we review the research literature that underpinned our study: theoretical and empirical studies related to the definition and dimensionality of DL, DL performance of children in different age groups, and methodologies in assessing DL.

Method

This research adopted a cross-cohort panel design to examine performance differences among students in three different age cohorts, including one cohort of primary school students (Primary 3) and two cohorts of secondary students (Secondary 1 and Secondary 3) in Hong Kong. The current analysis reports on the development of the DL assessment instrument and cross-cohort differences in students’ DL performance using data from the first wave that was conducted in the 2018/2019 school year.

Dimensionality of DL

First, the analysis examined whether or not DL, as measured in the developed assessment, reflected the five-dimensional structure theorized in DigComp. The AIC and BIC were 105,483 and 106,378 for the 2PLM, respectively, and 105,460 and 106,411 for the M2PLM, respectively. These results suggested that an argument can be made for the M2PLM (based on the AICs) or the 2PLM (based on the BICs). Upon further inspection of the M2PLM, it revealed that the latent correlations among the five

Tension between the theoretical conceptualization of DL and empirical assessment results (RQ1)

This study aimed to develop a reliable DL performance assessment for three age cohorts. The assessment that was developed measured a broader range of DL aspects than most previous studies which made it particularly suitable for examining the dimensionality of DL. In particular, a goal of this research was to examine whether the instrument measured the five components that were conceptualized in the assessment framework, or if DL should be conceived of as a unidimensional literacy as several

Conclusion

This study has developed an age-appropriate assessment instrument measuring DL across three cohorts of primary and secondary school students. IRT analyses showed that the measure was essentially unidimensional, of high reliability, and comparable across students of different gender. This result may be considered as evidence for the construct validity of the assessment. The results also demonstrate that there is a tension between the theoretical conceptualizations of DL as a multidimensional

CRediT authorship contribution statement

Kuan-Yu Jin: Data curation, Investigation, Formal analysis, Writing - original draft, Visualization. Frank Reichert: Conceptualization, Data curation, Writing - original draft, Writing - review & editing, Supervision, Visualization. Louie P. Cagasan: Data curation, Investigation. Jimmy de la Torre: Conceptualization, Methodology, Writing - review & editing, Supervision, Funding acquisition. Nancy Law: Conceptualization, Methodology, Writing - review & editing, Supervision, Project

Acknowledgment

We would like to thank all the schools, teachers and students who have given their time to participate in our study.Funding: This work was supported by the Research Grants Council of the Hong Kong Special Administrative Region, China [Project No. T44-707/16-N].Declarations of interest: None.

References (57)

  • F. Siddiq et al.

    Taking a future perspective by learning from the past – a systematic review of assessment instruments that aim to measure primary and secondary school students' ICT literacy

    Educational Research Review

    (2016)
  • F. Siddiq et al.

    Is there a gender gap? A meta-analysis of the gender differences in students' ICT literacy

    Educational Research Review

    (2019)
  • A. Ünlüsoy et al.

    Gender differences in adolescents' out-of-school literacy practices: A multifaceted approach

    Computers & Education

    (2010)
  • H. Akaike

    Information theory and an extension of the maximum likelihood principle

  • NAP sample assessment ICT literacy: Years 6 & 10

    (2018)
  • D. Bawden

    Origins and concepts of digital literacy

  • A. Birnbaum

    Some latent trait models and their use in inferring an examinee's ability

  • D.F. Bjorklund et al.

    Children's thinking: Cognitive development and individual differences

    (2018)
  • S. Carretero et al.

    DigComp 2.1: The Digital Competence Framework for Citizens with eight proficiency levels and examples of use (No. JRC106281)

    (2017)
  • H. Dib et al.

    Testing patient-reported outcome measurement equivalence in multinational clinical trials: An exemplar using the 12-item Multiple Sclerosis Walking Scale

    Multiple Sclerosis Journal – Experimental Translational Clinical

    (2017)
  • Education Bureau

    Information literacy for Hong Kong students. Hong Kong: The government of the Hong Kong SAR

    (2018)
  • Y. Eshet-Alkalai et al.

    Changes over time in digital literacy

    CyberPsychology and Behavior

    (2009)
  • A. Ferrari et al.

    Digcomp: A framework for developing and understanding digital competence in europe

    eLearning Papers

    (2014)
  • J. Fraillon

    International large-scale computer-based studies on information technology literacy in education

  • J. Fraillon et al.

    International computer and information literacy study 2018: Assessment framework

    (2019)
  • J. Fraillon et al.

    Preparing for life in a digital world

  • J. Fraillon et al.

    IEA international computer and information literacy study 2018 technical report

  • J. Fraillon et al.

    Preparing for life in a digital age

  • Cited by (62)

    View all citing articles on Scopus
    View full text