Internet testing: Equivalence between proctored lab and unproctored field conditions

Abstract

Companies that use web-based testing do not need to invite applicants to their offices for screening purposes, and applicants are not required to travel. Given the world-wide accessibility of the Internet and the savings in travel costs, web-based testing expands the applicant pool to geographically distant regions. This advantage comes along with the tangible drawback of less control over the testing situation and therefore possible influence on the data quality of scores obtained via Internet testing. This study examined the equivalence of proctored and unproctored web-based psychological testing. Results from 163 potential applicants who participated in a combined laboratory–field and between-subject/within-subject design study with two experimental conditions and two control conditions did not provide evidence that testing conditions affected test results.

Introduction

Studies have established the conditions under which paper–pencil tests and computerized tests show equivalent results (Mead and Drasgow, 1993, Richman et al., 1999). With the technological advances of the World Wide Web and the Internet, researchers have turned their interest to the question whether web-based test administration (Internet testing) does also provide equivalent results when compared to paper–pencil tests. Recent studies confirmed an earlier review by Lievens and Harris (2003) who saw initial evidence for measurement equivalence between web-based and traditional testing (Bartram and Brown, 2004, Bressani and Downs, 2002, Carlbring et al., 2007, Coyne et al., 2005, Epstein et al., 2001, Herrero and Meneses, 2006, Huang, 2006, Knapp and Kirk, 2003, Luce et al., 2007, Oswald et al., 2001, Ployhart et al., 2003, Salgado and Moscoso, 2003, Whitaker, 2007).

Companies that use web-based testing do not need to invite applicants to their offices and do not need to employ a test administrator when screening applicants; and applicants are not required to travel. Given the world-wide accessibility of the Internet and the savings in travel costs, Internet testing expands the applicant pool to geographically distant regions. This means the real advantage of web-based tests will only be fully used when applicants take them under unproctored field conditions. However, Buchanan and Smith (1999) pointed at a number of potential challenges to the validity of web-based testing. These revolve around the lack of control in the testing situation and the possibility of extraneous factors, e.g. distraction, environmental cues, technical variability between different hardware and software configurations; or temporary factors, e.g. fatigue, intoxication, that may influence responses and of which the recruiter or test administrator might not even be aware of.

A limitation of most studies so far was that researchers administered web-based test versions under proctored laboratory conditions. Only a few studies included examining the instruments under real-life conditions in the field. Miller et al. (2002) applied commonly used measures of alcohol use and compared a paper–pencil version with two web-based testing conditions. Data collection was conducted over two separate 48-h periods one week apart. The two web-based conditions were identical with the exception that the second group was asked to take a break from the survey by quitting the browser at any point and reconnecting to the web site at a later time. The interruption provided a proxy of real world interruptions (e.g., participant fatigue, lack of time to initiate or complete the survey) that may be common with self-paced home-based testing. Results showed no significant differences between the test conditions. Oswald et al. (2001) compared supervised and unsupervised Internet testing. Multiple group confirmatory factor analyses showed for personality measures a better model fit for supervised groups than for unsupervised groups. The model fit was equal for cognitive ability tests. Chuah, Drasgow, and Roberts (2006) did not find indications for inequivalence when they compared paper-and-pencil, proctored computer lab, and unproctored Internet testing conditions for scales consisting of adjective markers of the Big-Five personality traits. Do, Shepherd, and Drasgow (2005) found negligible differences and measurement equivalence for proctored and unproctored versions of personality scales, a cognitive ability test, and a biodata instrument. Coyne et al. (2005) found no significant differences in coefficients of equivalence and stability (CES) for groups that either completed the online version of a test battery under unsupervised conditions or the PC version under supervised and standardized conditions. The test battery included ability, interest, and personality tests.

However, a limitation of these studies was that equivalence had not been tested directly. Groups of different participants had been compared, instead of testing the same participants on two occasions under proctored and unproctored conditions. The study of Beaty, Fallon, and Shepherd (2002) is an exception, as they used a within-subject design. In their conference paper they reported negligible differences in test scores for participants who first took a cognitive ability test under unproctored conditions and at a later time under proctored conditions.

To our knowledge, there exist no published studies that have researched the equivalence of unproctored and proctored Internet testing of non-cognitive personality-related dimensions and cognitive abilities with a within-group design or a combined within-group/between-group design. The objective of this paper was therefore to examine the equivalence of web-based psychological measures administered under proctored lab conditions and under unproctored field conditions – both between groups and within groups over time. Specifically, we tested whether results from proctored web-based tests taken in the lab would be equivalent to results from unproctored web-based tests taken under field conditions, as reflected in similar means and variances between test conditions and over time, and as reflected in similar coefficients of equivalence and stability (CES).