Development and validation of the Multimodal Presence Scale for virtual reality environments: A confirmatory factor analysis and item response theory approach

Highlights

• A new scale measuring physical, social, and self-presence was developed.

• Confirmatory factor analysis supported the three-factor solution.

• Item response theory supported the validity and reliability of the 15 item measure.

• A cross-validation study provided evidence for the generalizability of the results.

Abstract

Presence is one of the most important psychological constructs for understanding human-computer interaction. However, different terminology and operationalizations of presence across fields have plagued the comparability and generalizability of results across studies. Lee's (2004) unified understanding of presence as a multidimensional construct made up of physical, social, and self-presence, has created a unified theory of presence; nevertheless, there are still no psychometrically valid measurement instruments based on the theory. Two studies were conducted that describe the development of a standardized multidimensional measure of presence (the MPS) for a VR learning context based on this theory, and its validation using confirmatory factor analysis and item response theory. The results from Study 1 which included 161 medical students from Denmark indicated that the items used in the MPS measure a three dimensional theoretical model of presence: physical, social, and self-presence. Furthermore, IRT analyses indicated that it was possible to limit the number of items in the MPS to 15 (five items per sub-dimension) while maintaining the construct validity and reliability of the measure. The results of Study 2, which included 118 biology students from Scotland, supported the validity and generalizability of the MPS in a new context.

Introduction

The construct of presence has great practical relevance for the design and evaluation of media products, especially in education, entertainment, telecommunications, psychology, and health care (Klimmt and Vorderer, 2003, Lee, 2004). Consequently, several scholars have attempted to define the concept of presence in order to measure and operationalize it as a psychological construct. There is some consensus regarding defining presence as the experience or feeling of being present in a mediated environment, rather than the immediate physical environment wherein one is currently bodily present (Steuer, 1992, Witmer and Singer, 1998). However, different fields of research have typically used different terminology (Lee, 2004), which has made it difficult for researchers to create a unified theory of presence, thereby creating difficulties in comparing and generalizing results across studies. However, through an extensive explication process, Lee (2004) made a thorough attempt to provide a unified understanding and general terminology of presence which had heretofore been lacking.

Although there has been great progress in further understanding presence as a psychological construct since this development, one area where the progress has stagnated is on the operationalization and measurement of presence according to Lee's (2004) definition of the construct. This is surprising, considering Lee's emphasis on the importance of creating standard validated measures of presence so that comparison between and among studies, and generalization across studies, would be possible. With the increasing availability of sophisticated technologies for simulating interactions between people and places such as Virtual Reality (VR), the need among researchers and media content developers for a validated standardized measure of presence, using a unified theory such as Lee's (2004) to create such measures seems crucial.

The objective of this study was therefore to develop a standardized measure of presence for VR environments based on Lee's (2004) explication of presence, and to validate this measure using modern test theory; that is Confirmatory Factor Analysis (CFA; Brown, 2015) and Item Response Theory (IRT; Embertson & Reise, 2000). Accordingly, the scope of this study was not to develop a measure of presence to be used when actual people and places that are physically separated are brought together through technology, but rather to develop a presence scale pertaining to computer simulations and other programs employing VR. Furthermore, the aim of this study was not to validate Lee's (2004) theory of presence, but to validate a standardized measure of presence based on this theory.

This paper describes the development of the Multimodal Presence Scale (MPS), and the evaluation of its psychometric properties with CFA and IRT. First, a brief introduction to Lee's (2004) explication of presence and an outline of previous measures of presence is presented, followed by a description of the development of the MPS based on existing acknowledged measures of presence. Next, CFA and IRT are explained. This is followed by a description of two studies wherein these methodologies were used to test the psychometric properties of the MPS. Finally, the strengths and weaknesses of the MPS and practical considerations involving its use are discussed.

According to Lee (2004), both real and virtual experience can be divided into three domains; physical, social, and self. Physical experience involves the experience of physical objects and environments; social experience refers to the experience of social actors; and self-experience describe the experiences people have of their own selves.

In contrast to real experience, there are two ways in which an experience can become virtual (Lee, 2004). Firstly, an experience can become virtual when the act of experience is mediated by, or made possible by, human-made technology and the technology enables users to experience mediated versions of actual entities which hold some kind of valid connection with the actual entities that they represent (para-authentic entities). Secondly, an experience can also become virtual when experienced entities are artificially created or simulated by human-made technology. In this latter case the entities do not actually exist in the real world, but are experienced as if they would exist in the real world due to human-made technology (artificial entities). In sum, an experience can become virtual when the act of experiencing physical objects or environments, social actors and/or the self is mediated, or artificially constructed, by human-made technology.

In addition to the before-mentioned aspects of real and virtual experience, Lee (2004) also distinguishes between sensory and non-sensory experiences. By including this distinction in his definition of presence Lee (2004) is able to incorporate the possibility of experiencing presence during the use of low-tech non-sensory media, such as books.

Based on the outlined theoretical framework, Lee (2004) defines physical, social and self-presence as a psychological state in which virtual (para-authentic or artificial) physical objects, social actors, and the self, respectively, are experienced as actual entities in either sensory or non-sensory ways. However, since the focus of this paper is on VR environments, the distinction between sensory and non-sensory experiences is not relevant here, as VR environments are sensory in nature.

The development of the MPS is based on Lee's (2004) theory of presence, specifically in the division of presence into three sub-dimensions; physical, social and self. However, since Lee (2004) does not provide a specific operationalization of physical, social, or self-presence, it is necessary to review the available literature of some of the most acknowledged previous measures of presence to develop adequate content validity for these three constructs.

Through a review of the literature, it is evident that there are various acknowledged measures of presence available; and many of these capture some aspects of each of the three constructs: physical, social, and self-presence. However, no attempts have been made to measure all three together and none of these measures have been reported in the literature to have been validated using both CFA and IRT. The Igroup Presence Questionnaire (IPQ) developed by Schubert, Friedmannm, and Regenbrecht (2001) did use CFA to identify a general structure, but without the utilization of IRT they lack information on how well the specific items function within each sub-dimension. More importantly, the IPQ fits entirely within the realm of physical presence, and is therefore unable to measure either social or self-presence as opposed to the MPS.

Considering the vast amount of acknowledged measures of presence, the MPS will consist of previously developed items that are selected through an extensive extraction process and validated through psychometric testing with CFA and IRT.