Theme-based content analysis: a flexible method for virtual environment evaluation

Abstract

Virtual environment and multimedia technology are developing rapidly in many areas. These include visual complexity, the opportunity to provide multi-sensory input and output, affordability and a variety of system designs and applications. Involving the users in the design and development process can result in more appropriate and usable interfaces. In addition, an iterative evaluation throughout the process of technology development can result in a large amount of useful information being gathered from users. However, there can be problems with this—the data collection and analysis process can be time consuming; it can be difficult to report information back to the developers in a meaningful form, and thus the results of the evaluation may not get incorporated into interface design; and some evaluation techniques can be specific to the application (e.g. assessment of learning from an educational virtual environment application) or user group (e.g. people with learning disabilities).

This paper presents an evaluation method that has been successfully used in virtual environment and multimedia evaluation at the Virtual Reality Applications Research Team (VIRART), and has overcome some of these problems. Theme-based content analysis (TBCA) is a qualitative method that provides useful, detailed information about user opinions or behaviour, and can also provide general indications of results in the user population by the grouping of data into meaningful categories. A number of different data collection methods can be used (e.g. short interview, open-ended questionnaire questions, observation) allowing the time and expertise of the virtual environment researcher to be most usefully employed, and the needs and abilities of the user population to be met. The analysis process is less time-consuming, and allows both summarization of the results and retention of the raw data. As described in the paper, this flexible method can be applied in a number of different circumstances, with a variety of different virtual reality technologies (desktop, projection or head mounted display (HMD) systems). In addition, the results from this method can be presented in a simple format to allow an easy feedback of user opinions and behaviours to virtual environment developers, providing contextual examples and an indication of the proportion of users experiencing usability problems. This facilitates a direct input of the evaluation data into the virtual environment development process.