Reconstructing experiences with iScale

Abstract

We present iScale, a survey tool for the retrospective elicitation of longitudinal user experience data. iScale aims to minimize retrospection bias and employs graphing to impose a process during the reconstruction of one's experiences. Two versions, the constructive and the value-account iScale, were motivated by two distinct theories on how people reconstruct emotional experiences from memory. These two versions were tested in two separate studies. Study 1 aimed at providing qualitative insight into the use of iScale and compared its performance to that of free-hand graphing. Study 2 compared the two versions of iScale to free recall, a control condition that does not impose structure on the reconstruction process. Overall, iScale resulted in an increase in the amount, the richness, and the test–retest consistency of recalled information as compared to free recall. These results provide support for the viability of retrospective techniques as a cost-effective alternative to longitudinal studies.

Introduction

Understanding the use and acceptance of interactive products beyond initial use has always been an interest of the human–computer interaction (HCI) community (Erickson, 1996, Prümper et al., 1992). However, two recent trends make the call for a more longitudinal view more urgent (Karapanos et al., 2009). First, legislation and competition within the consumer electronics industry have led to prolonged product warranties, resulting in an alarmingly increasing number of products being returned on the basis of failing to satisfy their users' “true” needs (Den Ouden et al., 2006). Second, products have become more embedded into services. Often, products are being sold for low prices or even given away for free and revenues stem mainly from the supported service and their prolonged use (Karapanos et al., 2009). Thus, the overall focus on product quality shifts from a focus on the classic phases of pre-purchase and purchase to a more longitudinal perspective, trying to better understand use and liking over time. This is a shift increasingly taken up by the HCI community (Gerken et al., 2007, Barendregt et al., 2006, Fenko et al., 2009, Karapanos et al., 2008, von Wilamowitz-Moellendorff et al., 2006, Courage et al., 2009, Vaughan et al., 2008, Kjeldskov et al., in press).

From a methodological perspective, one may distinguish three approaches to understanding the development of usage and experience over time (von Wilamowitz-Moellendorff et al., 2006): cross-sectional, pre-post/longitudinal, and retrospective reconstruction. Cross-sectional approaches are the most popular in the HCI domain (Prümper et al., 1992, Bednarik et al., 2005). Cross-sectional studies distinguish, for example, user groups with different levels of expertise, for instance, novice and expert users. Observed variation in experience or behavior is then attributed to expertise in the sense of a quasi-experimental variable. This approach is, however, limited as it is prone to confounding variables, such as failing to control for external variation and, more importantly, falsely attributing variation across the user groups to expertise. Prümper et al. (1992) already highlighted this problem, by showing that different definitions of novice and expert users lead to different results.

Beyond the cross-sectional, one may further distinguish pre-post and true longitudinal approaches. Pre-post designs study the same participants at two points in time. For instance, Kjeldskov et al. (in press) studied the same seven nurses, using a healthcare system, right after the system was introduced and 15 months later. Karapanos et al. (2008) explored how 10 individuals formed overall evaluative judgments of a novel pointing device, during the first week of use as well as after 4 weeks of using the product. While these approaches study the same participants over an extended period of time, they cannot tell much about the exact form of change, due to the limited number of only two observations. True longitudinal designs take more measurements and employ a number of statistical techniques to track change in general and to estimate the impact of particular events on change. Because of their laborious nature, however, they are only rarely used in practice and research.

Different granularities in longitudinal studies can be distinguished (see von Wilamowitz-Moellendorff et al., 2006): a micro-perspective (e.g. an hour), a meso-perspective (e.g. 5 weeks) and a macro-perspective, with a scope of years of use. Studies with a micro-perspective assess how users' experience changes through increased exposure over the course of a single session of use. For instance, Minge (2008) elicited judgments of perceived usability, innovativeness and the overall attractiveness of computer-based simulations of a digital audio player at three distinct points: (a) after participants had seen but not interacted with the product, (b) after 2 min of interaction and (c) after 15 min of interaction. An example of a study with a meso-perspective is Karapanos et al. (2009). They followed six individuals after the purchase of a product over the course of 5 weeks. One week before the purchase of the product, participants started reporting their expectations. After product purchase, participants were asked to narrate the three most influential experiences of each day. Studies with a macro-perspective are ‘nearly non-existent’ (von Wilamowitz-Moellendorff et al., 2006).

A third approach is the retrospective reconstruction of personally meaningful experiences from memory. Different variants of the Critical Incident Technique, popular in marketing and service management research (Edvardsson and Roos, 2001, Flanagan, 1954), ask participants to report critical incidents over periods of weeks, months or the complete time-span of the use of a product or service. In a survey study, Fenko et al. (2009) asked participants to recall their single most pleasant and unpleasant experience with different types of products and to assess the most important sensory modality (i.e. vision, audition, touch, smell and taste) at different points in time (i.e. when choosing the product in the shop, during the first week, after the first month, and after the first year of usage). von Wilamowitz-Moellendorff et al., 2006, von Wilamowitz-Moellendorff et al., 2007 proposed a structured interview technique called CORPUS (Change Oriented analysis of the Relation between Product and User) for the retrospective assessment of the dynamics in users' perceptions of different facets of perceived product quality. CORPUS starts by asking participants to assess the currently perceived quality of “their” product on a number of defined facets (usability, utility, beauty, stimulation, identification, and global evaluation). Subsequently, they are asked to “go back in time” and to compare their current perception and evaluation of the product to the moment right after purchasing the product. If change has occurred, participants are further prompted to indicate the direction and shape of change (e.g., accelerated improvement, steady deterioration). Finally, participants are asked to elaborate the reasons that induced the changes in the form of short narratives, so-called “change incidents”. The obtained data can be used quantitatively by constructing graphs of change (see Fig. 1 for an example) and qualitatively by exploring the reasons people give for changes.

A common critique of methods relying on memory is the degree to which recalled experiences are biased or incomplete. In the context of perceived product quality, we argue that this is of minor importance. While a given reconstruction from memory should be truthful (i.e., reflect what the participant really thinks), it seems less important, whether the reconstructed timeline as well as the reasons given are true (i.e. reflect what actually happened) as long as the participant is convinced that what she is reporting actually happened. This is because, we are foremost interested in subjective reconstructions because those (and not “objective” data) will be communicated to others as well as guide the individual's future activities. In other words, it may not matter how good a product is objectively, it is the “subjective”, the “experienced”, which matters (Hassenzahl et al., 2006). See also Norman (2009). To give a further example: Redelmeier and Kahneman (1996) found retrospective assessment of the pain experienced during colonoscopy to be biased. People put an extra weight on the most painful moment and the end of the examination. This has interesting consequences. One can, for example, deliberately prolong the examination (something not approved by the patients), but make sure that these last, additional 2 min are not painful. The consequence is an overall assessment of the examination as less painful compared to people without the additional 2 min. While this is clearly a bias, people simply have no memory for all the moments they experience, but will remember their overall impression of the examination. The retrospective judgment is more real to them than what actually happened. While the validity of remembered experiences may not be crucial, their consistency across multiple recalls is. It seems at least desirable that participants would report their experiences consistently over multiple trials. If recall would be purely “random”, the value of respective reports for design would be questionable. In other words, what we remember might be different from what we experienced; however, as long as these memories are consistent over multiple recalls, they provide valuable information.

In the area of critical incident research, interviewing techniques have been developed with the aim to assist participants in remembering more details of and contextual information around experienced critical incidents (Edvardsson and Roos, 2001). However, interviews in general, however, need substantial skills and resources. It, thus, seems desirable to create a self-reporting approach. Consequently, this paper presents iScale, a survey tool that was designed to increase participants' effectiveness in reconstructing their experiences with a product over time. iScale uses a graphical representation of change over time as a major support (i.e., time-line graphing). Other than previous approaches (von Wilamowitz-Moellendorff et al., 2006, Kujala et al., 2011), the employed procedure is more firmly grounded on theory, actually deriving variants of the procedure based on competing theoretical models of the retrospective reconstruction of episodes and experiences from memory. Graphing is assumed to support the reconstruction process through what Goldschmidt (1991) calls interactive imagery (i.e., “the simultaneous or almost simultaneous production of a display and the generation of an image that it triggers”). The idea of using graphing as an approach to introspecting on past emotional experiences can be traced back to Sonnemans and Frijda (1994).

We begin with laying out two different ways of obtaining retrospective reconstructions of experiences and their theoretical foundation. We then present the results of two studies. Study 1 acquired a qualitative understanding of the use of iScale in comparison to its analog equivalent (i.e. free-hand graphing). Study 2 assessed how iScale compares to an experience reporting tool without graphic support, which, can be seen as a control condition to assess the impact of iScale on participants' effectiveness and test–retest consistency in reconstructing experiences.