Transfer of expertise: An eye tracking and think aloud study using dynamic medical visualizations

Abstract

Expertise research has produced mixed results regarding the problem of transfer of expertise. Is expert performance context-bound or can the underlying processes be applied to more general situations? The present study tests whether expert performance and its underlying processes transfer to novel tasks within a domain. A mixed method study using eye-tracking and quantitative and qualitative analyses of think aloud protocols was conducted with medical professionals in radiology and nuclear medicine who diagnosed identical patient cases displayed with three different computer-based imaging technologies: a familiar, a semi-familiar, and an unfamiliar imaging technology. Results indicate that expert performance, as well as its underlying processes, transferred from the familiar to the semi-familiar, but not to the unfamiliar imaging technology. Educational implications of these findings are discussed in terms of their significance for designing technology-enhanced learning environments to promote the transfer of expertise.

Introduction

Expertise research has produced somewhat mixed results when looking for evidence of transfer of expertise (Hatano & Inagaki, 1986; Mayer & Wittrock, 1996; Sims & Mayer, 2002). Is expert performance in representative tasks context-bound or can the processes underlying expertise be applied to more general situations? The present study examines whether expertise and the processes underlying expertise transfer to novel tasks within a domain.

Expert performance can be defined as maximal adaptations to representative tasks within a domain (Ericsson, 2004; Tynjälä, Nuutinen, Eteläpelto, Kirjonen, & Remes, 1997). However, consider a scenario in which representative tasks of a domain change. These changes can occur in the work patterns associated with digital technologies. Specifically, in rapidly developing domains such as medicine, new imaging technologies for analyzing the human anatomy and its functions enter the clinical workplace on a regular basis (Burri, 2008; Joyce, 2006). In radiology and nuclear medicine departments worldwide, positron emission tomography (PET) and computer tomography (CT) are combined to form a new kind of fusion image, PET/CT, as shown in Fig. 1. PET shows metabolism of the body. CT shows anatomic structures. To interpret such novel visualizations, those skilled in PET diagnosis and those skilled in CT diagnosis need to adapt their skills to the complementary technology. If and to what extent these adaptations are successful, remains an empirical question to be answered. Will medical professionals be able to transfer their diagnostic skills from an imaging technology they are accustomed with to a new one with different affordances? Or do they experience a regression in their skills? Questions of this kind have attracted much empirical research on visualizations in cognitive and educational psychological research over the past decades (Gilmore, 1996; Holyoak & Spellman, 1993; Martin & Schwartz, 2005; Svendsen, 1991). There is still discussion about the extent to which expertise is domain-specific. For example, some researchers argued that expertise is context-bound (Ericsson, 2004; Mayer & Wittrock, 1996) while other researchers argued that expertise contains elements that can transfer to novel tasks (Feltovich, Spiro, & Coulson, 1997; Perkins & Solomon, 1989). We describe studies on each of these perspectives in more detail below.

Specifically, some researchers have argued that expertise is context-bound and specific to the domain in which it has developed. Sims and Mayer (2002) reported that spatial expertise in playing Tetris transferred to shapes that were very similar to those used in Tetris, but not to other tests of spatial ability. Several other studies also indicated that experts outperformed non-experts in playing games such as bridge (Sternberg & Frensch, 1992), chess (Frensch & Sternberg, 1989), Go (Na, 2006), or baseball (Wiley, 1998) only as long as the rules of the game remained unmodified. After the rule modification, the structure of the experts' knowledge base and the automatization of knowledge led to inflexible adaptations to the novel task demands. This phenomenon, which has been extensively examined in transfer research (Mayer & Wittrock, 1996), has been associated with functional fixedness or the reductive bias (Feltovich et al., 1997), and, generally, it has been considered as evidence of rigidifying effects of long term practice. Based on this evidence, we would therefore expect that expert performance primarily must be thought of as domain specific.

However, other researchers have argued that expertise is not only domain-specific but contains general elements that do transfer to novel and unfamiliar situations (Feltovich et al., 1997; Hatano & Inagaki, 1986; Perkins & Solomon, 1989). For example, Novick (1988) showed that when arithmetic word problems shared structural similarities, experts outperformed novices and transferred their skills, even if the word problems had dissimilar surface features. More recent evidence supports the notion that structural similarity between the target problem and its analog seems to be an important ingredient for the “optimal adaptability corridor” (Schwartz, Bransford, & Sears, 2005) within which transfer of expert strategies may occur (Barnett & Koslowski, 2002). Based on this evidence, we would therefore expect that expert performance can indeed transfer to novel tasks in a domain.

In summary, and to reactivate our example from the outset of this section, it is still unclear whether medical professionals facing newly introduced visualizations of the human anatomy and its functions may or may not show transfer of their visual diagnostic expertise, given the mixed results in the literature.

In an attempt to organize the mixed findings emerging from previous research on domain specificity, Mayer and Wittrock (1996) have synthesized existing theories of transfer in terms of three different views: transfer of domain-general skills, transfer of domain-specific skills, and transfer of domain-specific skills in context.

First, the transfer of domain-general skills view predicts that engaging in deliberate practice in a domain helps develop general heuristics that can be applied in a wide range of settings, at least as long as the target settings are structurally similar to the analog (Novick, 1988). Thus, expertise in interpreting one type of medical visualization should promote the ability to interpret comparable types of medical visualizations (i.e., visualizations displaying the same body parts and providing information on the same diseases). In this view, the comprehension of visualizations is a general ability, facilitated by a repertoire of strategies “operating at whatever database of knowledge to be needed” (Perkins & Solomon, 1989, p. 17). If it is true that expertise is associated with the development of general heuristics, then it follows that expertise in interpreting familiar visualizations transfers to structurally similar visualizations irrespective of surface similarity (cf. Novick, 1988). For example, a PET expert would be able to interpret both CT and PET/CT with high accuracy (see Fig. 1).

Second, the transfer of domain-specific skills view predicts that engaging in deliberate practice in a domain supports the development of component skills required in the novel task (Mayer & Wittrock, 1996). In this view, the comprehension of visualizations is not a general skill, but confined to the context within which it is situated. This view acknowledges the importance of domain-specific adjustments (Barnett & Koslowski, 2002; Perkins & Solomon, 1989). Examples of component skills in medical diagnosis are illness scripts, which are extensive and highly differentiated schematizations of knowledge. This knowledge base is necessary for expert flexibility to occur (Feltovich et al., 1997). If it is true that expertise is associated with the development of component skills, then it follows that expertise in interpreting familiar visualizations would transfer to other visualizations with both structure and surface similarity, but would not transfer to other visualizations with surface similarity only. For example, a CT expert would be able to interpret PET/CT with high accuracy, but not PET (see Fig. 1).

Third, the transfer of domain specific skills in context view predicts that engaging in deliberate practice in a domain helps develop situated skills necessary for reproducibly superior performance. These skills are contingent on familiar work settings because of high levels of stability in the expert's domain schemas (Ericsson, 2004; Na, 2006) and incompatibility of stored mental representations with unfamiliar stimuli (Frensch & Sternberg, 1989; Sims & Mayer, 2002; Wiley, 1998). If it is true that expertise is associated with the development of highly context-bound, situated skills that have little application to other domains, then it follows that expertise in interpreting familiar visualizations will not easily transfer to other visualizations. For example, a PET expert will be able to diagnose PET scans with high accuracy, but not CT or PET/CT scans (see Fig. 1). Before examining the predictive validity of these three hypotheses, a brief introduction to the context of the present study is in order.

Previous research on the transfer of expertise has largely focused on a performance level—that is, performance measures have been compared across familiar and unfamiliar tasks to indicate whether or not transfer has occurred. While a focus on performance is a necessary element to determine the presence of transfer, a second focus can be directed toward the processes underlying performance, largely because uncovering these processes would advance our understanding of skill superiority (Ericsson, 2004; Helle et al., 2011; Tynjälä et al., 1997). In the present study, processes underlying the comprehension of visualizations are described by means of three activities outlined in Mayer's (2009) cognitive theory of multimedia learning: selecting data, organizing data, and integrating data with prior knowledge. For example, when diagnosing a lung CT, a radiologist selects relevant parts of the presented material, such as areas of the radiograph where nodules are likely to be detected. This selection strategy indicates to what kind of information attentional resources are allocated (Gegenfurtner, Siewiorek, Lehtinen, & Säljö, in press; Haider & Frensch, 1999; Seppänen & Gegenfurtner, 2012). The radiologist then organizes each detected detail in working memory into a pictorial model (Mayer, 2009)—in this example, a mental representation of the presented patient case. This organization strategy indicates how internal connections among the selected material are made (Gegenfurtner et al., in press). Once these connections are organized, the radiologist integrates the organized material with prior knowledge retrieved from long-term memory, such as the biomedical knowledge of disease schemata or clinical knowledge of previously seen patient cases (Boshuizen & Schmidt, 1992). This integration strategy indicates how the selected and organized material is connected with prior knowledge (Mayer, 2009).

The next question is how to capture these processes thoroughly. We may assume that in perceptually complex tasks, as the one at hand, all three activities are crucial. Selecting information can be investigated by means of eye tracking (Gegenfurtner, Lehtinen, & Säljö, 2011; Seppänen & Gegenfurtner, 2012; Van Gog & Scheiter, 2010). Eye tracking is a method to capture the movements of the eyeballs to infer where on a stimulus a person looked at, for how long, and in which order. Organizing information and integrating information with prior knowledge, on the other hand, can be investigated by means of thinking-aloud (Ericsson & Simon, 1980). Thinking-aloud is a method to capture thought processes without changing their order or content by asking participants to utter their inner speech. Therefore, we may assume that eye tracking and think aloud can be used to document and measure the three processes of selecting, organizing, and integrating information (Ericsson & Simon, 1980; Gegenfurtner et al., 2011, in press; Mayer, 2009; Van Gog & Scheiter, 2010).

The development of expertise alters these comprehension processes. Several theories address these alterations. First, the theory of long-term working memory (Ericsson & Kintsch, 1995) focuses on qualitative changes in memory structures. This theory assumes that expertise extends the capacities for information processing owing to the acquisition of retrieval structures that allow advanced learners to rapidly encode information in long-term memory and efficiently access it for later task operations. If we assume that experts encode and retrieve information more rapidly in familiar than in unfamiliar tasks, then it follows that experts' rapid information processing should be reflected in eye movement and think aloud data, specifically through shorter fixation durations, a smaller number of verbalizations of selecting and organizing information, and a higher number of verbalizations on integrating information with prior knowledge.

Alterations are also addressed by a second theory. The information-reduction hypothesis (Haider & Frensch, 1999) focuses on the learned selectivity of information processing. This theory proposes that expertise optimizes the amount of processed information by neglecting task-irrelevant information and actively focusing on task-relevant information. This is accomplished through strategic considerations to selectively allocate attentional resources. Haider and Frensch (1999, p. 188) noted that, because of learning and training, “redundant information is perceptually ignored whenever this is possible.” Information reduction thus results from a growing ability to differentiate between the variables of a stimulus array. If the assumption is true that experts select the amount of information they attend to, then it follows that experts in familiar tasks compared with experts in unfamiliar tasks should have fewer fixations of shorter duration on task-redundant areas and more fixations of longer duration on task-relevant areas.

Finally, alterations are addressed by a third theory. The encapsulation theory (Boshuizen & Schmidt, 1992) proposes that biomedical knowledge is encapsulated into clinical knowledge and used implicitly during clinical reasoning. However, if needed, biomedical knowledge can be unfolded again when facing difficult or atypical cases. This theory proposes a reorganization of expert knowledge. The theory also indicates the development of a more precise language, reflected in uttering a higher number of technical rather than vernacular terms (Boshuizen & Schmidt, 1992; Helle et al., 2011). If it is true that expertise alters the encapsulation of biomedical-declarative knowledge into clinical-experiential knowledge for routine cases, then it follows that biomedical knowledge should be verbalized more often in unfamiliar than in familiar tasks, for which in turn more technical terms should be used.

The present study aimed to explore the extent to which expertise in the comprehension of computer-based dynamic medical visualizations is domain general or domain specific. To assess transfer of expertise, we measured both diagnostic performance and the processes underlying diagnostic performance. The study makes a new contribution to the field because, while previous research on expertise in the comprehension of visualizations used static, two-dimensional stimuli, this study uses dynamic, three-dimensional stimuli; these visualizations were chosen largely because we wanted to address the current technological development in the medical imaging field. The study also combined eye tracking and think aloud methods, which, to the authors' knowledge, is a novel approach in research on expertise in interpreting medical visualizations. In the present study, we examined the transfer of expertise from PET and CT to PET/CT. Three hypotheses were formulated:

Hypothesis 1

The domain-general-skills hypothesis predicts that expertise in the familiar task transfers to the semi-familiar task (which shares surface and structural similarity) and to the unfamiliar task (which shares structural similarity only).

Hypothesis 2

The domain-specific-skills hypothesis predicts that expertise transfers to the semi-familiar task, but not to the unfamiliar task.

Hypothesis 3

Finally, the domain-specific-skills-in-context hypothesis predicts that expertise does transfer neither to the semi-familiar, nor to the unfamiliar task condition.

Transfer was measured as the difference between task conditions. The underlying rationale was that smaller differences from familiar to semi-familiar and unfamiliar tasks would indicate transfer, while larger differences would indicate lack of transfer.