Measuring the structure of expertise

doi:10.1016/S0020-7373(85)80064-X

International Journal of Man-Machine Studies

Volume 23, Issue 6, December 1985, Pages 699-728

https://doi.org/10.1016/S0020-7373(85)80064-X Get rights and content

This report reviews work on defining and measuring conceptual structures of expert and novice fighter pilots. Individuals with widely varying expertise were tested. Cognitive structures were derived using multidimensional scaling (MDS) and link-weighted networks (Pathfinder). Experience differences among pilots were reflected in the conceptual structures. Detailed analyses of individual differences point to factors that distinguish experts and novices. Analysis of individual concepts identified areas of agreement and disagreement in the knowledge structures of experts and novices. Applications in selection, training and knowledge engineering are discussed.

References (30)

W.G. Chase et al.
Perception in chess
Cognitive Psychology
(1973)
M.T.H. Chi et al.
Categorization and representation of physics problems by experts and novices
Cognitive Science
(1981)
A.M. Collins et al.
Retrieval time from semantic memory
Journal of Verbal Learning and Verbal Behavior
(1969)
M.L. Friendly
In search of the M-gram: the structure of organization in free recall
Cognitive Psychology
(1977)
K.B. McKeithen et al.
Knowledge organization and skill differences in computer programmers
Cognitive Psychology
(1981)
J.S. Reitman
Skilled perception in Go: deducing memory structures from interresponse times
Cognitive Psychology
(1976)
L.J. Rips et al.
Semantic distance and the verification of semantic relations
Journal of Verbal Learning and Verbal Behavior
(1973)
B. Adelson
Problem solving and the development of abstract categories in programming languages
Memory and Cognition
(1981)
J.D. Carroll et al.
Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition
Psychometric
(1970)
N. Christofides

A.M. Collins et al.

A spreading activation theory of semantic processing

Psychological Review

(1975)

N.M. Cooke

Memory structures of expert and novice computer programmers: Recall order vs. similarity ratings

(1983)

F.T. Durso et al.

Empirical networks of natural categories

R.W. Engle et al.

Memory processes among bridge players of differing expertise

American Journal of Psychology

(1978)

S. Fillenbaum et al.

Cited by (139)

Parameterisation of domain knowledge for rapid and iterative prototyping of knowledge-based systems
2022, Expert Systems with Applications
In critical infrastructure applications, timely and consistent fault detection and diagnosis is an increasingly important operational process, especially in the energy sector where safety is of the utmost importance. To realise this, engineers have to manually analyse data acquired from several assets using predefined diagnostic processes, but this is a time-consuming process requiring significant amounts of specialist expert knowledge. Data-driven approaches to support fault detection and diagnosis, and other similar problems, can produce accurate results comparable to what the engineers can achieve in a fraction of the time. However, the majority of these data-driven techniques are black box techniques and lack explainability which is often necessary for explaining decisions about critical assets in the power generation industry. Knowledge-based systems, such as rule-based expert systems have been shown to provide not only accurate decisions but also the explanation and reasoning behind these decisions in some related applications. However, there is a significant time cost associated with the development of knowledge-based systems, and in particular with the knowledge elicitation process, where the domain expert’s knowledge is formalised and is encoded into the system. This challenge is commonly referred to as the knowledge elicitation bottleneck.
In this paper, we present a novel approach to performing the knowledge elicitation using a set of symbolic primitives (rise, fall, fluctuate, and stable) to parameterise typical time-series condition monitoring data. The knowledge is represented by using a common language that can easily be communicated with (and from) the domain experts. This allows for the quick and accurate elicitation of the domain experts knowledge, but also the formalisation and implementation of the knowledge into a rapidly produced diagnostic system. Further to this, due to the parametrisation of the knowledge, it is possible to iteratively improve the knowledge base by updating these parameters based on new unseen data. This approach was applied to the Tennessee Eastman dataset, which is simulated data of a real-world industrial process. It was found that by using this approach it was possible to accurately and quickly capture the knowledge required to detect several faults within the case study dataset, but also provided fully explained reasons why each fault was detected by relating the explanations to the symbolic primitives previously defined.
Information processing in aviation
2022, Human Factors in Aviation and Aerospace, Third Edition
In this chapter, we represent the pilot as an information processing system. We first describe breakdowns in pilot information processing as illustrated by four tragic accidents. We then present a framework for information processing and discuss how technological developments in aviation have influenced pilot information processing. Then, in separate sections we discuss information in aviation as applied to:
- •
  manual (flight) control,
- •
  communications & working memory,
- •
  performance measurement (the speed-accuracy tradeoff),
- •
  mental workload and SA,
- •
  attention in multi-tasking,
- •
  expertise,
- •
  decision making.
Critical aspects of pilot visual attention, perception, and spatial cognition are described in Chapter 6, and more elaborate treatments of workload and situation awareness and of pilot DM are described in Chapters 7 and 21, respectively.
Using Pathfinder networks to discover alignment between expert and consumer conceptual knowledge from online vaccine content
2017, Journal of Biomedical Informatics
Citation Excerpt :
For example, increased knowledge of vaccines may improve vaccine perception and uptake [15–17]. Due to socio-cultural differences, educational or training pedigree, expert and consumer representations of knowledge may differ [18]. This has implications in the health domain, and therefore, needs to be examined if health care-related artifacts are to be targeted for particular audiences [19].
This study demonstrates the use of distributed vector representations and Pathfinder Network Scaling (PFNETS) to represent online vaccine content created by health experts and by laypeople. By analyzing a target audience’s conceptualization of a topic, domain experts can develop targeted interventions to improve the basic health knowledge of consumers. The underlying assumption is that the content created by different groups reflects the mental organization of their knowledge. Applying automated text analysis to this content may elucidate differences between the knowledge structures of laypeople (heath consumers) and professionals (health experts). This paper utilizes vaccine information generated by laypeople and health experts to investigate the utility of this approach. We used an established technique from cognitive psychology, Pathfinder Network Scaling to infer the structure of the associational networks between concepts learned from online content using methods of distributional semantics. In doing so, we extend the original application of PFNETS to infer knowledge structures from individual participants, to infer the prevailing knowledge structures within communities of content authors. The resulting graphs reveal opportunities for public health and vaccination education experts to improve communication and intervention efforts directed towards health consumers. Our efforts demonstrate the feasibility of using an automated procedure to examine the manifestation of conceptual models within large bodies of free text, revealing evidence of conflicting understanding of vaccine concepts among health consumers as compared with health experts. Additionally, this study provides insight into the differences between consumer and expert abstraction of domain knowledge, revealing vaccine-related knowledge gaps that suggest opportunities to improve provider-patient communication.
Where’s Waldo, Ohio? Using Cognitive Models to Improve the Aggregation of Spatial Knowledge
2024, Computational Brain and Behavior
Mapping the minds of spectators during an extreme ritual: a network perspective
2024, Religion, Brain and Behavior
Exploring the Relationships within Product Meanings
2023, SSRN

View all citing articles on Scopus

^‡: Visiting professor from the University of Oklahoma.

^§: Now at the University of Mexico.

View full text

Measuring the structure of expertise

Cognitive Psychology

Cognitive Science

Journal of Verbal Learning and Verbal Behavior

Cognitive Psychology

Cognitive Psychology

Cognitive Psychology

Journal of Verbal Learning and Verbal Behavior

Problem solving and the development of abstract categories in programming languages

Memory and Cognition

Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition

Psychometric

A spreading activation theory of semantic processing

Psychological Review

Memory structures of expert and novice computer programmers: Recall order vs. similarity ratings

Empirical networks of natural categories

Memory processes among bridge players of differing expertise

American Journal of Psychology