Abstract
This paper describes elicitation by critiquing (EBC) as a cognitive task analysis (CTA) methodology. EBC takes advantage of the ability to analyze another’s task performance, a necessary skill for all domains. This technique can be used to help address some barriers to CTA methods such as domain access restrictions, frequency and predictability of observable and self-reported events, and difficulties in recruiting domain experts to participate. The technique enables controlled presentation of problem stimuli in order to obtain repeated measuring of the same task from multiple participants. To investigate this method, our team performed a CTA of inferential analysis using the EBC technique. Specifically, we observed six expert intelligence analysts critiquing a trainee analyzing the Ariane 501 launch failure. A second trainee was critiqued for reference. The method can be combined with other CTA methods to build knowledge about a domain, can be considered as an addition to participatory design methods, and can be varied depending on the domain being investigated.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ericsson K, Simon H (1993) Protocol analysis: verbal reports as data. MIT Press, Cambridge
Fowlkes JE, Salas E, Baker DP, Cannon-Bowers JA, Stout RJ (2000) The utility of event-based knowledge elicitation. Hum Factors 42(1):24–35
Gordon SE, Gill RT (1997) Cognitive task analysis. In: Zsambok C, Klein G (eds) Naturalistic decision making. Lawrence Erlbaum Associates, Mahwah, pp 131–140
Hoffman RR, Shadbolt NR, Burton AM, Klein G (1995) Eliciting knowledge from experts: a methodological analysis. Organ Behav Hum Decis Process 62(2):129–158
Hoffman RR, Coffey JW, Carnot MJ, Novack JD (2002) An empirical comparison of methods for eliciting and modeling expertise. In: Proceedings of the Human Factors and Ergonomics Society 46th annual meeting, Baltimore, MD, pp 482–486
Jorgensen DL (1989) Participant observation: a methodology for human studies. Sage Publications, London
Kensing F, Blomberg J (eds) (1998) Comput Support Coop Work (special issue on participatory design) 7(3–4)
Klein GA, Calderwood R, MacGregor D (1989) Critical decision method for eliciting knowledge. IEEE Trans Syst Man Cybern 19:462–472
Madsen KH (1999) Commun ACM (special issue on usability in Scandinavia and the US) 42(5)
Militello LG, Hutton RJB, Pliske RM, Knight BJ, Klein G (1997) Applied cognitive task analysis (ACTA) methodology. Final technical report prepared for the Navy Personnel Research and Development Center under Contract No. N66001-94-C-7034. Klein Associates, Fairborn
Monk AF, Wright PC, Haber J, Davenport L (1993) Improving your human–computer interface: a practical technique. Prentice-Hall, Englewood Cliffs
Muller MJ, Kuhn S (1993) Commun ACM (special issue on participatory design) 36(6)
Patterson ES, Roth EM, Woods DD (2001) Predicting vulnerabilities in computer-supported inferential analysis under data overload. Cogn Technol Work 3(4):224–237
Patterson ES, Rogers ML, Render ML (2004) Simulation-based embedded probe technique for artifact analysis. Cogn Technol Work 6(3):197–205
Potter SS, Roth EM, Woods DD, Elm W (1998) Toward the development of a computer-aided cognitive engineering tool to facilitate the development of advanced decision support systems for information warfare. Prepared for Armstrong Laboratory Crew Systems Integration Branch (AL/CFHI), Wright-Patterson AFB, Ohio
Potter SS, Roth EM, Woods DD, Elm W (2000) Bootstrapping multiple converging cognitive task analysis techniques for system design. In: Schraagen JMC, Chipman SF, Shalin VL (eds) Cognitive task analysis. Lawrence Erlbaum Associates, Mahwah
Sarter NB, Woods DD (2000) Teamplay with a powerful and independent agent: a full-mission simulation study. Hum Factors 42(3):390–402
Schon DA (1983) The reflective practitioner: how professionals think in action. Basic Books, New York
Schraagen JMC, Chipman SF, Shalin VL (eds) (2000) Cognitive task analysis. Lawrence Erlbaum Associates, Mahwah
Silverman BG (1991) Criticism based knowledge acquisition for document generation. Innovative applications of AI. AAA Press, Menlo Park, pp 291–319
Smith PJ, Obradovich JH, Guerlain SA, Rudman S, Strohm P, Smith JW, Svirbely J, Sachs L (1998) Successful use of an expert system to teach diagnostic reasoning for antibody identification. In: Goettle, Halff, Redfield, Shute (eds) Intelligent tutoring systems, 4th international conference, ‘IT ‘98, San Antonio, TX August 16–19, 1998, pp 354–363
Spradley J (1980) Participant observation. Holt, Rinehart & Winston, New York
Woods DD (1993) Process tracing methods for the study of cognition outside of the experimental psychology laboratory. In: Klein G, Orasanu J, Calderwood R (eds) Decision making in action: models and methods. Ablex Publishing Corporation, Norwood, pp 228–251
Wright PC, Monk AF (1991) The use of think-aloud evaluation methods in design. SIGCHI Bull 23(1):55–57
Acknowledgements
David B. Sanders provided invaluable insight into information retrieval issues at the National Air and Space Intelligence Center and was a constant source of intellectual stimulation. We thank the study expert participants for donating their time and expertise, and especially the trainee participants who were game to be investigated. We are also deeply appreciative to the National Air and Space Intelligence Center for supporting the research.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Miller, J., Patterson, E. & Woods, D. Elicitation by critiquing as a cognitive task analysis methodology. Cognition,Technology & Work 8, 90–102 (2006). https://doi.org/10.1007/s10111-005-0023-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10111-005-0023-7