Skip to main content
SpringerLink
Log in

Verbal reports and domain-specific knowledge: a comparison between collegial and retrospective verbalisation

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

Abstract

One way to investigate and account for the role of experience in dynamic decision-making tasks is to use a knowledge elicitation method, for example verbal protocols. Recently, methods for verbalisation by other subjects have been suggested as a way to elicit more information on thinking. One of these methods is here investigated in a quasi-experimental study on highly experienced and skilled train dispatchers. Collegial verbalisation is based on the procedure of videotaping practitioners while they perform their normal work tasks in their normal work setting. This is followed up by having a close colleague of the practitioner watch the video recordings and verbalise. The general hypothesis that verbal reports based on collegial verbalisations can provide protocol data that are close to the structure and content of verbal reports based on retrospective verbalisations was investigated. From this systematic comparison it is concluded that collegial verbalisations produce verbal protocols that are close to the retrospective verbal protocols on protocol and topic levels, but not on statement level. From this study, we conclude that the collegial protocols can be used as an independent source of data. It seems possible for a colleague to report verbally on a practitioners’ observable behaviour in the same way as when the practitioner is doing a retrospective verbalisation and it may also be possible for a colleague to explain some of the non-observable behaviour of the practitioner.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Arunachalam V, Sasso W (1996) Cognitive processes in program comprehension: an empirical analysis in the context of software reengineering. J Syst Softw 34:177–189

    Article  Google Scholar 

  • Bainbridge L (1979/1999) Verbal reports as evidence of the process operator’s knowledge. Int J Human Comput Stud 51:213–238

    Google Scholar 

  • Bartl C, Dörner D (1998) Speechless when thinking: about the role of speech for problem solving. Sprache und Kognition 17:224–238

    Google Scholar 

  • Bielaczyc K, Pirolli PL, Brown AL (1995) Training in self-explanation and self-regulation strategies: investigating the effects of knowledge acquisition activities on problem solving. Cogn Instr 13(2):221–252

    Article  Google Scholar 

  • Boren MT, Ramey J (2000) Thinking aloud: reconciling theory and practice. IEEE Trans Prof Commun 43(3):261–278

    Article  Google Scholar 

  • Brehmer B (1992) Dynamic decision making: human control of complex systems. Acta Psychol 81:211–241

    Article  Google Scholar 

  • Chi MTH, Bassok M, Lewis MW, Reimann P, Glaser R (1989) Self-explanations: how students study and use examples in learning to solve problems. Cogn Sci 13:145–182

    Article  Google Scholar 

  • Cook RI, Woods DD (1994) Operating at the sharp end: the complexity of human error. In: Bogner MS (ed) Human errors in medicine. Erlbaum, Hillsdale

    Google Scholar 

  • Cooke NJ (1999) Knowledge elicitation. In: Durso F, Nickerson RS, Schvaneveldt RW, Dumais ST, Lindsay DS, Chi MTH (eds) Handbook of applied cognition. Wiley, New York, pp 479–509

    Google Scholar 

  • D’astous P, Detienne F, Visser W, Robillard PN (2000) On the use of functional and interactional approaches for the analysis of technical review meetings. In: Blackwll AF, Bilotta E (eds) Proceedings of the twelfth annual conference of the Psychology of Programming Interest Group (PPIG 12), pp 155–170

  • Dickson J, McLennan J, Omodei MM (2000) Effects of concurrent verbalisation on a time-critical, dynamic decision making task. J Gen Psychol 127:217

    Google Scholar 

  • Ericsson KA (2001) Protocol analysis in psychology. In: Smelser N, Baltes P (eds) International encyclopedia of the social & behavioral sciences. Elsevier, Oxford, pp 12256–12262

    Chapter  Google Scholar 

  • Ericsson KA, Crutcher RJ (1991) Introspection and verbal reports on cognitive processes—two approaches to the study of thought processes: a response to Howe. New Ideas Psychol 9:57–71

    Article  Google Scholar 

  • Ericsson KA, Simon HA (1980) Verbal reports as data. Psychol Rev 87:215–251

    Article  Google Scholar 

  • Ericsson KA, Simon HA (1984) Protocol analysis: verbal reports as data. MIT Press, Cambridge

    Google Scholar 

  • Erlandsson M, Jansson A (2007) Collegial verbalisation—a case study on a new method on information acquisition. Behav Inf Technol 26(6):535–543

    Article  Google Scholar 

  • Gibbons FX (1983) Self-attention and self-report: the “veridicality” hypothesis. J Pers Assess 51:517–554

    Google Scholar 

  • Hammond KR (1993) Naturalistic decision making from a Brunswikian viewpoint: its past, present, future. In: Klein GA, Orasanu J, Calderwood R, Zsambok CE (eds) Decision making in action: models and methods. Ablex Publishing, Norwood

    Google Scholar 

  • Hammond KR (2000) Judgments under stress. Oxford University Press, New York

    Google Scholar 

  • Hughes J, Parkes S (2003) Trends in the use of verbal protocol analysis in software engineering research. Behav Inf Technol 22(2):127–140

    Article  Google Scholar 

  • Jansson A, Olsson E, Erlandsson M (2006) Bridging the gap between analysis and design: improving existing driver interfaces with tools from the framework of cognitive work analysis. Special Issue Int J Cogn Technol Work 8:41–49

    Article  Google Scholar 

  • Kauppi A, Wikström J, Sandblad B, Andersson AW (2006a) Future train traffic control: control be re-planning. Special issue Int J Cogn Technol Work 8:50–56

    Article  Google Scholar 

  • Kauppi A, Wikström J, Sandblad B, Andersson AW (2006b) Control strategies for managing train traffic—difficulties today and a vision for the future. Technical report 2006-24. Department of Information technology, Uppsala University, ISSN 1404-3203

  • Klein GA (1993) A recognition-primed decision (RPD) model of rapid decision making. In: Klein GA, Orasanu J, Calderwood R, Zsambok CE (eds) Decision making in action: models and methods. Ablex Publishing, Norwood

    Google Scholar 

  • Kuipers BJ, Kassirer JP (1983) How to discover a knowledge representation for causal reasoning by studying an expert physician. In: Proceedings of the American Association of Artificial Intelligence. Morgan Kauffmann Publishers, Inc., Los Altos

  • Kuusela H, Paul P (2000) A comparison of concurrent and retrospective verbal protocol analysis. Am J Psychol 113(3):387–404

    Article  Google Scholar 

  • Letovsky S, Pinto J, Lampert R, Soloway E (1987) A cognitive analysis of a code inspection. In: Olson G, Sheppard S, Soloway E (eds) Empirical studies of programmers. Second workshop. Ablex, Norwood, pp 231–247

  • McIlroy RC, Stanton NA (2010) Observing the observer: non-intrusive verbalisations using the concurrent observer narrative technique. Cogn Technol Work. doi:10.1007/s10111-010-0160-5

  • McNeese MD (2004) How video informs cognitive systems engineering: making experience count. Cogn Technol Work 6:186–196

    Article  Google Scholar 

  • Miller JE, Patterson ES, Woods DD (2006) Elicitation by critiquing as a cognitive task analysis methodology. Cogn Technol Work 8:90–102

    Article  Google Scholar 

  • Narayanan NH, Suwa M, Motoda H (1995) Diagram-based problem solving: the case of an impossible problem. In: Proceedings of the 17th annual conference of the Cognitive Science Society, Lawrence Erlbaum Associates, Hillsdale, pp 206–211

  • Nisbett RE, Wilson TD (1977) Telling more than we can know: verbal reports on mental processes. Psychol Rev 84:231–259

    Article  Google Scholar 

  • Pennington N (1987) Comprehension strategies in programming. In: Olson GM, Sheppard S, Soloway E (eds) Empirical studies of programmers, second workshop. Ablex, Norwood, pp 100–113

    Google Scholar 

  • Pennington N, Nicolich R, Rahm J (1995) Transfer of training between cognitive subskills: is knowledge use specific? Cogn Psychol 28:175–224

    Article  Google Scholar 

  • Pew RW, Miller DC, Feehrer CE (1981) Evaluation of proposed control room improvements through analysis of critical operator decisions, NP-1982. Electric Power Research Institute, Palo Alto

    Google Scholar 

  • Polanyi M (1974) Personal knowledge: towards a post-critical philosophy. University of Chicago Press, Chicago

    Google Scholar 

  • Rasmussen J (1983) Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern, SMC-13, 257–267

  • Rasmussen J, Pejtersen AM (1995) Virtual ecology of work. In: Flack J, Hancock P, Caird J, Vicente KJ (eds) Global perspectives on the ecology of human–machine systems. Lawrence Erlbaum Associates, Hillsdale, pp 121–156

    Google Scholar 

  • Reason J (1990) Human error. Cambridge University Press, Cambridge, pp 53–54

  • Sarter NB, Woods DD (1995) How in the world did we ever get into that mode? Mode error and awareness in supervisory control. Hum Factors 37(1):5–19

    Article  Google Scholar 

  • Schenk KD, Vitalari NP, Davis KS (1998) Differences between novice and expert systems analysts: what do we know and what do we do? J Manage Inf Syst 15(1):9–50

    Google Scholar 

  • Sonnentag S (1998) Expertise in professional software design: a process study. J Appl Psychol 83(5):703–715

    Article  Google Scholar 

  • Swann WB, Griffin JJ, Predmore SC, Gaines B (1987) The cognitive affective crossfire: when self-consistency confronts self-enhancements. J Pers Soc Psychol 52:881–889

    Article  Google Scholar 

  • van Der Veer GC (1993) Mental representations of computer languages—a lesson from practice. In: Lemut E, du Boulay B, Dettori G (eds) Cognitive models and intelligent environments for learning programming. Springer, Berlin, pp 21–33

    Google Scholar 

  • van Someren MW, Barnard YF, Sandberg JAC (1994) The think aloud method: a practical guide to modeling cognitive process. Academic Press, London

    Google Scholar 

  • Vessey I (1985) Expertise in debugging computer programs: a process analysis. Int J Man Mach Stud 23:459–494

    Article  Google Scholar 

  • Vicente KJ (1999) Cognitive work analysis: toward safe, productive, and healthy computer-based work. Lawrence Erlbaum Associates, Mahwah

    Google Scholar 

  • Whyte J IV, Cormier E, Pickett-Hauber R (2010) Cognitions associated with nurse performance: a comparison of concurrent and retrospective verbal reports of nurse performance in a simulated task environment. Int J Nurs Stud 47(4):446–451

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, Uppsala University, Box 337, 751 05, Uppsala, Sweden

    Mikael Erlandsson & Anders Jansson

Authors
  1. Mikael Erlandsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anders Jansson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mikael Erlandsson.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Erlandsson, M., Jansson, A. Verbal reports and domain-specific knowledge: a comparison between collegial and retrospective verbalisation. Cogn Tech Work 15, 239–254 (2013). https://doi.org/10.1007/s10111-012-0223-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-012-0223-x

Keywords

Search

Navigation