Skip to main content
SpringerLink
Log in

Development of instruments for assessment of individuals’ and teams’ non-technical skills in healthcare: a critical review

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

Abstract

Focus on patient safety has increased over the last decades; however, patient safety still relies on technology and the performance of healthcare professionals. Technology improves rapidly but despite numerous reports on how to improve the performance of healthcare professionals, this improvement has been more slow than expected. The performance of healthcare professionals is a product of each individual’s medical knowledge, technical and non-technical skills as well as the settings in which these skills are used. Training and the assessment of non-technical skills for individuals or teams in healthcare have shown to improve a safe and efficient performance. However, the implementation has been slow. One reason for this might be the shortage of assessment instrument to assess individuals’ and teams’ non-technical skills. In this paper, we review the development process of 23 instruments for assessment of non-technical skills for individuals or teams within healthcare. The instruments are mainly for use in the operating room or for teams handling emergency situations. Several of the instruments are developed based on a thorough needs analysis by a team having the same professional background as the target group. Most of the instruments consist of almost the same categories of non-technical skills and many use behavioural markers. Overall, the instruments have been validated to some extent, but there is room for improvement. However, there seems to be a lack of training of the raters. The importance of providing feedback after the assessment is emphasised. The criteria on which the behavioural markers are developed should undergo continuous changes, representing the development of the patient safety culture in healthcare organisations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Andersen PO, Jensen MK, Lippert A, Østergaard D, Klausen TW (2010) Development of a formative assessment tool for measurement of performance in multi-professional resuscitation teams. Resuscitation 81(6):703–711

    Article  Google Scholar 

  • Antonovsky A (1996) The salutogenic model as a theory to guide health promotion. Health Promot Int 11(1):11–18

    Article  Google Scholar 

  • Bateson G, Bateson MC (2000) Steps to an ecology of mind (vol 1972). University of Chicago Press Chicago. http://sspa.boisestate.edu/anthropology/files/2010/08/BATESON-Experiments-in-Thinking.pdf. Accessed 20 May 2014

  • Bested KM, Malling B, Skjelsager K, Østergaard D, Østergaard HT, Ringsted C (2011) Rater bias in postgraduate medical education. Ugeskr Laeger 173(44):2788–2790

    Google Scholar 

  • Cooper JB, Newbower RS, Long CD, McPeek B (2002) Preventable anesthesia mishaps: a study of human factors*. Qual Saf Health Care 11(3):277–282. doi:10.1136/qhc.11.3.277

    Article  Google Scholar 

  • Cooper S, Cant R, Porter J, Sellick K, Somers G, Kinsman L, Nestel D (2010) Rating medical emergency teamwork performance: development of the Team Emergency Assessment Measure (TEAM). Resuscitation 81(4):446–452

    Article  Google Scholar 

  • Corrigan JM (2005) Crossing the quality chasm. In: Reid PP, Compton WD, Grossman JH et al (eds) Building a better delivery system: a new engineering/health care partnership. National Academies Press, Washington, DC. Crossing the Quality Chasm. http://www.ncbi.nlm.nih.gov/books/NBK22857/. Accessed 20 Apr 2014

  • Crossingham GV, Sice PJA, Roberts MJ, Lam WH, Gale TCE (2012) Development of workplace‐based assessments of non‐technical skills in anaesthesia*. Anaesthesia 67(2):158–164. doi:10.1111/j.1365-2044.2011.06977.x

  • Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R (2003) Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 90(5):580–588

    Article  Google Scholar 

  • Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R (2004) Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cognit Technol Work 6(3). doi:10.1007/s10111-004-0158-y

  • Flin R, Patey R (2011) Non-technical skills for anaesthetists: developing and applying ANTS. Best Pract Res Clin Anaesthesiol 25(2):215–227. doi:10.1016/j.bpa.2011.02.005

    Article  Google Scholar 

  • Flin R, Martin L, Goeters K-M, Hörmann H-J, Amalberti R, Valot C, Nijhuis H (2005) Development of the NOTECHS (Non-Technical Skills) system for assessing pilots’ CRM skills. In: Harris D, Muir HC (eds) Contemporary issues in human factors and aviation safety. Ashgate, Aldershot, pp 133–154

  • Fuhrmann L, Østergaard D, Lippert A, Perner A (2009) A multi-professional full-scale simulation course in the recognition and management of deteriorating hospital patients. Resuscitation 80(6):669–673

    Article  Google Scholar 

  • Gaba David M (2000) Anaesthesiology as a model for patient safety in health care. BMJ 320(7237):785–788

    Article  Google Scholar 

  • Gaba David M, DeAnda A (1988) A comprehensive anesthesia simulation environment: re-creating the operating room for research and training. Anesthesiology 69(3):387

    Article  Google Scholar 

  • Gaba David M, Maxwell M, DeAnda A (1987) Anesthetic mishaps: breaking the chain of accident evolution. Anesthesiology 66(5):670

    Article  Google Scholar 

  • Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R (1998) Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 89(1):8

    Article  Google Scholar 

  • Glavin RJ (2011) Skills, training, and education. Simul Healthc 6(1):4–7. doi:10.1097/SIH.0b013e31820aa1ee

    Google Scholar 

  • Graham J, Hocking G, Giles E (2010) Anaesthesia Non-Technical Skills: can anaesthetists be trained to reliably use this behavioural marker system in 1 day? Br J Anaesth 104(4):440–445. doi:10.1093/bja/aeq032

    Article  Google Scholar 

  • Guise JM, Deering SH, Kanki BG, Osterweil P, Li H, Mori M, Lowe NK (2008) Validation of a tool to measure and promote clinical teamwork. Simul Healthc 3(4):217–223

    Article  Google Scholar 

  • Healey A, Undre S, Vincent C (2004) Developing observational measures of performance in surgical teams. Qual Saf Health Care 13(Suppl 1):i33–i40. doi:10.1136/qshc.2004.009936

    Article  Google Scholar 

  • Hodges B (2006) Medical education and the maintenance of incompetence. Med Teach 28(8):690–696

    Article  Google Scholar 

  • Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N (2011) Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg 212(2):234–243.e5. doi:10.1016/j.jamcollsurg.2010.11.001

  • Jepsen RMHG, Lyk-Jensen HT, Spanager L, Dieckmann P, Østergaard D (2012) Anaesthetists’ non-technical skills—adapting the system to another setting. In: Short communications, AMEE 2012

  • Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J (2006) A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Crit Care Med 34(8):2167–2174. doi:10.1097/01.CCM.0000229877.45125.CC

    Article  Google Scholar 

  • Klampfer B, Flin R, Helmreich RL, Häusler R, Sexton B, Fletcher G, Field P, Staender S, Lauche K, Dieckmann P, Amacher A (2001) Group interaction in high risk environments. Enhancing performance in high risk environments: recommendations for the use of behavioural markers. Retrieved from http://www.abdn.ac.uk/iprc/documents/ants/GIHRE21_rec_for_use_of_beh_markers.pdf. Accessed 20 May 2014

  • Klinect JR, Murray P, Merritt A, Helmreich R (2003) Line operations safety audit (LOSA): definition and operating characteristics. In: Proceedings of the 12th international symposium on aviation psychology, pp 663–668. Retrieved from https://www.faa.gov/about/initiatives/maintenance_hf/losa/publications/media/klinect_operatingcharacteristics2003.pdf. Accessed 20 Apr 2014

  • Kohn LT, Corrigan JM, Donaldson MS (1999) To err is human. Building a safer health system. National Academy Press, Washington, DC. http://books.nap.edu/openbook.php?record_id=9728. Accessed 20 Apr 2014

  • Kontogiannis T, Malakis S (2013) Strategies in coping with complexity: development of a behavioural marker system for air traffic controllers. Saf Sci 57:27–34

    Article  Google Scholar 

  • Lambden S, DeMunter C, Dowson A, Cooper M, Gautama S, Sevdalis N (2013) The imperial paediatric emergency training toolkit (IPETT) for use in paediatric emergency training: development and evaluation of feasibility and validity. Resuscitation 84(6):831–836

    Article  Google Scholar 

  • Lingard L, Reznick R, Espin S, Regehr G, DeVito I (2002) Team communications in the operating room: talk patterns, sites of tension, and implications for novices. Acad Med 77(3):232–237

    Article  Google Scholar 

  • Lyk-Jensen H, Jepsen RM, Spanager L, Dieckmann P, Østergaard D (2014) Assessing nurse anaesthetists’ non-technical skills in the operating room. Acta Anaesthesiol Scand 58:794–801

    Article  Google Scholar 

  • Malec JF, Torsher LC, Dunn WF, Wiegmann DA, Arnold JJ, Brown DA, Phatak V (2007) The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc 2(1):4–10

    Article  Google Scholar 

  • Manser T (2009) Teamwork and patient safety in dynamic domains of healthcare: a review of the literature. Acta Anaesthesiol Scand 53(2):143–151. doi:10.1111/j.1399-6576.2008.01717.x

    Article  Google Scholar 

  • Mishra A, Catchpole K, McCulloch P (2009) The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care 18(2):104–108

    Article  Google Scholar 

  • Morey JC, Simon R, Jay GD, Wears RL, Salisbury M, Dukes KA, Berns SD (2002) Error reduction and performance improvement in the emergency department through formal teamwork training: evaluation results of the MedTeams project. Health Serv Res 37(6):1553–1581

    Article  Google Scholar 

  • Morgan PJ, Tregunno D, Pittini R, Tarshis J, Regehr G, Desousa S et al (2012) Determination of the psychometric properties of a behavioural marking system for obstetrical team training using high-fidelity simulation. BMJ Qual Saf 21(1):78–82. doi:10.1136/bmjqs-2011-000296

    Article  Google Scholar 

  • Neily J, Mills PD, Young-Xu Y, Carney BT, West P, Berger DH et al (2010) Association between implementation of a medical team training program and surgical mortality. JAMA 304(15):1693–1700. doi:10.1001/jama.2010.1506

  • Nestel D, Walker K, Simon R, Aggarwal R, Andreatta P (2011) Nontechnical skills: an inaccurate and unhelpful descriptor? Simul Healthc 6(1):2

    Google Scholar 

  • Parker SH, Flin R, McKinley A, Yule S (2013) The Surgeons’ Leadership Inventory (SLI): a taxonomy and rating system for surgeons’ intraoperative leadership skills. Am J Surg 205(6):745–751

    Article  Google Scholar 

  • Plant JL, Van Schaik SM, Sliwka DC, Boscardin CK, O’Sullivan PS (2011) Validation of a self-efficacy instrument and its relationship to performance of crisis resource management skills. Adv Health Sci Educ 16(5):579–590

    Article  Google Scholar 

  • Rasmussen MB, Dieckmann P, Barry Issenberg S, Ostergaard D, Søreide E, Ringsted CV (2012) Long-term intended and unintended experiences after advanced life support training. Resuscitation. doi:10.1016/j.resuscitation.2012.07.030

    Google Scholar 

  • Schraagen JM, Schouten T, Smit M, Haas F, van der Beek D, van de Ven J, Barach P (2010) Assessing and improving teamwork in cardiac surgery. Qual Saf Health Care 19(6):1–6. doi:10.1136/qshc.2009.040105

    Google Scholar 

  • Sevdalis N, Davis R, Koutantji M, Undre S, Darzi A, Vincent CA (2008) Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg 196(2):184

    Article  Google Scholar 

  • Sevdalis N, Hull L, Birnbach DJ (2012) Improving patient safety in the operating theatre and perioperative care: obstacles, interventions, and priorities for accelerating progress. Br J Anaesth 109(Suppl 1):i3–i16. doi:10.1093/bja/aes391

    Article  Google Scholar 

  • Spanager L, Lyk-Jensen HT, Dieckmann P, Wettergren A, Rosenberg J, Ostergaard D (2012) Customization of a tool to assess Danish surgeons’ non-technical skills in the operating room. Dan Med J 59(11):A4526

    Google Scholar 

  • Spanager L, Beier-Holgersen R, Dieckmann P, Konge L, Rosenberg J, Oestergaard D (2013) Reliable assessment of general surgeons’ non-technical skills based on video-recordings of patient simulated scenarios. Am J Surg 206(5):810–817. doi:10.1016/j.amjsurg.2013.04.002

  • Steinemann S, Berg B, DiTullio A, Skinner A, Terada K, Anzelon K, Ho HC (2012) Assessing teamwork in the trauma bay: introduction of a modified “NOTECHS” scale for trauma. Am J Surg 203(1):69–75. doi:10.1016/j.amjsurg.2011.08.004

    Article  Google Scholar 

  • Thomas EJ, Sexton JB, Helmreich RL (2004) Translating teamwork behaviours from aviation to healthcare: development of behavioural markers for neonatal resuscitation. Qual Saf Health Care 13(Suppl_1):i57–i64. doi:10.1136/qshc.2004.009811

  • Undre S, Sevdalis N, Healey A, Darzi A, Vincent C (2007) Observational teamwork assessment for surgery (OTAS): refinement and application in urological surgery. World J Surg 31(7):1373–1381. doi:10.1007/s00268-007-9053-z

    Article  Google Scholar 

  • Walker S, Brett S, McKay A, Lambden S, Vincent C, Sevdalis N (2011) Observational skill-based clinical assessment tool for resuscitation (OSCAR): development and validation. Resuscitation 82(7):835–844. doi:10.1016/j.resuscitation.2011.03.009

    Article  Google Scholar 

  • Weller J, Frengley R, Torrie J, Shulruf B, Jolly B, Hopley L et al (2011) Evaluation of an instrument to measure teamwork in multidisciplinary critical care teams. BMJ Qual Saf 20(3):216–222. doi:10.1136/bmjqs.2010.041913

    Article  Google Scholar 

  • Westli HK, Johnsen BH, Eid J, Rasten I, Brattebø G (2010) Teamwork skills, shared mental models, and performance in simulated trauma teams: an independent group design. Scand J Trauma, Resusc Emerg Med 18(1):47. doi:10.1186/1757-7241-18-47

    Article  Google Scholar 

  • Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D (2006) Development of a rating system for surgeons’ non-technical skills. Med Educ 40(11):1098–1104. doi:10.1111/j.1365-2929.2006.02610.x

    Article  Google Scholar 

  • Yule Steven, Flin R, Maran N, Rowley D, Youngson G, Paterson-Brown S (2008) Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg 32(4):548–556. doi:10.1007/s00268-007-9320-z

    Article  Google Scholar 

  • Yule S, Rowley D, Flin R, Maran N, Youngson G, Duncan J, Paterson-Brown S (2009) Experience matters: comparing novice and expert ratings of non-technical skills using the NOTSS system. ANZ J Surg 79(3):154–160

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Danish Institute for Medical Simulation, Capital Region of Denmark and University of Copenhagen, Herlev Ringvej 75, 2730, Herlev, Denmark

    Rikke M. H. G. Jepsen, Doris Østergaard & Peter Dieckmann

Authors
  1. Rikke M. H. G. Jepsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Doris Østergaard
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter Dieckmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rikke M. H. G. Jepsen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jepsen, R.M.H.G., Østergaard, D. & Dieckmann, P. Development of instruments for assessment of individuals’ and teams’ non-technical skills in healthcare: a critical review. Cogn Tech Work 17, 63–77 (2015). https://doi.org/10.1007/s10111-014-0306-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-014-0306-y

Keywords

Search

Navigation