Skip to main content

Advertisement

SpringerLink
Log in

Objective assessment of operator performance during ultrasound-guided procedures

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Simulation permits objective assessment of operator performance in a controlled and safe environment. Image-guided procedures often require accurate needle placement, and we designed a system to monitor how ultrasound guidance is used to monitor needle advancement toward a target. The results were correlated with other estimates of operator skill.

Methods

The simulator consisted of a tissue phantom, ultrasound unit, and electromagnetic tracking system. Operators were asked to guide a needle toward a visible point target. Performance was video-recorded and synchronized with the electromagnetic tracking data. A series of algorithms based on motor control theory and human information processing were used to convert raw tracking data into different performance indices. Scoring algorithms converted the tracking data into efficiency, quality, task difficulty, and targeting scores that were aggregated to create performance indices. After initial feasibility testing, a standardized assessment was developed. Operators (N = 12) with a broad spectrum of skill and experience were enrolled and tested. Overall scores were based on performance during ten simulated procedures. Prior clinical experience was used to independently estimate operator skill.

Results

When summed, the performance indices correlated well with estimated skill. Operators with minimal or no prior experience scored markedly lower than experienced operators. The overall score tended to increase according to operator’s clinical experience. Operator experience was linked to decreased variation in multiple aspects of performance. The aggregated results of multiple trials provided the best correlation between estimated skill and performance. A metric for the operator’s ability to maintain the needle aimed at the target discriminated between operators with different levels of experience.

Conclusions

This study used a highly focused task model, standardized assessment, and objective data analysis to assess performance during simulated ultrasound-guided needle placement. The performance indices were closely related to operator experience.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Dawson S (2006) Procedural simulation: a primer. J Vasc Interv Radiol 17: 205–213

    Article  PubMed  Google Scholar 

  2. Dawson S, Gould DA (2007) Procedural simulation’s developing role in medicine. Lancet 369: 1671–1673

    Article  PubMed  Google Scholar 

  3. Gaba DM (2004) The future vision of simulation in health care. Qual Saf Health Care 13(Suppl 1): i2–i10

    Article  PubMed  Google Scholar 

  4. Rodriguez-Paz JM, Kennedy M, Salas E et al (2009) Beyond “see one, do one, teach one”: toward a different training paradigm. Qual Saf Health Care 18: 63–68

    PubMed  CAS  Google Scholar 

  5. Kubiszyn T, Borich G (2007) Educational testing and measurement, 8th edn. Wiley, Hoboken

    Google Scholar 

  6. Swanson DB, Norman GR, Linn RL (1995) Performance-based assessment: lessons from the health professions. Educ Res 24: 5–11

    Google Scholar 

  7. Clauser BE (2000) Recurrent issues and recent advances in scoring performance assessments. Appl Psychol Meas 24: 310–324

    Article  Google Scholar 

  8. Christensen CM, Grossman JH, Hwang J (2009) The Innovator’s prescription. McGraw-Hill Professional, New York

    Google Scholar 

  9. Ericsson KA, Charness N, Feltovich PJ, Hoffman RR (2006) The Cambridge handbook of expertise and expert performance. Cambridge University Press, Cambridge

    Google Scholar 

  10. Teplitz CJ (1991) The learning curve deskbook. Quorum Books, New York

    Google Scholar 

  11. Williams AM, Hodges NJ (2004) Skill acquisition in sport: research, theory and practice. Routledge, London

    Google Scholar 

  12. Jonassen D, Hannum W, Tessmer M (1989) Handbook of task analysis procedures. Praeger Publishers, Westport

    Google Scholar 

  13. Beta E, Parikh AS, Street M, Duncan JR (2009) Capture and analysis of data from image-guided procedures. J Vasc Interv Radiol 20: 769–781

    Article  PubMed  Google Scholar 

  14. Kendall JL, Faragher JP (2007) Ultrasound-guided central venous access: a homemade phantom for simulation. Can J Emerg Med 9: 371–373

    Google Scholar 

  15. Duncan JR, Glaiberman CB (2006) Analysis of simulated angiographic procedures. Part 1: capture and presentation of audio and video recordings. J Vasc Interv Radiol 17: 1979–1989

    Article  PubMed  Google Scholar 

  16. Fitts P (1954) The information capacity of the human motor system in controlling the amplitude of movement. J Exp Psychol 47: 381–391

    Article  PubMed  CAS  Google Scholar 

  17. MacKenzie I (1992) Fitts’ law as a research and design tool in human-computer interaction. Hum Comput Interact 7: 91–139

    Article  Google Scholar 

  18. Hick WE (1952) On the rate of gain of information. Q J Exp Psychol 4: 11–26

    Article  Google Scholar 

  19. Hyman R (1953) Stimulus information as a determinant of reaction time. J Exp Psychol 45: 188–196

    Article  PubMed  CAS  Google Scholar 

  20. Pierce JR (1980) An introduction to information theory: symbols, signals & noise, 2nd rev. edn. Dover Publications, New York

    Google Scholar 

  21. Shannon CE, Weaver W (1998) The mathematical theory of communication. University of Illinois Press, Urbana

    Google Scholar 

  22. Mislevy RJ, Steinberg LS, Almond RG (2003) On the structure of educational assessments. Measurement 1: 3–62

    Google Scholar 

  23. Bucholz EI, Duncan JR (2008) Assessing system performance. J Vasc Interv Radiol 19: 987–994

    Article  PubMed  Google Scholar 

  24. Schmidt R, Lee T (2005) Motor control and learning: a behavioral emphasis, 4th edn. Human Kinetics, Champaign

    Google Scholar 

  25. Jacobs B, Duncan JR (2009) Improving quality and patient safety by minimizing unnecessary variation. J Vasc Interv Radiol 20: 157–163

    Article  PubMed  Google Scholar 

  26. Montgomery D (2005) Introduction to statistical quality control, 5th edn. Wiley, Hoboken

    Google Scholar 

  27. Embretson SE, Reise SP (2000) Item response theory for psychologists. Lawrence Erlbaum Associates, Mahwah

    Google Scholar 

  28. Kane M (2006) In: Brennan R (ed) Validation, educational measurement. Praeger Publishers, Westport pp 17–64

  29. Kane M (2004) Certification testing as an illustration of argument-based validation. Measurement 2: 135–170

    Google Scholar 

  30. Battles JB (2006) Improving patient safety by instructional systems design. Qual Saf Health Care 15(Suppl 1): i25–i29

    Article  PubMed  Google Scholar 

  31. Messick S (1993) In: Linn RL (ed) Validity, educational measurement. Oryx Press, Phoenix pp 13–104

  32. Wainer H, Braun H (1988) Test validity. Lawrence Erlbaum Associates, Hillsdale

    Google Scholar 

  33. Gould DA, Reekers JA, Kessel DO et al (2006) Simulation devices in interventional radiology: validation pending. J Vasc Interv Radiol 17: 215–216

    Article  PubMed  Google Scholar 

  34. Mislevy RJ (1992) Linking educational assessments: concepts, issues, methods and prospects. Policy Information Center, Educational Testing Service, Princeton

    Google Scholar 

  35. Argote L (1999) Organizational learning: creating, retaining, and transferring. Kluwer Academic Publishers, Norwell

    Google Scholar 

  36. Speelman CP, Kirsner K (2005) Beyond the learning curve. Oxford University Press, New York

    Book  Google Scholar 

  37. Heathcote A, Brown S, Mewhort D (2000) The power law repealed: the case for an exponential law of practice. Psychol Bull Rev 7: 185–207

    Article  CAS  Google Scholar 

  38. Duncan JR, Evens RG (2009) Using information to optimize medical outcomes. JAMA 301: 2383–2385

    Article  PubMed  CAS  Google Scholar 

  39. Ericsson K (2004) Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 79: S70–S81

    Article  PubMed  Google Scholar 

  40. Bloom B (1984) The 2 sigma problem: the search for methods of group instruction as effective as one-to-one tutoring. Educ Res 13: 4–16

    Google Scholar 

  41. Clark R, Nguyen F, Sweller J (2006) Efficiency in learning: evidence-based guidelines to manage cognitive load. Pfeiffer, San Francisco

    Google Scholar 

  42. Clark R (2003) Building expertise cognitive. Methods for training and performance improvement, 2nd edn. International Society for Performance Improvement, Silver Spring

    Google Scholar 

  43. Gagne R, Wager W, Golas K, Keller J (2005) Principles of instructional design, 5th edn. Wadsworth/Thomson Learning, Belmont

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Rush Medical College, Chicago, IL, USA

    David M. Tabriz

  2. Mallinckrodt Institute of Radiology, Washington University School of Medicine, 510 S. Kingshighway Blvd, St. Louis, MO, 63110, USA

    Mandie Street, Thomas K. Pilgram & James R. Duncan

Authors
  1. David M. Tabriz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mandie Street
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas K. Pilgram
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. James R. Duncan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James R. Duncan.

Additional information

The authors report that an abstract describing this work was presented at the 2009 Society of Interventional Radiology Annual Meeting. The work was also presented at the CARS 2010 in Geneva.

Electronic Supplementary Material

The Below is the Electronic Supplementary Material.

ESM 1 (PDF 424 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tabriz, D.M., Street, M., Pilgram, T.K. et al. Objective assessment of operator performance during ultrasound-guided procedures. Int J CARS 6, 641–652 (2011). https://doi.org/10.1007/s11548-010-0541-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-010-0541-5

Keywords

Search

Navigation