Skip to main content
SpringerLink
Log in

Automated video-based assessment of surgical skills for training and evaluation in medical schools

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Routine evaluation of basic surgical skills in medical schools requires considerable time and effort from supervising faculty. For each surgical trainee, a supervisor has to observe the trainees in person. Alternatively, supervisors may use training videos, which reduces some of the logistical overhead. All these approaches however are still incredibly time consuming and involve human bias. In this paper, we present an automated system for surgical skills assessment by analyzing video data of surgical activities.

Method

We compare different techniques for video-based surgical skill evaluation. We use techniques that capture the motion information at a coarser granularity using symbols or words, extract motion dynamics using textural patterns in a frame kernel matrix, and analyze fine-grained motion information using frequency analysis.

Results

We were successfully able to classify surgeons into different skill levels with high accuracy. Our results indicate that fine-grained analysis of motion dynamics via frequency analysis is most effective in capturing the skill relevant information in surgical videos.

Conclusion

Our evaluations show that frequency features perform better than motion texture features, which in-turn perform better than symbol-/word-based features. Put succinctly, skill classification accuracy is positively correlated with motion granularity as demonstrated by our results on two challenging video datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Dennis BM, Long EL, Zamperini KM, Nakayama DK (2013) The effect of the 16-hour intern workday restriction on surgical residents’ in-hospital activities. J Surg Educ 70(6):800–805

    Article  PubMed  Google Scholar 

  2. Awad S, Liscum K, Aoki N, Awad S, Berger D (2002) Does the subjective evaluation of medical student surgical knowledge correlate with written and oral exam performance? J Surg Res 104(1):36–39

    Article  CAS  PubMed  Google Scholar 

  3. Martin J, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (osats) for surgical residents. Br J Surg 84(2):273–278

    Article  CAS  PubMed  Google Scholar 

  4. Reznick R, MacRae H (2006) Teaching surgical skills-changes in the wind. N Engl J Med 355(25):2664

    Article  CAS  PubMed  Google Scholar 

  5. Yu T, Wheeler B, Hill A (2011) Clinical supervisor evaluations during general surgery clerkships. Med Teach 33(9):479–484

    Article  Google Scholar 

  6. Datta V, Bann S, Mandalia M, Darzi A (2006) The surgical efficiency score: a feasible, reliable, and valid method of skills assessment. Am J Surg 192(3):372–378

    Article  PubMed  Google Scholar 

  7. Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. BMJ Br Med J 327(7422):1032

    Article  Google Scholar 

  8. Twinanda AP, Shehata S, Mutter D, Marescaux J, de Mathelin M, Padoy N (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. arXiv preprint arXiv:1602.03012

  9. Lea C, Hager GD, Vidal R (2015) An improved model for segmentation and recognition of fine-grained activities with application to surgical training tasks. In: 2015 IEEE winter conference on applications of computer vision, pp 1123–1129

  10. Zia A, Sharma Y, Bettadapura V, Sarin EL, Clements MA, Essa I (2015) Automated assessment of surgical skills using frequency analysis. In: Medical image computing and computer-assisted intervention–MICCAI 2015. Springer, pp 430–438

  11. Sharma Y, Plötz T, Hammerla N, Mellor S, Roisin M, Olivier P, Deshmukh S, McCaskie A, Essa I (2014) Automated surgical OSATS prediction from videos. In: ISBI, IEEE

  12. Sharma Y, Bettadapura V, Plötz T, Hammerla N, Mellor S, McNaney R, Olivier P, Deshmukh S, McCaskie A, Essa I (2014) Video based assessment of OSATS using sequential motion textures. In: International workshop on modeling and monitoring of computer assisted interventions (M2CAI)-workshop

  13. Tao L, Zappella L, Hager GD, Vidal R (2013) Surgical gesture segmentation and recognition. In: Medical image computing and computer-assisted intervention–MICCAI 2013. Springer, pp 339–346

  14. Bettadapura V, Schindler G, Plötz T, Essa I (2013) Augmenting bag-of-words: data-driven discovery of temporal and structural information for activity recognition. In: IEEE CVPR

  15. Haro BB, Zappella L, Vidal R (2012) Surgical gesture classification from video data. In: MICCAI 2012. Springer, pp 34–41

  16. Zappella L, Béjar B, Hager G, Vidal R (2013) Surgical gesture classification from video and kinematic data. Med Image Anal 17(7):732–745

    Article  PubMed  Google Scholar 

  17. Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N (2012) Statistical modeling and recognition of surgical workflow. Med Image Anal 16(3):632–641

    Article  PubMed  Google Scholar 

  18. Lalys F, Riffaud L, Bouget D, Jannin P (2011) An application-dependent framework for the recognition of high-level surgical tasks in the or. In: Medical image computing and computer-assisted intervention—MICCAI 2011. Springer, pp 331–338

  19. Blum T, Feußner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: Medical image computing and computer-assisted intervention—MICCAI 2010. Springer, pp 400–407

  20. Lin H, Hager G (2009) User-independent models of manipulation using video contextual cues. In: International workshop on modeling and monitoring of computer assisted interventions (M2CAI)

  21. Judkins TN, Oleynikov D, Stergiou N (2009) Objective evaluation of expert and novice performance during robotic surgical training tasks. Surg Endosc 23(3):590–597

    Article  PubMed  Google Scholar 

  22. Pirsiavash H, Vondrick C, Torralba A (2014) Assessing the quality of actions. In: ECCV. Springer, pp 556–571

  23. Wang H, Kläser A, Schmid C, Liu CL (2013) Dense trajectories and motion boundary descriptors for action recognition. Int J Comput Vis 103(1):60–79

    Article  Google Scholar 

  24. Liu J, Kuipers B, Savarese S (2011) Recognizing human actions by attributes. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 3337–3344

  25. Niebles JC, Chen CW, Fei-Fei L (2010) Modeling temporal structure of decomposable motion segments for activity classification. In: Computer vision–ECCV 2010. Springer, pp 392–405

  26. Laptev I, Lindeberg T (2003) Space-time interest points. In: IN ICCV, pp 432–439

  27. Wang H, Ullah MM, Kläser A, Laptev I, Schmid C (2009) Evaluation of local spatio-temporal features for action recognition. In: BMVC

  28. Reiley C, Lin H, Yuh D, Hager G (2011) Review of methods for objective surgical skill evaluation. Surg Endosc 25(2):356–366

    Article  PubMed  Google Scholar 

  29. Reiley CE, Hager GD (2009) Decomposition of robotic surgical tasks: an analysis of subtasks and their correlation to skill. In: M2CAI workshop. MICCAI, London

Download references

Author information

Authors and Affiliations

  1. Georgia Tech College of Computing, 801, Atlantic Drive, Atlanta, GA, 30332, USA

    Aneeq Zia, Yachna Sharma, Vinay Bettadapura, Mark A. Clements & Irfan Essa

  2. Department of Surgery, Emory University, 1364 Clifton Road, NE, Atlanta, GA, 30322, USA

    Eric L. Sarin

  3. School of Computing Science, Newcastle University, Claremont Tower, Newcastle, UK

    Thomas Ploetz

Authors
  1. Aneeq Zia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yachna Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vinay Bettadapura
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eric L. Sarin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thomas Ploetz
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mark A. Clements
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Irfan Essa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aneeq Zia.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zia, A., Sharma, Y., Bettadapura, V. et al. Automated video-based assessment of surgical skills for training and evaluation in medical schools. Int J CARS 11, 1623–1636 (2016). https://doi.org/10.1007/s11548-016-1468-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-016-1468-2

Keywords

Search

Navigation