Skip to main content

Advertisement

SpringerLink
Log in

Objective classification of psychomotor laparoscopic skills of surgeons based on three different approaches

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Background

The determination of surgeons’ psychomotor skills in minimally invasive surgery techniques is one of the major concerns of the programs of surgical training in several hospitals. Therefore, it is important to assess and classify objectively the level of experience of surgeons and residents during their training process. The aim of this study was to investigate three classification methods for establishing automatically the level of surgical competence of the surgeons based on their psychomotor laparoscopic skills.

Methods

A total of 43 participants, divided into an experienced surgeons group with ten experts (> 100 laparoscopic procedures performed) and non-experienced surgeons group with 24 residents and nine medical students (< 10 laparoscopic procedures performed), performed three tasks in the EndoViS training system. Motion data of the instruments were captured with a video-tracking system built into the EndoViS simulator and analyzed using 13 motion analysis parameters (MAPs). Radial basis function networks (RBFNets), K-star (K*), and random forest (RF) were used for classifying surgeons based on the MAPs’ scores of all participants. The performance of the three classifiers was examined using hold-out and leave-one-out validation techniques.

Results

For all three tasks, the K-star method was superior in terms of accuracy and AUC in both validation techniques. The mean accuracy of the classifiers was 93.33% for K-star, 87.58% for RBFNets, and 84.85% for RF in hold-out validation, and 91.47% for K-star, 89.92% for RBFNets, and 83.72% for RF in leave-one-out cross-validation.

Conclusions

The three proposed methods demonstrated high performance in the classification of laparoscopic surgeons, according to their level of psychomotor skills. Together with motion analysis and three laparoscopic tasks of the Fundamental Laparoscopic Surgery Program, these classifiers provide a means for objectively classifying surgical competence of the surgeons for existing laparoscopic box trainers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Cuschieri A (2005) Laparoscopic surgery: current status, issues and future developments. Surgeon 3:125–130

    Article  CAS  Google Scholar 

  2. Ritter EM, Scott DJ (2007) Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov 14:107–112. https://doi.org/10.1177/1553350607302329

    Article  PubMed  Google Scholar 

  3. van Hove PD, Tuijthof GJ, Verdaasdonk EG, Stassen LP, Dankelman J (2010) Objective assessment of technical surgical skills. Br J Surg 97:972–987. https://doi.org/10.1002/bjs.7115

    Article  PubMed  Google Scholar 

  4. Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. BMJ 327:1032–1037. https://doi.org/10.1136/bmj.327.7422.1032

    Article  PubMed  PubMed Central  Google Scholar 

  5. Harrysson I, Hull L, Sevdalis N, Darzi A, Aggarwal R (2014) Development of a knowledge, skills, and attitudes framework for training in laparoscopic cholecystectomy. Am J Surg 207:790–796. https://doi.org/10.1016/j.amjsurg.2013.08.049

    Article  PubMed  Google Scholar 

  6. Chipman JG, Schmitz CC (2009) Using objective structured assessment of technical skills to evaluate a basic skills simulation curriculum for first-year surgical residents. J Am Coll Surg 209:364–370.e2. https://doi.org/10.1016/j.jamcollsurg.2009.05.005

    Article  PubMed  Google Scholar 

  7. Seymour NE (2008) VR to OR: a review of the evidence that virtual reality simulation improves operating room performance. World J Surg 32:182–188. https://doi.org/10.1007/s00268-007-9307-9

    Article  PubMed  Google Scholar 

  8. Sturm LP, Windsor JA, Cosman PH, Cregan P, Hewett PJ, Maddern GJ (2008) A systematic review of skills transfer after surgical simulation training. Ann Surg 248:166–179. https://doi.org/10.1097/SLA.0b013e318176bf24

    Article  PubMed  Google Scholar 

  9. Bansal VK, Raveendran R, Misra MC, Bhattacharjee H, Rajan K, Krishna A, Kumar P, Kumar S (2014) A prospective randomized controlled blinded study to evaluate the effect of short-term focused training program in laparoscopy on operating room performance of surgery residents (CTRI /2012/11/003113). J Surg Educ 71:52–60. https://doi.org/10.1016/j.jsurg.2013.06.012

    Article  PubMed  Google Scholar 

  10. Uemura M, Tomikawa M, Kumashiro R, Miao T, Souzaki R, Ieiri S, Ohuchida K, Lefor AT, Hashizume M (2014) Analysis of hand motion differentiates expert and novice surgeons. J Surg Res 188:8–13. https://doi.org/10.1016/j.jss.2013.12.009

    Article  PubMed  Google Scholar 

  11. Hagelsteen K, Sevonius D, Bergenfelz A, Ekelund M (2016) Simball box for laparoscopic training with advanced 4D motion analysis of skills. Surg Innov 23:309–316. https://doi.org/10.1177/1553350616628678

    Article  PubMed  Google Scholar 

  12. Chmarra MK, Bakker NH, Grimbergen CA, Dankelman J (2006) TrEndo, a device for tracking minimally invasive surgical instruments in training setups. Sens Actuators A Phys 126:328–334. https://doi.org/10.1016/j.sna.2005.10.040

    Article  CAS  Google Scholar 

  13. Pérez-Escamirosa F, Chousleb-Kalach A, Hernández-Baro MD, Sánchez-Margallo JA, Lorias-Espinoza D, Minor-Martínez A (2016) Construct validity of a video-tracking system based on orthogonal cameras approach for objective assessment of laparoscopic skills. Int J Comput Assist Radiol Surg 11:2283–2293. https://doi.org/10.1007/s11548-016-1388-1

    Article  PubMed  Google Scholar 

  14. Oropesa I, Sánchez-González P, Chmarra MK, Lamata P, Fernández A, Sánchez-Margallo JA, Jansen FW, Dankelman J, Sánchez-Margallo FM, Gómez EJ (2013) EVA: laparoscopic instrument tracking based on endoscopic video analysis for psychomotor skills assessment. Surg Endosc 27:1029–1039. https://doi.org/10.1007/s00464-012-2513-z

    Article  PubMed  Google Scholar 

  15. Oropesa I, Chmarra MK, Sánchez-González P, Lamata P, Rodrigues SP, Enciso S, Sánchez-Margallo FM, Jansen FW, Dankelman J, Gómez EJ (2013) Relevance of motion-related assessment metrics in laparoscopic surgery. Surg Innov 20:299–312. https://doi.org/10.1177/1553350612459808

    Article  PubMed  Google Scholar 

  16. Sánchez-Margallo JA, Sánchez-Margallo FM, Oropesa I, Enciso S, Gómez EJ (2016) Objective assessment based on motion-related metrics and technical performance in laparoscopic suturing. Int J Comput Assist Radiol Surg. https://doi.org/10.1007/s11548-016-1459-3

    Article  PubMed  Google Scholar 

  17. Escamirosa FP, Flores RMO, García IO, Vidal CR, Martínez AM (2015) Face, content, and construct validity of the EndoViS training system for objective assessment of psychomotor skills of laparoscopic surgeons. Surg Endosc Other Interv Tech 29:3392–3403. https://doi.org/10.1007/s00464-014-4032-6

    Article  Google Scholar 

  18. Chmarra MK, Grimbergen CA, Jansen F-W, Dankelman J (2010) How to objectively classify residents based on their psychomotor laparoscopic skills? Minim Invasive Ther Allied Technol 19:2–11. https://doi.org/10.3109/13645700903492977

    Article  PubMed  Google Scholar 

  19. Chmarra MK, Klein S, de Winter JC, Jansen FW, Dankelman J (2010) Objective classification of residents based on their psychomotor laparoscopic skills. Surg Endosc 24:1031–1039. https://doi.org/10.1007/s00464-009-0721-y

    Article  PubMed  Google Scholar 

  20. Oropesa I, Sánchez-González P, Chmarra MK, Lamata P, Pérez-Rodríguez R, Jansen FW, Dankelman J, Gómez EJ (2014) Supervised classification of psychomotor competence in minimally invasive surgery based on instruments motion analysis. Surg Endosc 28:657–670. https://doi.org/10.1007/s00464-013-3226-7

    Article  PubMed  Google Scholar 

  21. Gao Y, Vedula SS, Reiley CE, Ahmidi N, Varadarajan B, Lin HC, Tao L, Zappella L, Béjar B, Yuh DD, Chen CCG, Vidal R, Khudanpur S, Hager GD (2014) JHU-ISI gesture and skill assessment working set (JIGSAWS): a surgical activity dataset for human motion modeling. In: MICCAI workshop: M2CAI, vol 3, p 3

  22. Zia A, Essa I (2018) Automated surgical skill assessment in RMIS training. Int J Comput Assist Radiol Surg 13(5):731–739

    Article  Google Scholar 

  23. Ismail Fawaz H, Forestier G, Weber J, Idoumghar L, Muller PA (2018) Evaluating surgical skills from kinematic data using convolutional neural networks. In: MICCAI, pp 214–221

  24. Wang Z, Majewicz Fey A (2018) Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery. Int J Comput Assist Radiol Surg 13(12):1959–1970

    Article  Google Scholar 

  25. Funke I, Mees ST, Weitz J, Speidel S (2019) Video-based surgical skill assessment using 3D convolutional neural networks. Int J Comput Assist Radiol Surg 14(7):1217–1225. https://doi.org/10.1007/s11548-019-01995-1

    Article  PubMed  Google Scholar 

  26. Allen B, Nistor V, Dutson E, Carman G, Lewis C, Faloutsos P (2010) Support vector machines improve the accuracy of evaluation for the performance of laparoscopic training tasks. Surg Endosc 24:170–178. https://doi.org/10.1007/s00464-009-0556-6

    Article  PubMed  Google Scholar 

  27. Fried GM (2008) FLS assessment of competency using simulated laparoscopic tasks. J Gastrointest Surg 12:210–212. https://doi.org/10.1007/s11605-007-0355-0

    Article  PubMed  Google Scholar 

  28. Wenger L, Richardson C, Tsuda S (2015) Retention of fundamentals of laparoscopic surgery (FLS) proficiency with a biannual mandatory training session. Surg Endosc 29:810–814. https://doi.org/10.1007/s00464-014-3759-4

    Article  PubMed  Google Scholar 

  29. Xeroulis G, Dubrowski A, Leslie K (2009) Simulation in laparoscopic surgery: a concurrent validity study for FLS. Surg Endosc 23:161–165. https://doi.org/10.1007/s00464-008-0120-9

    Article  PubMed  Google Scholar 

  30. Fraser SA, Klassen DR, Feldman LS, Ghitulescu GA, Stanbridge D, Fried GM (2003) Evaluating laparoscopic skills. Surg Endosc 17:964–967. https://doi.org/10.1007/s00464-002-8828-4

    Article  CAS  PubMed  Google Scholar 

  31. Feldman LS, Sherman V, Fried GM (2004) Using simulators to assess laparoscopic competence: ready for widespread use? Surgery 135:28–42. https://doi.org/10.1016/S0039-6060(03)00155-7

    Article  PubMed  Google Scholar 

  32. Vassiliou MC, Ghitulescu GA, Feldman LS, Stanbridge D, Leffondré K, Sigman HH, Fried GM (2006) The MISTELS program to measure technical skill in laparoscopic surgery. Surg Endosc 20:744–747. https://doi.org/10.1007/s00464-005-3008-y

    Article  CAS  PubMed  Google Scholar 

  33. Pérez F, Sossa H, Martínez R, Lorias D, Minor A (2013) Video-based tracking of laparoscopic instruments using an orthogonal webcams system. World Acad Sci Eng Technol Int J Medical Heal Biomed Bioeng Pharm Eng 7:440–443

    Google Scholar 

  34. Park J, Sandberg IW (1991) Universal approximation using radial-basis-function networks. Neural Comput 3:246–257. https://doi.org/10.1162/neco.1991.3.2.246

    Article  CAS  PubMed  Google Scholar 

  35. Jain AK (2010) Data clustering: 50 years beyond K-means. Pattern Recognit Lett. https://doi.org/10.1016/j.patrec.2009.09.011

    Article  Google Scholar 

  36. Painuli S, Elangovan M, Sugumaran V (2014) Tool condition monitoring using K-star algorithm. Expert Syst Appl 41:2638–2643. https://doi.org/10.1016/j.eswa.2013.11.005

    Article  Google Scholar 

  37. Cleary JG, Cleary JG, Trigg LE (1995) K*: an instance-based learner using an entropic distance measure. in: Proceedings of 12TH international conference on machine learning, pp 108–114

    Chapter  Google Scholar 

  38. Oshiro TM, Perez PS, Baranauskas JA (2012) How many trees in a random forest?. Springer, Berlin, pp 154–168

    Google Scholar 

  39. Breiman L (2001) Random forests. Mach Learn 45:5–32. https://doi.org/10.1023/A:1010933404324

    Article  Google Scholar 

  40. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software. ACM SIGKDD Explor Newsl 11:10–18. https://doi.org/10.1145/1656274.1656278

    Article  Google Scholar 

  41. Lin Z, Uemura M, Zecca M, Sessa S, Ishii H, Tomikawa M, Hashizume M, Takanishi A (2013) Objective skill evaluation for laparoscopic training based on motion analysis. IEEE Trans Biomed Eng 60:977–985. https://doi.org/10.1109/TBME.2012.2230260

    Article  PubMed  Google Scholar 

  42. Hofstad EF, Våpenstad C, Chmarra MK, Langø T, Kuhry E, Mårvik R (2013) A study of psychomotor skills in minimally invasive surgery: what differentiates expert and nonexpert performance. Surg Endosc 27:854–863. https://doi.org/10.1007/s00464-012-2524-9

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors want to thank all the surgeons, residents, and medical students for their enthusiastic and kindly participation in this study.

Funding

The authors declare that no grants or funding sources were received for this work.

Author information

Authors and Affiliations

  1. Instituto de Ciencias Aplicadas y Tecnología (ICAT), Universidad Nacional Autónoma de México (UNAM), Circuito Exterior S/N, Ciudad Universitaria, Coyoacán, 04510, Ciudad de México, Mexico

    Fernando Pérez-Escamirosa

  2. Department of Biomedical Informatics, Faculty of Medicine, Universidad Nacional Autónoma de México (UNAM), Circuito Interior, Av. Universidad 3000, Ciudad Universitaria, Coyoacán, 04510, Ciudad de México, Mexico

    Fernando Pérez-Escamirosa

  3. Laboratory of Computing Technologies and Electronics, Faculty of Engineering, Universidad Autónoma de Guerrero, Av. Lázaro Cárdenas S/N, Ciudad Universitaria, 39070, Chilpancingo, Guerrero, Mexico

    Antonio Alarcón-Paredes & Gustavo Adolfo Alonso-Silverio

  4. Biomedical Engineering and Telemedicine Centre (GBT), ETSI Telecomunicación, Center for Biomedical Technology, Universidad Politécnica de Madrid (UPM), Avda Complutense, 30, 28040, Madrid, Spain

    Ignacio Oropesa

  5. Intelligent Computing Laboratory, Centro de Innovación y Desarrollo Tecnológico en Computación (CIDETEC-IPN), Av. Juan de Dios Bátiz, Col. Nueva Industrial Vallejo, 07700, Ciudad de México, Mexico

    Oscar Camacho-Nieto

  6. Department of Electrical Engineering, Bioelectronics Section, Centro de Investigación y de Estudios Avanzados del Instituto Politécnico Nacional (CINVESTAV–IPN), Av. IPN 2508, Col. San Pedro Zacatenco, 07360, Ciudad de México, Mexico

    Daniel Lorias-Espinoza & Arturo Minor-Martínez

Authors
  1. Fernando Pérez-Escamirosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonio Alarcón-Paredes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gustavo Adolfo Alonso-Silverio
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ignacio Oropesa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Oscar Camacho-Nieto
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel Lorias-Espinoza
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Arturo Minor-Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonio Alarcón-Paredes.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pérez-Escamirosa, F., Alarcón-Paredes, A., Alonso-Silverio, G.A. et al. Objective classification of psychomotor laparoscopic skills of surgeons based on three different approaches. Int J CARS 15, 27–40 (2020). https://doi.org/10.1007/s11548-019-02073-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-019-02073-2

Keywords

Search

Navigation