Abstract
This study focuses on the comparison of expert assessment of virtual surgical procedures through Objective Structured Assessment of Technical Skills (OSATS) with the automatic assessment and feedback provided by a surgical simulator for hysteroscopic procedures. The existing multi-metric scoring system of the simulator was extended to include hysteroscopic myomectomy. The original OSATS was also modified for the examined surgical procedure. OSATS reliability, expert coherence, and interrater agreement with simulator feedback were investigated in a study with eight experts. The same selection of six movies showing virtual procedures performed at a hysteroscopy training course was rated by each expert. For the task-specific checklist, the reliability of the simulator was significantly higher than that of the individual human raters (p=0.006). In addition, the ranked order of the overall scores of all movies was the same for both simulator and expert consensus opinion. This is a first step to providing simulator feedback with the same reliability as an expert panel, thus facilitating competency-based surgical education and assessment in the near future.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Moorthy, K., Munz, Y., Sarker, S., Darzi, A.: Objective assessment of technical skills in surgery. BMJ 327(7422), 1032–1037 (2003)
Gallagher, A., Ritter, E., Champion, H., Higgins, G., Fried, M., Moses, G., Smith, C.D., Satava, R.M.: Virtual reality simulation for the operating room: proficiencybased training as a paradigm shift in surgical skills training. Ann. Surg. 241(2), 364–372 (2005)
Reznick, R., Regehr, G., MacRae, H., Martin, J., McCulloch, W.: Testing technical skill via an innovative ”bench station” examination. Am. J. Surg. 173(3), 226–230 (1997)
Grantcharov, T.P., Kristiansen, V.B., Bendix, J., Bardram, L., Rosenberg, J., Funch-Jensen, P.: Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br. J. Surg. 91(2), 146–150 (2004)
Matsumoto, E.D., Hamstra, S.J., Radomski, S.B., Cusimano, M.D.: The effect of bench model fidelity on endourological skills: a randomized controlled study. J. Urol. 167(3), 1243–1247 (2002)
Cremers, S.L., Lora, A.N., Ferrufino-Ponce, Z.K.: Global rating assessment of skills in intraocular surgery (GRASIS). Ophthalmology 112(10), 1655–1660 (2005)
Mandel, L.P., Lentz, G.M., Goff, B.A.: Teaching and evaluating surgical skills. Obstet. Gynecol. 95(5), 783–785 (2000)
Van Blaricom, A.L., Goff, B.A., Chinn, M., Icasiano, M.M., Nielsen, P., Mandel, L.: A new curriculum for hysteroscopy training as demonstrated by an objective structured assessment of technical skills (OSATS). Am. J. Obstet. Gynecol. 193(5), 1856–1865 (2005)
Darzi, A., Smith, S., Taffinder, N.: Assessing operative skill. Needs to become more objective. BMJ 318(7188), 887–888 (1999)
Rosen, J., Hannaford, B., Richards, C.G., Sinanan, M.N.: Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans. Biomed. Eng. 48(5), 579–591 (2001)
Moorthy, K., Munz, Y., Dosis, A., Bello, F., Darzi, A.: Motion analysis in the training and assessment of minimally invasive surgery. Minim. Invasive Ther. Allied Technol. 12(3), 137–142 (2003)
Stylopoulos, N., Cotin, S., Maithel, S.K., Ottensmeye, M., Jackson, P.G., Bardsley, R.S., Neumann, P.F., Rattner, D.W., Dawson, S.L.: Computer-enhanced laparoscopic training system (celts): bridging the gap. Surg. Endosc. 18(5), 782–789 (2004)
Ritter, E.M., McClusky, D.A., Gallagher, A.G., Smith, C.D.: Real-time objective assessment of knot quality with a portable tensiometer is superior to execution time for assessment of laparoscopic knot-tying performance. Surg. Innov. 12(3), 233–237 (2005)
Sewell, C.: Automatic Performance Evaluation in Surgical Simulation. PhD thesis, Stanford University (2007)
Harders, M., Bachofen, D., Bajka, M., Grassi, M., Heidelberger, B., Sierra, R., Spaelter, U., Steinemann, D., Teschner, M., Tuchschmid, S., Zatonyi, J., Szekely, G.: Virtual reality based simulation of hysteroscopic interventions. Presence: Teleoperators and Virtual Environments 17(5), 441–462 (2008)
Website: VirtaMed HystSim (2009), http://www.simbionix.com/HystSim/HystSim.html
Bajka, M., Tuchschmid, S., Streich, M., Fink, D., Szekely, G., Harders, M.: Evaluation of a new virtual-reality training simulator for hysteroscopy. Surg. Endosc. 23(9), 2026–2033 (2009); Epub Apr 24, 2008
Tuchschmid, S., Bajka, M., Bachofen, D., Szekely, G., Harders, M.: Objective surgical performance assessment for virtual hysteroscopy. Stud. Health Technol. Inform. 125, 473–478 (2007)
Bajka, M., Tuchschmid, S., Fink, D., Szekely, G., Harders, M.: Establishing construct validity of a virtual reality training simulator for hysteroscopy via a multi metric scoring system. Surg. Endosc. (2009); (Epub ahead of print, June 24)
Dath, D., Regehr, G., Birch, D., Schlachta, C., Poulin, E., Mamazza, J., Reznick, R., MacRae, H.M.: Toward reliable operative assessment: the reliability and feasibility of videotaped assessment of laparoscopic technical skills. Surg. Endosc. 18(12), 1800–1804 (2004)
Gallagher, A.G., Ritter, E.M., Satava, R.M.: Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg. Endosc. 17(10), 1525–1529 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Tuchschmid, S., Bajka, M., Harders, M. (2010). Comparing Automatic Simulator Assessment with Expert Assessment of Virtual Surgical Procedures. In: Bello, F., Cotin, S. (eds) Biomedical Simulation. ISBMS 2010. Lecture Notes in Computer Science, vol 5958. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11615-5_19
Download citation
DOI: https://doi.org/10.1007/978-3-642-11615-5_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-11614-8
Online ISBN: 978-3-642-11615-5
eBook Packages: Computer ScienceComputer Science (R0)