Skip to main content
Log in

A survey of context recognition in surgery

  • Review Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

With the introduction of operating rooms of the future context awareness has gained importance in the surgical environment. This paper organizes and reviews different approaches for recognition of context in surgery. Major electronic research databases were queried to obtain relevant publications submitted between the years 2010 and 2015. Three different types of context were identified: (i) the surgical workflow context, (ii) surgeon’s cognitive and (iii) technical state context. A total of 52 relevant studies were identified and grouped based on the type of context detected and sensors used. Different approaches were summarized to provide recommendations for future research. There is still room for improvement in terms of methods used and evaluations performed. Machine learning should be used more extensively to uncover hidden relationships between different properties of the surgeon’s state, particularly when performing cognitive context recognition. Furthermore, validation protocols should be improved by performing more evaluations in situ and with a higher number of unique participants. The paper also provides a structured outline of recent context recognition methods to facilitate development of new generation context-aware surgical support systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

References

  1. Abowd GD, Dey AK, Brown PJ, Davies N, Smith M, Steggles P (1999) Towards a better understanding of context and context-awareness. In: Proceedings of the 1st international symposium on handheld and ubiquitous computing, HUC ’99. Springer-Verlag, London, pp 304–307

    Chapter  Google Scholar 

  2. Ahmidi N, Gao Y, Béjar B, Vedula SS, Khudanpur S, Vidal R, Hager GD (2013) String motif-based description of tool motion for detecting skill and gestures in robotic surgery. In: medical image computing and computer-assisted intervention–MICCAI 2013, pp 26–33. Springer

    Google Scholar 

  3. Ahmidi N, Hager GD, Ishii L, Fichtinger G, Gallia GL, Ishii M (2010) Surgical task and skill classification from eye tracking and tool motion in minimally invasive surgery. In: medical image computing and computer-assisted intervention–MICCAI 2010, pp 295–302. Springer

    Google Scholar 

  4. Andreatta PB, Hillard M, Krain LP (2010) The impact of stress factors in simulation-based laparoscopic training. Surgery 147(5):631–639

    Article  PubMed  Google Scholar 

  5. Arora S, Sevdalis N, Nestel D, Woloshynowych M, Darzi A, Kneebone R (2010) The impact of stress on surgical performance: a systematic review of the literature. Surgery 147(3):318–330.e6

    Article  PubMed  Google Scholar 

  6. Bardram JE, Doryab A, Jensen RM, Lange PM, Nielsen KL, Petersen ST (2011) Phase recognition during surgical procedures using embedded and body-worn sensors. In: 2011 IEEE international conference on pervasive computing and communications (PerCom), pp 45–53. IEEE

    Google Scholar 

  7. Berguer R, Smith WD, Chung YH (2001) Performing laparoscopic surgery is significantly more stressful for the surgeon than open surgery. Surg Endosc 15:1204–1207

    Article  CAS  PubMed  Google Scholar 

  8. Bharathan R, Aggarwal R, Darzi A (2013) Operating room of the future. Best Pract Res Cl Ob 27 (3):311–322

    Article  Google Scholar 

  9. Bitterman N (2006) Technologies and solutions for data display in the operating room. J Clin Monitor Comp 20(3):165–173

    Article  Google Scholar 

  10. Blum T, Feußner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: medical image computing and computer-assisted intervention–MICCAI 2010, pp 400–407. Springer

    Google Scholar 

  11. Bogner M (2003) Misadventures in health care: inside stories. Human Error and Safety Series. Taylor & Francis

  12. Bootsma M, Swenne CA, Van Bolhuis HH, Chang PC, Cats VM, Bruschke AV (1994) Heart rate and heart rate variability as indexes of sympathovagal balance. Am J Physiol Heart Circ Physiol 266(4):H1565–H1571

  13. Bricon-Souf N, Newman CR (2007) Context awareness in health care: a review. Int J Med Inform 76 (1):2–12

    Article  PubMed  Google Scholar 

  14. Cao C, MacKenzie CL, Payandeh S (1996) Task and motion analyses in endoscopic surgery. In: Proceedings ASME dynamic systems and control division, pp 583–590

    Google Scholar 

  15. Carswell CM, Clarke D, Seales WB (2005) Assessing mental workload during laparoscopic surgery. Surg Innov 12(1):80–90

    Article  PubMed  Google Scholar 

  16. Cundy TP, Thangaraj E, Rafii-Tari H, Payne CJ, Azzie G, Sodergren MH, Yang GZ, Darzi A (2015) Force-sensing enhanced simulation environment (ForSense) for laparoscopic surgery training and assessment. Surgery 157(4):723–731

    Article  PubMed  Google Scholar 

  17. Cvach M (2012) Monitor alarm fatigue. Biomed Instrum Techn 46(4):268–277

    Article  Google Scholar 

  18. Di Stasi LL, McCamy MB, Macknik SL, Mankin JA, Hooft N, Catena A, Martinez-Conde S (2014) Saccadic eye movement metrics reflect surgical residents fatigue. Ann Surg 259(00):824–829

    Article  PubMed  Google Scholar 

  19. van Dooren M, de Vries JGJ, Janssen JH (2012) Emotional sweating across the body: comparing 16 different skin conductance measurement locations. Physiol Behav 106(2):298–304

  20. Egi H, Okajima M, Kawahara T, Yoshimitsu M, Sumitani D, Tokunaga M, Takeda H, Itamoto T, Ohdan H (2010) Scientific assessment of endoscopic surgical skills. Minim Invasiv Ther 19(1):30–34

    Article  Google Scholar 

  21. El-Fakdi A, Gamero F, Meléndez J, Auffret V, Haigron P (2014) EXiTCDSS: A framework for a workflow-based CBR for interventional clinical decision support systems and its application to TAVI. Expert Syst Appl 41(2):284–294

    Article  Google Scholar 

  22. Engert V, Vogel S, Efanov SI, Duchesne A, Corbo V, Ali N, Pruessner JC (2011) Investigation into the cross-correlation of salivary cortisol and alpha-amylase responses to psychological stress. Psychoneuroendocrino 36(9):1294–1302

    Article  CAS  Google Scholar 

  23. Ericsson KA, Simon HA (1980) Verbal reports as data. Psychol Rev 87(3):215

    Article  Google Scholar 

  24. Franke S, Meixensberger J, Neumuth T (2013) Intervention time prediction from surgical low-level tasks. J Biomed Inform 46(1):152–159

    Article  PubMed  Google Scholar 

  25. Franke S, Meixensberger J, Neumuth T (2015) Multi-perspective workflow modeling for online surgical situation models. J Biomed Inform 54:158–166

    Article  PubMed  Google Scholar 

  26. Fritz T, Begel A, Müller SC, Yigit-Elliott S, Züger M (2014) Using psycho-physiological measures to assess task difficulty in software development. In: Proceedings of the 36th international conference on software engineering, pp 402–413. ACM

    Google Scholar 

  27. Gaggioli A, Pallavicini F, Morganti L, Serino S, Scaratti C, Briguglio M, Crifaci G, Vetrano N, Giulintano A, Bernava G, Tartarisco G, Pioggia G, Raspelli S, Cipresso P, Vigna C, Grassi A, Baruffi M, Wiederhold B, Riva G (2014) Experiential virtual scenarios with real-time monitoring (interreality) for the management of psychological stress: a block randomized controlled trial. J Med Internet Res 16(7):e167

    Article  PubMed  PubMed Central  Google Scholar 

  28. Gallina S, Di Mauro M, D’Amico MA, D’Angelo E, Sablone A, Di Fonso A, Bascelli A, Izzicupo P, Di Baldassarre A (2011) Salivary chromogranin A, but not α-amylase, correlates with cardiovascular parameters during high-intensity exercise. Clin Endocrinol 75(6):747–752

    Article  CAS  Google Scholar 

  29. Garbarino M, Lai M, Tognetti S, Picard R, Bender D (2014) Empatica E3 - a wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In: 4th international conference on wireless mobile communication and healthcare - transforming healthcare through innovations in mobile and wireless technologies

    Google Scholar 

  30. Gibson JNA, Cowie JG, Iprenburg M (2012) Transforaminal endoscopic spinal surgery: the future ‘gold standard’ for discectomy?—a review. Surg J R Coll Surg E 10(5):290–296

    Google Scholar 

  31. Glaser B, Dänzer S, Neumuth T (2015) Intra-operative surgical instrument usage detection on a multi-sensor table. Int J Comput Assist Radiol Surg 10(3):351–362

    Article  PubMed  Google Scholar 

  32. Groscurth P (2002) Anatomy of sweat glands. Hyperhidrosis and Botulinum Toxin in Dermatology 30

  33. Guru KA, Esfahani ET, Raza SJ, Bhat R, Wang K, Hammond Y, Wilding G, Peabody JO, Chowriappa AJ (2015) Cognitive skills assessment during robot-assisted surgery: separating the wheat from the chaff. BJU Int 115(1):166–174

    Article  PubMed  Google Scholar 

  34. Harada K, Minakawa Y, Baek Y, Kozuka Y, Sora S, Morita A, Sugita N, Mitsuishi M (2011) Microsurgical skill assessment: toward skill-based surgical robotic control. In: Engineering in medicine and biology society, EMBC, 2011 Annual international conference of the IEEE, pp 6700–6703. IEEE

    Google Scholar 

  35. Haro BB, Zappella L, Vidal R (2012) Surgical gesture classification from video data. In: Medical image computing and computer-assisted intervention–MICCAI 2012, pp 34–41. Springer

    Google Scholar 

  36. Harvey A, Vickers JN, Snelgrove R, Scott MF, Morrison S (2014) Expert surgeon’s quiet eye and slowing down: expertise differences in performance and quiet eye duration during identification and dissection of the recurrent laryngeal nerve. Am J Surg 207(2):187–193

    Article  PubMed  Google Scholar 

  37. Hofstad EF, Vå penstad C, Chmarra MK, Langø T, Kuhry E, Mårvik R (2013) A study of psychomotor skills in minimally invasive surgery: what differentiates expert and nonexpert performance. Surg Endosc 27(3):854–863

    Article  PubMed  Google Scholar 

  38. Hong JY, Suh EH, Kim SJ (2009) Context-aware systems: a literature review and classification. Expert Syst Appl 36(4):8509–8522

    Article  Google Scholar 

  39. Hsu KE, Man FY, Gizicki RA, Feldman LS, Fried GM (2008) Experienced surgeons can do more than one thing at a time: effect of distraction on performance of a simple laparoscopic and cognitive task by experienced and novice surgeons. Surg Endosc 22:196–201

    Article  CAS  PubMed  Google Scholar 

  40. Hughes-Hallett A, Mayer EK, Marcus HJ, Pratt P, Mason S, Darzi A, Vale JA (2015) Inattention blindness in surgery. Surg Endosc

  41. Jiang X, Zheng B, Tien G, Atkins MS (2013) Pupil response to precision in surgical task execution. St Heal T 184:210–214

    Google Scholar 

  42. Jones KI, Amawi F, Bhalla A, Peacock O, Williams JP, Lund JN (2015) Assessing surgeon stress when operating using heart rate variability and the state trait anxiety inventory: will surgery be the death of us? Colorectal Dis 17(4):335–341

    Article  CAS  PubMed  Google Scholar 

  43. Kirby GS, Kwasnicki RM, Hargrove C, Rees JL, Sodergren MH, Yang GZ, Lo BP (2014) Wireless body sensor for objective assessment of surgical performance on a standardised fls task. In: Proceedings of the 9th international conference on body area networks, pp 147–153. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering)

    Google Scholar 

  44. Kuhn EW, Choi YH, Schönherr M, Liakopoulos OJ, Rahmanian PB, Choi CYU, Wittwer T, Wahlers T (2013) Intraoperative stress in cardiac surgery: attendings versus residents. J Surg Res 182(2):e43—9

    Article  PubMed  Google Scholar 

  45. Kurita Y, Tsuji T, Kawahara T (2013) Force-based automatic classification of basic manipulations with grasping forceps. Int J Life Sci Med Res 3(2):76–82

    Article  Google Scholar 

  46. Lalys F, Bouget D, Riffaud L, Jannin P (2013) Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures. Int J Comput Assist Radiol Surg 8(1):39–49

    Article  PubMed  Google Scholar 

  47. Lalys F, Jannin P (2014) Surgical process modelling: a review. Int J Comput Assist Radiol Surg 9(3):495–511

    Article  PubMed  Google Scholar 

  48. Lalys F, Riffaud L, Bouget D, Jannin P (2011) An application-dependent framework for the recognition of high-level surgical tasks in the OR. In: Medical image computing and computer-assisted intervention–MICCAI 2011, pp 331–338. Springer

    Google Scholar 

  49. Lalys F, Riffaud L, Morandi X, Jannin P (2011) Surgical phases detection from microscope videos by combining svm and hmm. In: Medical computer vision. Recognition Techniques and Applications in Medical Imaging, pp 54–62. Springer

    Google Scholar 

  50. Lea C, Hager GD, Vidal R (2015) An improved model for segmentation and recognition of fine-grained activities with application to surgical training tasks. In: Applications of computer vision (WACV), 2015 IEEE winter conference on, pp 1123–1129. IEEE

    Google Scholar 

  51. Loukas C, Georgiou E (2011) Multivariate autoregressive modeling of hand kinematics for laparoscopic skills assessment of surgical trainees. IEEE Trans Biomed Eng 58(11):3289– 3297

    Article  PubMed  Google Scholar 

  52. Loukas C, Georgiou E (2013) Surgical workflow analysis with Gaussian mixture multivariate autoregressive (GMMAR) models: a simulation study. Comput Aided Surg 18(3-4):47–62

    Article  PubMed  Google Scholar 

  53. Loukas C, Rouseas C, Georgiou E (2013) The role of hand motion connectivity in the performance of laparoscopic procedures on a virtual reality simulator. Med Biol Eng Comput 51(8):911– 922

    Article  PubMed  Google Scholar 

  54. Meißner C, Meixensberger J, Pretschner A, Neumuth T (2014) Sensor-based surgical activity recognition in unconstrained environments. Minim Invasiv Ther 23(4):198–205

    Article  Google Scholar 

  55. Muaremi A, Arnrich B, Tröster G (2013) Towards measuring stress with smartphones and wearable devices during workday and sleep. J Bionanosci 3(2):172–183

    Article  Google Scholar 

  56. Neumuth T, Meißner C (2012) Online recognition of surgical instruments by information fusion. Int J Comput Assist Radiol Surg 7(2):297–304

    Article  PubMed  Google Scholar 

  57. Overby DW, Watson RA (2014) Hand motion patterns of fundamentals of laparoscopic surgery certified and noncertified surgeons. Am J Surg 207(2):226–230

    Article  PubMed  Google Scholar 

  58. Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N (2012) Statistical modeling and recognition of surgical workflow. Med Image Anal 16(3):632–641

    Article  PubMed  Google Scholar 

  59. Phitayakorn R, Minehart RD, Hemingway MW, Pian-Smith MCM, Petrusa E (2015) Relationship between physiologic and psychological measures of autonomic activation in operating room teams during a simulated airway emergency. Am J Surg 209(1):86–92

    Article  PubMed  Google Scholar 

  60. Phitayakorn R, Minehart RD, Pian-Smith MCM, Hemingway MW, Petrusa ER (2015) Practicality of using galvanic skin response to measure intraoperative physiologic autonomic activation in operating room team members. Surgery 158(5):1415–20

    Article  PubMed  Google Scholar 

  61. Poddar P, Ahmidi N, Vedula SS, Ishii L, Hager GD (2014) Automated objective surgical skill assessment in the operating room using unstructured tool motion. Med Image Comput Comput Assist Interv

  62. Prichard RS, O’Neill CJ, Oucharek JJ, Holmes CYV, Colinda YH, Delbridge LW, Sywak MS (2012) A prospective study of heart rate variability in endocrine surgery: surgical training increases consultant’s mental strain. J Surg Educ 69(4):453–8

    Article  PubMed  Google Scholar 

  63. Quellec G, Lamard M, Cochener B, Cazuguel G (2014) Real-time segmentation and recognition of surgical tasks in cataract surgery videos. IEEE T Med Imaging 33(12):2352–2360

    Article  Google Scholar 

  64. Quellec G, Lamard M, Cochener B, Cazuguel G (2015) Real-time task recognition in cataract surgery videos using adaptive spatiotemporal polynomials. IEEE T Med Imaging 34(4):877– 887

    Article  Google Scholar 

  65. Regehr G, MacRae H, Reznick RK, Szalay D (1998) Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73(9):993–997

    Article  CAS  PubMed  Google Scholar 

  66. Richstone L, Schwartz MJ, Seideman C, Cadeddu J, Marshall S, Kavoussi LR (2010) Eye metrics as an objective assessment of surgical skill. Ann Surg 252(1):177–182

    Article  PubMed  Google Scholar 

  67. Rieger A, Fenger S, Neubert S, Weippert M, Kreuzfeld S, Stoll R (2015) Psychophysical workload in the operating room: primary surgeon versus assistant. Surg Endosc 29(7):1990– 1998

    Article  PubMed  Google Scholar 

  68. Rieger A, Stoll R, Kreuzfeld S, Behrens K, Weippert M (2014) Heart rate and heart rate variability as indirect markers of surgeons’ intraoperative stress. Int Arch Occup Environ Health 87(2):165–174

    Article  PubMed  Google Scholar 

  69. Rutherford DN, D’Angelo ALD, Law KE, Pugh CM (2015) Advanced engineering technology for measuring performance. Surg Clin North Am 95(4):813–26

    Article  PubMed  Google Scholar 

  70. Schulz CM, Schneider E, Fritz L, Vockeroth J, Hapfelmeier A, Wasmaier M, Kochs EF, Schneider G (2011) Eye tracking for assessment of workload: a pilot study in an anaesthesia simulator environment. Brit J Anaesth 106(October 2010):44–50

    Article  CAS  PubMed  Google Scholar 

  71. Stauder R, Okur A, Peter L, Schneider A, Kranzfelder M, Feussner H, Navab N (2014) Random forests for phase detection in surgical workflow analysis. In: Information processing in computer-assisted interventions, pp 148–157. Springer

    Google Scholar 

  72. Stefanidis D, Wang F, Korndorffer JR, Dunne JB, Scott DJ (2010) Robotic assistance improves intracorporeal suturing performance and safety in the operating room while decreasing operator workload. Surg Endosc 24:377–382

    Article  PubMed  Google Scholar 

  73. Tao L, Elhamifar E, Khudanpur S, Hager GD, Vidal R (2012) Sparse hidden Markov models for surgical gesture classification and skill evaluation. In: Information processing in computer-assisted interventions, pp 167–177. Springer

    Google Scholar 

  74. Tao L, Zappella L, Hager GD, Vidal R (2013) Surgical gesture segmentation and recognition. In: Medical image computing and computer-assisted intervention–MICCAI 2013, pp 339–346. Springer

    Google Scholar 

  75. Thiemjarus S, James A, Yang GZ (2012) An eye-hand data fusion framework for pervasive sensing of surgical activities. Pattern Recogn 45(8):2855–2867

    Article  Google Scholar 

  76. Tien G, Zheng B, Atkins MS (2011) Quantifying surgeons’ vigilance during laparoscopic operations using eyegaze tracking. St Heal T 163:658–662

    Google Scholar 

  77. Tien T, Pucher PH, Sodergren MH, Sriskandarajah K, Yang GZ, Darzi A (2014) Differences in gaze behaviour of expert and junior surgeons performing open inguinal hernia repair. Surg Endosc 2: 405–413

    Google Scholar 

  78. Tomasko JM, Pauli EM, Kunselman AR, Haluck RS (2012) Sleep deprivation increases cognitive workload during simulated surgical tasks. Am J Surg 203(1):37–43

    Article  PubMed  Google Scholar 

  79. Twinanda AP, Marescaux J, De Mathelin M, Padoy N (2014) Towards better laparoscopic video database organization by automatic surgery classification. In: Information processing in computer-assisted interventions, pp 186–195. Springer

    Google Scholar 

  80. Uemura M, Tomikawa M, Kumashiro R, Miao T, Souzaki R, Ieiri S, Ohuchida K, Lefor AT, Hashizume M (2014) Analysis of hand motion differentiates expert and novice surgeons. J Surg Resh 188(1):8–13

    Article  Google Scholar 

  81. Unger M, Chalopin C, Neumuth T (2014) Vision-based online recognition of surgical activities. Int J Comput Assist Radiol Surg 9:979–986

    Article  PubMed  Google Scholar 

  82. Watson RA (2012) Computer-aided feedback of surgical knot tying using optical tracking. J Surg Educ 69 (3):306–310

    Article  PubMed  Google Scholar 

  83. Watson RA (2013) Quantification of surgical technique using an inertial measurement unit. Simul Healthc 8 (3):162–165

    Article  PubMed  Google Scholar 

  84. Watson RA (2014) Use of a machine learning algorithm to classify expertise: analysis of hand motion patterns during a simulated surgical task. Acad Med 89(8):1–5

    Article  Google Scholar 

  85. Wilson M, McGrath J, Vine S, Brewer J, Defriend D, Masters R (2010) Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from experts. Surg Endosc 24(10):2458–2464

    Article  PubMed  PubMed Central  Google Scholar 

  86. Wilson MR, McGrath JS, Vine SJ, Brewer J, Defriend D, Masters RSW (2011) Perceptual impairment and psychomotor control in virtual laparoscopic surgery. Surg Endosc 25:2268–2274

    Article  PubMed  PubMed Central  Google Scholar 

  87. Wilson MR, Poolton JM, Malhotra N, Ngo K, Bright E, Masters RSW (2011) Development and validation of a surgical workload measure: the surgery task load index (SURG-TLX). World J Surg 35(9):1961–1969

    Article  PubMed  PubMed Central  Google Scholar 

  88. Wilson MR, Vine SJ, Bright E, Masters RSW, Defriend D, McGrath JS (2011) Gaze training enhances laparoscopic technical skill acquisition and multi-tasking performance: a randomized, controlled study. Surg Endosc 25:3731–3739

    Article  PubMed  PubMed Central  Google Scholar 

  89. Yurko YY, Scerbo MW, Prabhu AS, Acker CE, Stefanidis D (2010) Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simul Healthc 5(5):267–271

    Article  PubMed  Google Scholar 

  90. Zheng B, Cassera MA, Martinec DV, Spaun GO, Swanström LL (2010) Measuring mental workload during the performance of advanced laparoscopic tasks. Surg Endosc 24:45–50

    Article  PubMed  Google Scholar 

  91. Zheng B, Jiang X, Atkins MS (2015) Detection of changes in surgical difficulty: evidence from pupil responses. Surg Innov 22: 1–7

    Article  Google Scholar 

  92. Zheng B, Jiang X, Tien G, Meneghetti A, Panton ONM, Atkins MS (2012) Workload assessment of surgeons: correlation between NASA TLX and blinks. Surg Endosc 26:2746–2750

    Article  PubMed  Google Scholar 

  93. Zheng B, Tien G, Atkins SM, Swindells C, Tanin H, Meneghetti A, Qayumi KA, Neely O, Panton M (2011) Surgeon’s vigilance in the operating room. Am J Surg 201(5):667–671

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Igor Pernek.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pernek, I., Ferscha, A. A survey of context recognition in surgery. Med Biol Eng Comput 55, 1719–1734 (2017). https://doi.org/10.1007/s11517-017-1670-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-017-1670-6

Keywords

Navigation