Abstract
With the introduction of operating rooms of the future context awareness has gained importance in the surgical environment. This paper organizes and reviews different approaches for recognition of context in surgery. Major electronic research databases were queried to obtain relevant publications submitted between the years 2010 and 2015. Three different types of context were identified: (i) the surgical workflow context, (ii) surgeon’s cognitive and (iii) technical state context. A total of 52 relevant studies were identified and grouped based on the type of context detected and sensors used. Different approaches were summarized to provide recommendations for future research. There is still room for improvement in terms of methods used and evaluations performed. Machine learning should be used more extensively to uncover hidden relationships between different properties of the surgeon’s state, particularly when performing cognitive context recognition. Furthermore, validation protocols should be improved by performing more evaluations in situ and with a higher number of unique participants. The paper also provides a structured outline of recent context recognition methods to facilitate development of new generation context-aware surgical support systems.

Similar content being viewed by others
References
Abowd GD, Dey AK, Brown PJ, Davies N, Smith M, Steggles P (1999) Towards a better understanding of context and context-awareness. In: Proceedings of the 1st international symposium on handheld and ubiquitous computing, HUC ’99. Springer-Verlag, London, pp 304–307
Ahmidi N, Gao Y, Béjar B, Vedula SS, Khudanpur S, Vidal R, Hager GD (2013) String motif-based description of tool motion for detecting skill and gestures in robotic surgery. In: medical image computing and computer-assisted intervention–MICCAI 2013, pp 26–33. Springer
Ahmidi N, Hager GD, Ishii L, Fichtinger G, Gallia GL, Ishii M (2010) Surgical task and skill classification from eye tracking and tool motion in minimally invasive surgery. In: medical image computing and computer-assisted intervention–MICCAI 2010, pp 295–302. Springer
Andreatta PB, Hillard M, Krain LP (2010) The impact of stress factors in simulation-based laparoscopic training. Surgery 147(5):631–639
Arora S, Sevdalis N, Nestel D, Woloshynowych M, Darzi A, Kneebone R (2010) The impact of stress on surgical performance: a systematic review of the literature. Surgery 147(3):318–330.e6
Bardram JE, Doryab A, Jensen RM, Lange PM, Nielsen KL, Petersen ST (2011) Phase recognition during surgical procedures using embedded and body-worn sensors. In: 2011 IEEE international conference on pervasive computing and communications (PerCom), pp 45–53. IEEE
Berguer R, Smith WD, Chung YH (2001) Performing laparoscopic surgery is significantly more stressful for the surgeon than open surgery. Surg Endosc 15:1204–1207
Bharathan R, Aggarwal R, Darzi A (2013) Operating room of the future. Best Pract Res Cl Ob 27 (3):311–322
Bitterman N (2006) Technologies and solutions for data display in the operating room. J Clin Monitor Comp 20(3):165–173
Blum T, Feußner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: medical image computing and computer-assisted intervention–MICCAI 2010, pp 400–407. Springer
Bogner M (2003) Misadventures in health care: inside stories. Human Error and Safety Series. Taylor & Francis
Bootsma M, Swenne CA, Van Bolhuis HH, Chang PC, Cats VM, Bruschke AV (1994) Heart rate and heart rate variability as indexes of sympathovagal balance. Am J Physiol Heart Circ Physiol 266(4):H1565–H1571
Bricon-Souf N, Newman CR (2007) Context awareness in health care: a review. Int J Med Inform 76 (1):2–12
Cao C, MacKenzie CL, Payandeh S (1996) Task and motion analyses in endoscopic surgery. In: Proceedings ASME dynamic systems and control division, pp 583–590
Carswell CM, Clarke D, Seales WB (2005) Assessing mental workload during laparoscopic surgery. Surg Innov 12(1):80–90
Cundy TP, Thangaraj E, Rafii-Tari H, Payne CJ, Azzie G, Sodergren MH, Yang GZ, Darzi A (2015) Force-sensing enhanced simulation environment (ForSense) for laparoscopic surgery training and assessment. Surgery 157(4):723–731
Cvach M (2012) Monitor alarm fatigue. Biomed Instrum Techn 46(4):268–277
Di Stasi LL, McCamy MB, Macknik SL, Mankin JA, Hooft N, Catena A, Martinez-Conde S (2014) Saccadic eye movement metrics reflect surgical residents fatigue. Ann Surg 259(00):824–829
van Dooren M, de Vries JGJ, Janssen JH (2012) Emotional sweating across the body: comparing 16 different skin conductance measurement locations. Physiol Behav 106(2):298–304
Egi H, Okajima M, Kawahara T, Yoshimitsu M, Sumitani D, Tokunaga M, Takeda H, Itamoto T, Ohdan H (2010) Scientific assessment of endoscopic surgical skills. Minim Invasiv Ther 19(1):30–34
El-Fakdi A, Gamero F, Meléndez J, Auffret V, Haigron P (2014) EXiTCDSS: A framework for a workflow-based CBR for interventional clinical decision support systems and its application to TAVI. Expert Syst Appl 41(2):284–294
Engert V, Vogel S, Efanov SI, Duchesne A, Corbo V, Ali N, Pruessner JC (2011) Investigation into the cross-correlation of salivary cortisol and alpha-amylase responses to psychological stress. Psychoneuroendocrino 36(9):1294–1302
Ericsson KA, Simon HA (1980) Verbal reports as data. Psychol Rev 87(3):215
Franke S, Meixensberger J, Neumuth T (2013) Intervention time prediction from surgical low-level tasks. J Biomed Inform 46(1):152–159
Franke S, Meixensberger J, Neumuth T (2015) Multi-perspective workflow modeling for online surgical situation models. J Biomed Inform 54:158–166
Fritz T, Begel A, Müller SC, Yigit-Elliott S, Züger M (2014) Using psycho-physiological measures to assess task difficulty in software development. In: Proceedings of the 36th international conference on software engineering, pp 402–413. ACM
Gaggioli A, Pallavicini F, Morganti L, Serino S, Scaratti C, Briguglio M, Crifaci G, Vetrano N, Giulintano A, Bernava G, Tartarisco G, Pioggia G, Raspelli S, Cipresso P, Vigna C, Grassi A, Baruffi M, Wiederhold B, Riva G (2014) Experiential virtual scenarios with real-time monitoring (interreality) for the management of psychological stress: a block randomized controlled trial. J Med Internet Res 16(7):e167
Gallina S, Di Mauro M, D’Amico MA, D’Angelo E, Sablone A, Di Fonso A, Bascelli A, Izzicupo P, Di Baldassarre A (2011) Salivary chromogranin A, but not α-amylase, correlates with cardiovascular parameters during high-intensity exercise. Clin Endocrinol 75(6):747–752
Garbarino M, Lai M, Tognetti S, Picard R, Bender D (2014) Empatica E3 - a wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition. In: 4th international conference on wireless mobile communication and healthcare - transforming healthcare through innovations in mobile and wireless technologies
Gibson JNA, Cowie JG, Iprenburg M (2012) Transforaminal endoscopic spinal surgery: the future ‘gold standard’ for discectomy?—a review. Surg J R Coll Surg E 10(5):290–296
Glaser B, Dänzer S, Neumuth T (2015) Intra-operative surgical instrument usage detection on a multi-sensor table. Int J Comput Assist Radiol Surg 10(3):351–362
Groscurth P (2002) Anatomy of sweat glands. Hyperhidrosis and Botulinum Toxin in Dermatology 30
Guru KA, Esfahani ET, Raza SJ, Bhat R, Wang K, Hammond Y, Wilding G, Peabody JO, Chowriappa AJ (2015) Cognitive skills assessment during robot-assisted surgery: separating the wheat from the chaff. BJU Int 115(1):166–174
Harada K, Minakawa Y, Baek Y, Kozuka Y, Sora S, Morita A, Sugita N, Mitsuishi M (2011) Microsurgical skill assessment: toward skill-based surgical robotic control. In: Engineering in medicine and biology society, EMBC, 2011 Annual international conference of the IEEE, pp 6700–6703. IEEE
Haro BB, Zappella L, Vidal R (2012) Surgical gesture classification from video data. In: Medical image computing and computer-assisted intervention–MICCAI 2012, pp 34–41. Springer
Harvey A, Vickers JN, Snelgrove R, Scott MF, Morrison S (2014) Expert surgeon’s quiet eye and slowing down: expertise differences in performance and quiet eye duration during identification and dissection of the recurrent laryngeal nerve. Am J Surg 207(2):187–193
Hofstad EF, Vå penstad C, Chmarra MK, Langø T, Kuhry E, Mårvik R (2013) A study of psychomotor skills in minimally invasive surgery: what differentiates expert and nonexpert performance. Surg Endosc 27(3):854–863
Hong JY, Suh EH, Kim SJ (2009) Context-aware systems: a literature review and classification. Expert Syst Appl 36(4):8509–8522
Hsu KE, Man FY, Gizicki RA, Feldman LS, Fried GM (2008) Experienced surgeons can do more than one thing at a time: effect of distraction on performance of a simple laparoscopic and cognitive task by experienced and novice surgeons. Surg Endosc 22:196–201
Hughes-Hallett A, Mayer EK, Marcus HJ, Pratt P, Mason S, Darzi A, Vale JA (2015) Inattention blindness in surgery. Surg Endosc
Jiang X, Zheng B, Tien G, Atkins MS (2013) Pupil response to precision in surgical task execution. St Heal T 184:210–214
Jones KI, Amawi F, Bhalla A, Peacock O, Williams JP, Lund JN (2015) Assessing surgeon stress when operating using heart rate variability and the state trait anxiety inventory: will surgery be the death of us? Colorectal Dis 17(4):335–341
Kirby GS, Kwasnicki RM, Hargrove C, Rees JL, Sodergren MH, Yang GZ, Lo BP (2014) Wireless body sensor for objective assessment of surgical performance on a standardised fls task. In: Proceedings of the 9th international conference on body area networks, pp 147–153. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering)
Kuhn EW, Choi YH, Schönherr M, Liakopoulos OJ, Rahmanian PB, Choi CYU, Wittwer T, Wahlers T (2013) Intraoperative stress in cardiac surgery: attendings versus residents. J Surg Res 182(2):e43—9
Kurita Y, Tsuji T, Kawahara T (2013) Force-based automatic classification of basic manipulations with grasping forceps. Int J Life Sci Med Res 3(2):76–82
Lalys F, Bouget D, Riffaud L, Jannin P (2013) Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures. Int J Comput Assist Radiol Surg 8(1):39–49
Lalys F, Jannin P (2014) Surgical process modelling: a review. Int J Comput Assist Radiol Surg 9(3):495–511
Lalys F, Riffaud L, Bouget D, Jannin P (2011) An application-dependent framework for the recognition of high-level surgical tasks in the OR. In: Medical image computing and computer-assisted intervention–MICCAI 2011, pp 331–338. Springer
Lalys F, Riffaud L, Morandi X, Jannin P (2011) Surgical phases detection from microscope videos by combining svm and hmm. In: Medical computer vision. Recognition Techniques and Applications in Medical Imaging, pp 54–62. Springer
Lea C, Hager GD, Vidal R (2015) An improved model for segmentation and recognition of fine-grained activities with application to surgical training tasks. In: Applications of computer vision (WACV), 2015 IEEE winter conference on, pp 1123–1129. IEEE
Loukas C, Georgiou E (2011) Multivariate autoregressive modeling of hand kinematics for laparoscopic skills assessment of surgical trainees. IEEE Trans Biomed Eng 58(11):3289– 3297
Loukas C, Georgiou E (2013) Surgical workflow analysis with Gaussian mixture multivariate autoregressive (GMMAR) models: a simulation study. Comput Aided Surg 18(3-4):47–62
Loukas C, Rouseas C, Georgiou E (2013) The role of hand motion connectivity in the performance of laparoscopic procedures on a virtual reality simulator. Med Biol Eng Comput 51(8):911– 922
Meißner C, Meixensberger J, Pretschner A, Neumuth T (2014) Sensor-based surgical activity recognition in unconstrained environments. Minim Invasiv Ther 23(4):198–205
Muaremi A, Arnrich B, Tröster G (2013) Towards measuring stress with smartphones and wearable devices during workday and sleep. J Bionanosci 3(2):172–183
Neumuth T, Meißner C (2012) Online recognition of surgical instruments by information fusion. Int J Comput Assist Radiol Surg 7(2):297–304
Overby DW, Watson RA (2014) Hand motion patterns of fundamentals of laparoscopic surgery certified and noncertified surgeons. Am J Surg 207(2):226–230
Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N (2012) Statistical modeling and recognition of surgical workflow. Med Image Anal 16(3):632–641
Phitayakorn R, Minehart RD, Hemingway MW, Pian-Smith MCM, Petrusa E (2015) Relationship between physiologic and psychological measures of autonomic activation in operating room teams during a simulated airway emergency. Am J Surg 209(1):86–92
Phitayakorn R, Minehart RD, Pian-Smith MCM, Hemingway MW, Petrusa ER (2015) Practicality of using galvanic skin response to measure intraoperative physiologic autonomic activation in operating room team members. Surgery 158(5):1415–20
Poddar P, Ahmidi N, Vedula SS, Ishii L, Hager GD (2014) Automated objective surgical skill assessment in the operating room using unstructured tool motion. Med Image Comput Comput Assist Interv
Prichard RS, O’Neill CJ, Oucharek JJ, Holmes CYV, Colinda YH, Delbridge LW, Sywak MS (2012) A prospective study of heart rate variability in endocrine surgery: surgical training increases consultant’s mental strain. J Surg Educ 69(4):453–8
Quellec G, Lamard M, Cochener B, Cazuguel G (2014) Real-time segmentation and recognition of surgical tasks in cataract surgery videos. IEEE T Med Imaging 33(12):2352–2360
Quellec G, Lamard M, Cochener B, Cazuguel G (2015) Real-time task recognition in cataract surgery videos using adaptive spatiotemporal polynomials. IEEE T Med Imaging 34(4):877– 887
Regehr G, MacRae H, Reznick RK, Szalay D (1998) Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73(9):993–997
Richstone L, Schwartz MJ, Seideman C, Cadeddu J, Marshall S, Kavoussi LR (2010) Eye metrics as an objective assessment of surgical skill. Ann Surg 252(1):177–182
Rieger A, Fenger S, Neubert S, Weippert M, Kreuzfeld S, Stoll R (2015) Psychophysical workload in the operating room: primary surgeon versus assistant. Surg Endosc 29(7):1990– 1998
Rieger A, Stoll R, Kreuzfeld S, Behrens K, Weippert M (2014) Heart rate and heart rate variability as indirect markers of surgeons’ intraoperative stress. Int Arch Occup Environ Health 87(2):165–174
Rutherford DN, D’Angelo ALD, Law KE, Pugh CM (2015) Advanced engineering technology for measuring performance. Surg Clin North Am 95(4):813–26
Schulz CM, Schneider E, Fritz L, Vockeroth J, Hapfelmeier A, Wasmaier M, Kochs EF, Schneider G (2011) Eye tracking for assessment of workload: a pilot study in an anaesthesia simulator environment. Brit J Anaesth 106(October 2010):44–50
Stauder R, Okur A, Peter L, Schneider A, Kranzfelder M, Feussner H, Navab N (2014) Random forests for phase detection in surgical workflow analysis. In: Information processing in computer-assisted interventions, pp 148–157. Springer
Stefanidis D, Wang F, Korndorffer JR, Dunne JB, Scott DJ (2010) Robotic assistance improves intracorporeal suturing performance and safety in the operating room while decreasing operator workload. Surg Endosc 24:377–382
Tao L, Elhamifar E, Khudanpur S, Hager GD, Vidal R (2012) Sparse hidden Markov models for surgical gesture classification and skill evaluation. In: Information processing in computer-assisted interventions, pp 167–177. Springer
Tao L, Zappella L, Hager GD, Vidal R (2013) Surgical gesture segmentation and recognition. In: Medical image computing and computer-assisted intervention–MICCAI 2013, pp 339–346. Springer
Thiemjarus S, James A, Yang GZ (2012) An eye-hand data fusion framework for pervasive sensing of surgical activities. Pattern Recogn 45(8):2855–2867
Tien G, Zheng B, Atkins MS (2011) Quantifying surgeons’ vigilance during laparoscopic operations using eyegaze tracking. St Heal T 163:658–662
Tien T, Pucher PH, Sodergren MH, Sriskandarajah K, Yang GZ, Darzi A (2014) Differences in gaze behaviour of expert and junior surgeons performing open inguinal hernia repair. Surg Endosc 2: 405–413
Tomasko JM, Pauli EM, Kunselman AR, Haluck RS (2012) Sleep deprivation increases cognitive workload during simulated surgical tasks. Am J Surg 203(1):37–43
Twinanda AP, Marescaux J, De Mathelin M, Padoy N (2014) Towards better laparoscopic video database organization by automatic surgery classification. In: Information processing in computer-assisted interventions, pp 186–195. Springer
Uemura M, Tomikawa M, Kumashiro R, Miao T, Souzaki R, Ieiri S, Ohuchida K, Lefor AT, Hashizume M (2014) Analysis of hand motion differentiates expert and novice surgeons. J Surg Resh 188(1):8–13
Unger M, Chalopin C, Neumuth T (2014) Vision-based online recognition of surgical activities. Int J Comput Assist Radiol Surg 9:979–986
Watson RA (2012) Computer-aided feedback of surgical knot tying using optical tracking. J Surg Educ 69 (3):306–310
Watson RA (2013) Quantification of surgical technique using an inertial measurement unit. Simul Healthc 8 (3):162–165
Watson RA (2014) Use of a machine learning algorithm to classify expertise: analysis of hand motion patterns during a simulated surgical task. Acad Med 89(8):1–5
Wilson M, McGrath J, Vine S, Brewer J, Defriend D, Masters R (2010) Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from experts. Surg Endosc 24(10):2458–2464
Wilson MR, McGrath JS, Vine SJ, Brewer J, Defriend D, Masters RSW (2011) Perceptual impairment and psychomotor control in virtual laparoscopic surgery. Surg Endosc 25:2268–2274
Wilson MR, Poolton JM, Malhotra N, Ngo K, Bright E, Masters RSW (2011) Development and validation of a surgical workload measure: the surgery task load index (SURG-TLX). World J Surg 35(9):1961–1969
Wilson MR, Vine SJ, Bright E, Masters RSW, Defriend D, McGrath JS (2011) Gaze training enhances laparoscopic technical skill acquisition and multi-tasking performance: a randomized, controlled study. Surg Endosc 25:3731–3739
Yurko YY, Scerbo MW, Prabhu AS, Acker CE, Stefanidis D (2010) Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simul Healthc 5(5):267–271
Zheng B, Cassera MA, Martinec DV, Spaun GO, Swanström LL (2010) Measuring mental workload during the performance of advanced laparoscopic tasks. Surg Endosc 24:45–50
Zheng B, Jiang X, Atkins MS (2015) Detection of changes in surgical difficulty: evidence from pupil responses. Surg Innov 22: 1–7
Zheng B, Jiang X, Tien G, Meneghetti A, Panton ONM, Atkins MS (2012) Workload assessment of surgeons: correlation between NASA TLX and blinks. Surg Endosc 26:2746–2750
Zheng B, Tien G, Atkins SM, Swindells C, Tanin H, Meneghetti A, Qayumi KA, Neely O, Panton M (2011) Surgeon’s vigilance in the operating room. Am J Surg 201(5):667–671
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pernek, I., Ferscha, A. A survey of context recognition in surgery. Med Biol Eng Comput 55, 1719–1734 (2017). https://doi.org/10.1007/s11517-017-1670-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11517-017-1670-6