Skip to main content

Advertisement

SpringerLink
Log in

Vision-based patient monitoring: a comprehensive review of algorithms and technologies

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Vision-based monitoring for assisted living is gaining increasing attention, especially in multi-modal monitoring systems owing to the several advantages of vision-based sensors. In this paper, a detailed survey of some of the important vision-based patient monitoring applications is presented, namely (a) fall detection (b) action and activity monitoring (c) sleep monitoring (d) respiration and apnea monitoring (e) epilepsy monitoring (f) vital signs monitoring and (g) facial expression monitoring. The challenges and state-of-art technologies in each of these applications is presented. This is the first work to present such a comprehensive survey with the focus on a set of seven most common applications pertaining to patient monitoring. Potential future directions are presented while also considering practical large scale deployment of vision-based systems in patient monitoring. One of the important conclusions drawn is that rather than applying generic algorithms, use of the application context of patient monitoring can be a useful way to develop novel techniques that are robust and yet cost-effective.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • AL-Khalidi F, Saatchi R, Burke D, Elphick H (2010) Tracking human face features in thermal images for respiration monitoring. In: 2010 IEEE/ACS Intl. Conf. on Computer Systems and Applications (AICCSA), pp 1–6

  • AL-Khalidi F, Saatchi R, Burke D, Elphick H, Tan S (2011) Respiration rate monitoring methods: a review. Pediatr Pulmonol 46(6):523–529

    Article  Google Scholar 

  • Alghowinem S, Goecke R, Wagner M, Parker G, Breakspear M (2013a) Eye movement analysis for depression detection. In: 2013 20th IEEE Intl Conf on Image Processing (ICIP), pp 4220–4224

  • Alghowinem S, Goecke R, Wagner M, Parkerx G, Breakspear M (2013b) Head Pose and Movement Analysis as an Indicator of Depression. In: 2013 Humaine Association Conf. on Affective Computing and Intelligent Interaction (ACII), pp 283–288

  • Alkali A, Saatchi R, Elphick H, Burke D (2013) Facial Tracking in Thermal Images for Real-Time Noncontact Respiration Rate Monitoring. In: Modelling Symposium (EMS), 2013 European, pp 265–270

  • Alvino C, Kohler C, Barrett F, Gur RE, Gur RC, Verma R (2007) Computerized measurement of facial expression of emotions in schizophrenia. J Neurosci Methods 163(2):350–361

    Article  Google Scholar 

  • Amoretti M, Copelli S, Wientapper F, Furfari F, Lenzi S, Chessa S (2011) Sensor data fusion for activity monitoring in the PERSONA ambient assisted living project. J Ambient Intell Human Comput 4(1):67–84

    Article  Google Scholar 

  • Anderson D, Luke RH, Keller JM, Skubic M, Rantz M, Aud M (2009) Linguistic Summarization of Video for Fall Detection Using Voxel Person and Fuzzy Logic. Comput Vis Image Underst 113(1):80–89

    Article  Google Scholar 

  • Aoki H, Koshiji K, Nakamura H, Takemura Y, Nakajima M (2005) Study on respiration monitoring method using near-infrared multiple slit-lights projection. In: 2005 IEEE Intl. Symposium on Micro-NanoMechatronics and Human Science, pp 291–296

  • Ashraf AB, Prkachin K, Chen T, Lucey S, Solomon P, Ambadar Z, Cohn JF, Theobald BJ (2007) The painful face—pain expression recognition using active appearance models. In: In 9th Intl Conf on Multimodal Interfaces ICMI, pp 9–14

  • Ashraf AB, Lucey S, Cohn JF, Chen T, Ambadar Z, Prkachin KM, Solomon PE (2009) The painful face—pain expression recognition using active appearance models. Image Vis Comput 27(12):1788–1796

    Article  Google Scholar 

  • Auvinet E, Reveret L, St-Arnaud A, Rousseau J, Meunier J (2008) Fall detection using multiple cameras. In: 30th Annual Intl Conf of the IEEE Engineering in Medicine and Biology Society, 2008. EMBS 2008, pp 2554–2557

  • Auvinet E, Rougier C, Meunier J, St-Arnaud A, Rousseau J (2010) Multiple cameras fall dataset. DIRO-Université de Montréal, Tech Rep 1350

  • Auvinet E, Multon F, Saint-Arnaud A, Rousseau J, Meunier J (2011) Fall detection with multiple cameras: an occlusion-resistant method based on 3-D silhouette vertical distribution. IEEE Trans Inf Technol Biomed 15(2):290–300

    Article  Google Scholar 

  • Bai YW, Li WT, Chen YW (2010) Design and implementation of an embedded monitor system for detection of a patient’s breath by double Webcams. In: 2010 IEEE Intl Workshop on Medical Measurements and Applications Proceedings (MeMeA), pp 171–176

  • Bai YW, Tsai CL, Wu SC (2012) Design of a breath detection system with multiple remotely enhanced hand-computer interaction devices. In: 2012 IEEE 16th Intl. Symposium on Consumer Electronics (ISCE), pp 1–5

  • Balakrishnan G, Durand F, Guttag J (2013) Detecting Pulse from Head Motions in Video. In: 2013 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp 3430–3437

  • Banerjee T, Keller J, Skubic M, Stone E (2014) Day or night activity recognition from video using fuzzy clustering techniques. IEEE Trans Fuzzy Syst 22(3):483–493

    Article  Google Scholar 

  • Belbachir A, Schraml S, Nowakowska A (2011) Event-driven stereo vision for fall detection. In: 2011 IEEE Computer Society Conf on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 78–83

  • Belbachir A, Litzenberger M, Schraml S, Hofstatter M, Bauer D, Schon P, Humenberger M, Sulzbachner C, Lunden T, Merne M (2012) CARE: A dynamic stereo vision sensor system for fall detection. In: 2012 IEEE Intl Symposium on Circuits and Systems (ISCAS), pp 731–734

  • Bevilacqua V, D’Ambruoso D, Mandolino G, Suma M (2011) A new tool to support diagnosis of neurological disorders by means of facial expressions. In: 2011 IEEE Intl. Workshop on Medical Measurements and Applications Proc. (MeMeA), pp 544–549

  • Bieber G, Hoffmeyer A, Gutzeit E, Peter C, Urban B (2009) Activity Monitoring by Fusion of Optical and Mechanical Tracking Technologies for User Behavior Analysis. In: Proc. of the 2Nd Intl. Conf. on PErvasive Technologies Related to Assistive Environments, ACM, PETRA ’09, pp 45:1–45:6

  • Bin Mansor MN, Yaacob S, Nagarajan R, Hariharan M (2010) Detection of facial changes for hospital ICU patients. In: 2010 6th Intl. Colloquium on Signal Processing and Its Applications (CSPA), pp 1–5

  • Brulin D, Benezeth Y, Courtial E (2012) Posture recognition based on fuzzy logic for home monitoring of the elderly. IEEE Trans Inf Technol Biomed 16(5):974–982

    Article  Google Scholar 

  • Brusco N, Paviotti A (2005) 3d quantification of facial edemas. In: Proc. of the 4th Intl. Symposium on Image and Signal Processing and Analysis, 2005. ISPA 2005, pp 191–196

  • Calhoun DA (2010) Sleep and hypertension. CHEST J 138(2):434

    Article  Google Scholar 

  • Chaaraoui AA, Climent-Prez P, Flrez-Revuelta F (2012) A review on vision techniques applied to human behaviour analysis for ambient-assisted living. Expert Syst Appl 39(12):10873–10888

    Article  Google Scholar 

  • Charfi I, Miteran J, Dubois J, Atri M, Tourki R (2012) Definition and Performance Evaluation of a Robust SVM Based Fall Detection Solution. In: 2012 Eighth Intl. Conf. on Signal Image Technology and Internet Based Systems (SITIS), pp 218–224

  • Charfi I, Miteran J, Dubois J, Atri M, Tourki R (2013) Optimized spatio-temporal descriptors for real-time fall detection: comparison of support vector machine and Adaboost-based classification. J Electron Imaging 22(4):041106

    Article  Google Scholar 

  • Chekmenev SY, Rara H, Farag AA (2005) Non-contact, wavelet-based measurement of vital signs using thermal imaging. In: The first Intl. Conf on graphics, vision, and image processing (GVIP), pp 107–112

  • Chen LL, Chen KW, Hung YP (2014) A sleep monitoring system based on audio, video and depth information for detecting sleep events. In: 2014 IEEE Intl. Conf. on Multimedia and Expo (ICME), pp 1–6

  • Chen MY (2010) Long Term Activity Analysis in Surveillance Video Archives. Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA, USA

  • Cheng H, Liu Z, Zhao Y, Ye G (2011) Real world activity summary for senior home monitoring. In: 2011 IEEE Intl. Conf. on Multimedia and Expo (ICME), pp 1–4

  • Chung PC, Liu CD (2008) A daily behavior enabled hidden Markov model for human behavior understanding. Pattern Recognit 41(5):1572–1580

    Article  MATH  Google Scholar 

  • Cohn J, Kruez T, Matthews I, Yang Y, Nguyen MH, Padilla M, Zhou F, De la Torre F (2009) Detecting depression from facial actions and vocal prosody. In: 3rd Intl. Conf. on Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009, pp 1–7

  • Crispim C, Bathrinarayanan V, Fosty B, Konig A, Romdhane R, Thonnat M, Bremond F (2013) Evaluation of a monitoring system for event recognition of older people. In: 2013 10th IEEE Intl. Conf. on Advanced Video and Signal Based Surveillance (AVSS), pp 165–170

  • Cucchiara R, Prati A, Vezzani R (2007) A multi-camera vision system for fall detection and alarm generation. Expert Syst 24(5):334–345

    Article  Google Scholar 

  • Cuppens K, Lagae L, Vanrumste B (2009) Towards automatic detection of movement during sleep in pediatric patients with epilepsy by means of video recordings and the optical flow algorithm. In: 4th European Conference of the International Federation for Medical and Biological Engineering, Springer, pp 784–789

  • Cuppens K, Lagae L, Ceulemans B, Huffel SV, Vanrumste B (2010) Automatic video detection of body movement during sleep based on optical flow in pediatric patients with epilepsy. Med Biol Eng Comput 48(9):923–931

    Article  Google Scholar 

  • Cuppens K, Chen CW, Wong KBY, Van de Vel A, Lagae L, Ceulemans B, Tuytelaars T, Van Huffel S, Vanrumste B, Aghajan H (2012) Using Spatio-Temporal Interest Points (STIP) for myoclonic jerk detection in nocturnal video. Conf Proc IEEE Eng Med Biol Soc 2012:4454–4457

    Google Scholar 

  • Dai Y, Shibata Y, Ishii T, Hashimoto K, Katamachi K, Noguchi K, Kakizaki N, Cai D (2001) An associate memory model of facial expressions and its application in facial expression recognition of patients on bed. In: IEEE Intl. Conf. on Multimedia and Expo, 2001. ICME 2001, pp 591–594, 00014

  • Dickinson P, Hunter A (2008) Using Inactivity to Detect Unusual behavior. In: IEEE Workshop on Motion and video Computing, 2008. WMVC 2008, pp 1–6

  • Diraco G, Leone A, Siciliano P (2010) An active vision system for fall detection and posture recognition in elderly healthcare. In: Design, Automation Test in Europe Conf Exhibition (DATE), 2010, pp 1536–1541

  • Eguchi K, Hoshide S, Ishikawa S, Shimada K, Kario K (2010) Short sleep duration is an independent predictor of stroke events in elderly hypertensive patients. J Am Soc Hypertens 4(5):255–262

    Article  Google Scholar 

  • Ekman P, Friesen W (1978) Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA

  • Ellgring H (1989) Non verbal Communication in Depression. Cambridge University Press, Cambridge, England

  • Engel J, Epilepsy Intl League Against, (ILAE), (2001) A proposed diagnostic scheme for people with epileptic seizures and with epilepsy: report of the ILAE Task Force on Classification and Terminology. Epilepsia 42(6):796–803

  • Espa F, Ondze B, Deglise P, Billiard M, Besset A (2000) Sleep architecture, slow wave activity, and sleep spindles in adult patients with sleepwalking and sleep terrors. Clin Neurophysiol 111(5):929–939

    Article  Google Scholar 

  • Falie D, Ichim M, David L (2008) Respiratory Motion Visualization and the Sleep Apnea Diagnosis with the Time of Flight (TOF) Camera

  • Falie D, David L, Ichim M (2009) Statistical algorithm for detection and screening sleep apnea. In: Intl. Symposium on Signals, Circuits and Systems, 2009. ISSCS 2009, pp 1–4

  • Fei J, Pavlidis I (2006) Analysis of breathing air flow patterns in thermal imaging. In: Engineering in Medicine and Biology Society, 2006. EMBS’06. 28th Annual International Conference of the IEEE, IEEE, pp 946–952

  • Fei J, Pavlidis I, Murthy J (2009) Thermal vision for sleep apnea monitoring. Med Image Comput Comput Assist Interv 12(Pt 2):1084–1091

    Google Scholar 

  • Fleck S, Strasser W (2008) Smart camera based monitoring system and its application to assisted living. Proc IEEE 96(10):1698–1714

    Article  Google Scholar 

  • Foo Siang Fook V, Thang PV, Htwe TM, Qiang Q, Wai AAP, Jayachandran M, Biswas J, Yap P (2007) Automated Recognition of Complex Agitation Behavior of Dementia Patients Using Video Camera. 2007 9th Intl. Conf. on e-Health Networking, Application and Services, pp 68–73

  • Frigola M, Amat J, Pagès J (2002) Vision Based Respiratory Monitoring System. In: Proceedings of the 10th Mediterranean Conference on Control and AutomationMED2002 Lisbon, Portugal

  • Fu Z, Delbruck T, Lichtsteiner P, Culurciello E (2008) An address-event fall detector for assisted living applications. IEEE Trans Biomed Circuits Syst 2(2):88–96

    Article  Google Scholar 

  • Gade R, Moeslund TB (2013) Thermal cameras and applications: a survey. Mach Vis Appl 25(1):245–262

    Article  Google Scholar 

  • Gault T, Farag A (2013) A Fully Automatic Method to Extract the Heart Rate from Thermal Video. In: 2013 IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 336–341

  • Gault T, Blumenthal N, Farag A, Starr T (2010) Extraction of the superficial facial vasculature, vital signs waveforms and rates using thermal imaging. In: 2010 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1–8

  • Gholami B, Haddad WM, Tannenbaum AR (2009) Agitation and pain assessment using digital imaging. Conf Proc IEEE Eng Med Biol Soc 2009:2176–2179

    Google Scholar 

  • Gilbert CA, Lilley CM, Craig KD, McGrath PJ, Court CA, Bennett SM, Montgomery CJ (1999) Postoperative pain expression in preschool children: validation of the child facial coding system. Clin J Pain 15(3):192–200

    Article  Google Scholar 

  • Gohil B, Gholamhhosseini H, Harrison MJ, Lowe A, Al-Jumaily A (2007) Intelligent monitoring of critical pathological events during anesthesia. Conf Proc IEEE Eng Med Biol Soc 2007:4343–4346

    Google Scholar 

  • Grassi M, Lombardi A, Rescio G, Malcovati P, Malfatti M, Gonzo L, Leone A, Diraco G, Distante C, Siciliano P, Libal V, Huang J, Potamianos G (2008) A hardware-software framework for high-reliability people fall detection. In: 2008 IEEE Sensors, pp 1328–1331

  • Hamm J, Kohler CG, Gur RC, Verma R (2011) Automated Facial Action Coding System for Dynamic Analysis of Facial Expressions in Neuropsychiatric Disorders. J Neurosci Methods 200(2):237–256

    Article  Google Scholar 

  • Hammal Z, Cohn JF (2012) Automatic Detection of Pain Intensity. In: Proc. of the 14th ACM Intl. Conf. on Multimodal Interaction, ACM, pp 47–52

  • Hammal Z, Kunz M (2012) Pain monitoring: a dynamic and context-sensitive system. Pattern Recognit 45(4):1265–1280

    Article  Google Scholar 

  • Hawley-Hague H, Boulton E, Hall A, Pfeiffer K, Todd C (2014) Older adults perceptions of technologies aimed at falls prevention, detection or monitoring: a systematic review. Intl J Med Inf 83(6):416–426

    Article  Google Scholar 

  • Hazelhoff L, Han J, With PH (2008) Video-Based Fall Detection in the Home Using Principal Component Analysis. In: Proceedings of the 10th Intl Conf on Advanced Concepts for Intelligent Vision Systems, Springer-Verlag, Berlin, Heidelberg, ACIVS ’08, pp 298–309

  • Hijaz F, Afzal N, Ahmad T, Hasan O (2010) Survey of fall detection and daily activity monitoring techniques. In: 2010 Intl Conf on Information and Emerging Technologies (ICIET), pp 1–6

  • Hirayama M, Nakamura T, Hori N, Koike Y, Sobue G (2008) The video images of sleep attacks in Parkinson’s disease. Mov Disord 23(2):288–290

    Article  Google Scholar 

  • Horgas A, Miller L (2008) Pain assessment in people with dementia. AJN Am J Nurs 108(7):62–70

    Article  Google Scholar 

  • Humenberger M, Schraml S, Sulzbachner C, Belbachir A, Srp A, Vajda F (2012) Embedded fall detection with a neural network and bio-inspired stereo vision. In: 2012 IEEE Computer Society Conf on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 60–67

  • Iasemidis L (2003) Epileptic seizure prediction and control. IEEE Trans Biomed Eng 50(5):549–558

    Article  Google Scholar 

  • Igual R, Medrano C, Plaza I (2013) Challenges, issues and trends in fall detection systems. BioMed Eng OnLine 12(1):66

    Article  Google Scholar 

  • Jansen B, Deklerck R (2006) Context aware inactivity recognition for visual fall detection. Pervasive Health Conf Workshops 2006:1–4

    Google Scholar 

  • Jansen B, Temmermans F, Deklerck R (2007) 3d human pose recognition for home monitoring of elderly. In: 29th Annual Intl Conf of the IEEE Engineering in Medicine and Biology Society, 2007. EMBS 2007, pp 4049–4051

  • Jansen B, Rebel S, Deklerck R, Mets T, Schelkens P (2008) Detection of activity pattern changes among elderly with 3d camera technology. In: Photonics Europe, International Society for Optics and Photonics, pp 70,000O–70,000O

  • Jiang G, Kang L (2007) Character Analysis of Facial Expression Thermal Image. In: IEEE/ICME Intl. Conf. on Complex Medical Engineering, 2007. CME 2007, pp 824–827

  • Johnson KD, Winkelman C, Burant CJ, Dolansky M, Totten V (2014) The factors that affect the frequency of vital sign monitoring in the emergency department. J Emerg Nurs 40(1):27–35

    Article  Google Scholar 

  • Johnson ML, Price PA, Jovanov E (2007) A new method for the quantification of breathing. In: Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, IEEE, pp 4568–4571

  • Joshi J, Dhall A, Goecke R, Breakspear M, Parker G (2012) Neural-net classification for spatio-temporal descriptor based depression analysis. In: 2012 21st Intl. Conf. on Pattern Recognition (ICPR), pp 2634–2638

  • Joshi J, Goecke R, Alghowinem S, Dhall A, Wagner M, Epps J, Parker G, Breakspear M (2013a) Multimodal assistive technologies for depression diagnosis and monitoring. J Multimodal User Interfaces 7(3):217–228

    Article  Google Scholar 

  • Joshi J, Goecke R, Parker G, Breakspear M (2013b) Can body expressions contribute to automatic depression analysis? In: 2013 10th IEEE Intl. Conf. and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–7

  • Joumier V, Romdhane R, Bremond F, Thonnat M, Mulin E, Robert P, Derreumaux A, Piano J, Lee J (2011) Video Activity Recognition Framework for assessing motor behavioural disorders in Alzheimer Disease Patients. In: International Workshop on Behaviour Analysis and Video Understanding (ICVS 2011), p 9

  • Junior CFC, Bremond F, Joumier V (2012) A Multi-Sensor Approach for Activity Recognition in Older Patients. In: Second International Conference on Ambient Computing, Applications, Services and Technologies

  • Kalitzin S, Petkov G, Velis D, Vledder B, Lopes da Silva F (2012) Automatic segmentation of episodes containing epileptic clonic seizures in video sequences. IEEE Trans Biomed Eng 59(12):3379–3385

    Article  Google Scholar 

  • Kanade T, Cohn J, Tian Y (2000) Comprehensive database for facial expression analysis. In: Fourth IEEE Intl. Conf. on Automatic Face and Gesture Recognition, 2000. Proc., pp 46–53

  • Karaman S, Benois-Pineau J, Mgret R, Dovgalecs V, Dartigues JF, Gastel Y (2010) Human Daily Activities Indexing in Videos from Wearable Cameras for Monitoring of Patients with Dementia Diseases. In: 2010 20th Intl. Conf. on Pattern Recognition (ICPR), pp 4113–4116

  • Kayyali HA, Weimer S, Frederick C, Martin C, Basa D, Juguilon JA, Jugilioni F (2008) Remotely attended home monitoring of sleep disorders. Telemed J E Health 14(4):371–374

    Article  Google Scholar 

  • Khan R, Meyer A, Konik H, Bouakaz S (2013) Pain detection through shape and appearance features. In: 2013 IEEE Intl. Conf. on Multimedia and Expo (ICME), pp 1–6

  • Khan Z, Sohn W (2011) Abnormal human activity recognition system based on R-transform and kernel discriminant technique for elderly home care. IEEE Trans Consum Electron 57(4):1843–1850

    Article  Google Scholar 

  • Kosmopoulos D, Antonakaki P, Valasoulis K, Katsoulas D (2008) Monitoring Human Behavior in an Assistive Environment Using Multiple Views. In: Proc. of the 1st Intl. Conf. on PErvasive Technologies Related to Assistive Environments, ACM, New York, NY, USA, PETRA ’08, pp 32:1–32:6

  • Kranjec J, Begu S, Gerak G, Drnovek J (2014) Non-contact heart rate and heart rate variability measurements: a review. Biomed Signal Process Control 13:102–112

    Article  Google Scholar 

  • Kroutil J, Laposa A, Husak M (2011) Respiration Monitoring During Sleeping. In: Proc. of the 4th Intl. Symposium on Applied Sciences in Biomedical and Communication Technologies, ACM, New York, NY, USA, ISABEL ’11, pp 33:1–33:5

  • Kunz M, Scharmann S, Hemmeter U, Schepelmann K, Lautenbacher S (2007) The facial expression of pain in patients with dementia. PAIN 133(1–3):221–228

    Article  Google Scholar 

  • Kuo YM, Lee JS, Pc Chung (2010) A visual context-awareness-based sleeping-respiration measurement system. IEEE Trans Inf Technol Biomed 14(2):255–265

    Article  Google Scholar 

  • Kwolek B, Kepski M (2014) Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput Methods Programs Biomed 117(3):489–501

    Article  Google Scholar 

  • Kwon S, Kim H, Park KS (2012) Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone. In: 2012 Annual Intl. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 2174–2177

  • Leone A, Diraco G, Siciliano P (2011) Detecting falls with 3d range camera in ambient assisted living applications: a preliminary study. Med Eng Phys 33(6):770–781

    Article  Google Scholar 

  • Lewandowska M, Ruminski J, Kocejko T, Nowak J (2011) Measuring pulse rate with a webcam; A non-contact method for evaluating cardiac activity. In: 2011 Federated Conf. on Computer Science and Information Systems (FedCSIS), pp 405–410

  • Li X, Chen J, Zhao G, Pietikainen M (2014) Remote Heart Rate Measurement from Face Videos under Realistic Situations. In: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, IEEE, pp 4264–4271

  • Li Z, da Silva A, Cunha J (2002) Movement quantification in epileptic seizures: a new approach to video-EEG analysis. IEEE Trans Biomed Eng 49(6):565–573

    Article  Google Scholar 

  • Liao W, Zhang W, Zhu Z, Ji Q (2005) A Real-Time Human Stress Monitoring System Using Dynamic Bayesian Network. In: IEEE Computer Society Conf. on Computer Vision and Pattern Recognition—Workshops, 2005. CVPR Workshops, pp 70–70

  • Liao WH, Yang CM (2008) Video-based activity and movement pattern analysis in overnight sleep studies. In: 19th Intl Conf on Pattern Recognition, 2008. ICPR 2008, pp 1–4

  • Littlewort GC, Bartlett MS, Lee K (2007) Faces of Pain: Automated Measurement of Spontaneousallfacial Expressions of Genuine and Posed Pain. In: Proc. of the 9th Intl. Conf. on Multimodal Interfaces, ACM, New York, NY, USA, ICMI ’07, pp 15–21

  • Liu Q, Sun M, Sclabassi R (2004) Illumination-invariant change detection model for patient monitoring video. In: 26th Annual Intl. Conf. of the IEEE Engineering in Medicine and Biology Society, 2004. IEMBS ’04, vol 1, pp 1782–1785

  • Lu H, Pan Y, Mandal B, Eng HL, Guan C, Chan DWS (2013) Quantifying limb movements in epileptic seizures through color-based video analysis. IEEE Trans Biomed Eng 60(2):461–469

    Article  Google Scholar 

  • Lucey P, Cohn J, Lucey S, Matthews I, Sridharan S, Prkachin KM (2009) Automatically detecting pain using facial actions. Int Conf Affect Comput Intell Interact Workshops 2009:1–8

    Google Scholar 

  • Lucey P, Cohn J, Matthews I, Lucey S, Sridharan S, Howlett J, Prkachin K (2011a) Automatically detecting pain in video through facial action units. IEEE Trans Syst Man Cybern Part B: Cybern 41(3):664–674

    Article  Google Scholar 

  • Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I (2011b) Painful data: The unbc-mcmaster shoulder pain expression archive database. In: Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, IEEE, pp 57–64

  • Lucey P, Cohn JF, Prkachin KM, Solomon PE, Chew S, Matthews I (2012) Painful monitoring: automatic pain monitoring using the UNBC-McMaster shoulder pain expression archive database. Image Vis Comput 30(3):197–205

    Article  Google Scholar 

  • Makantasis K, Protopapadakis E, Doulamis A, Grammatikopoulos L, Stentoumis C (2012) Monocular Camera Fall Detection System Exploiting 3d Measures: A Semi-supervised Learning Approach. In: Fusiello A, Murino V, Cucchiara R (eds) Computer Vision ECCV 2012. Workshops and Demonstrations, no. 7585 in Lecture Notes in Computer Science, Springer Berlin Heidelberg, pp 81–90

  • Malakuti K, Albu A (2010) Towards an Intelligent Bed Sensor: Non-intrusive Monitoring of Sleep Irregularities with Computer Vision Techniques. In: 2010 20th Intl Conf on Pattern Recognition (ICPR), pp 4004–4007

  • Mandal MK, Pandey R, Prasad AB (1998) Facial expressions of emotions and schizophrenia: a review. Schizophr Bull 24(3):399–412

    Article  Google Scholar 

  • Manfredi PL, Breuer B, Meier DE, Libow L (2003) Pain assessment in elderly patients with severe dementia. J Pain Symptom Manag 25(1):48–52

    Article  Google Scholar 

  • Martinez M, Stiefelhagen R (2012) Breath rate monitoring during sleep using near-ir imagery and PCA. In: 2012 21st Intl. Conf. on Pattern Recognition (ICPR), pp 3472–3475

  • Mastorakis G, Makris D (2012) Fall detection system using Kinects infrared sensor. J Real-Time Image Proc pp 1–12

  • Matic A, Mehta P, Rehg J, Osmani V, Mayora O (2010) AID-ME: Automatic identification of dressing failures through monitoring of patients and activity Evaluation. In: Pervasive Computing Technologies for Healthcare (PervasiveHealth), 2010 4th Intl. Conf. on-NO PERMISSIONS, pp 1–8

  • McDuff D, El Kaliouby R, Senechal T, Amr M, Cohn J, Picard R (2013) Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected In-the-Wild. In: 2013 IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 881–888

  • McIntyre G, Gocke R, Hyett M, Green M, Breakspear M (2009) An approach for automatically measuring facial activity in depressed subjects. In: 3rd Intl. Conf. on Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009, pp 1–8

  • Mihailidis A, Carmichael B, Boger J (2004) The use of computer vision in an intelligent environment to support aging-in-place, safety, and independence in the home. IEEE Trans Inf Technol Biomed 8(3):238–247

    Article  Google Scholar 

  • Mirmahboub B, Samavi S, Karimi N, Shirani S (2013) Automatic monocular system for human fall detection based on variations in silhouette area. IEEE Trans Biomed Eng 60(2):427–436

    Article  Google Scholar 

  • Mishima K, Sugahara T (2009) Analysis methods for facial motion. Jpn Dental Sci Rev 45(1):4–13

    Article  Google Scholar 

  • Mubashir M, Shao L, Seed L (2013) A survey on fall detection: principles and approaches. Neurocomputing 100:144–152

    Article  Google Scholar 

  • Murthy R, Pavlidis I, Tsiamyrtzis P (2004) Touchless monitoring of breathing function. Conf Proc IEEE Eng Med Biol Soc 2:1196–1199

    Google Scholar 

  • Nait-Charif H, McKenna S (2004) Activity summarisation and fall detection in a supportive home environment. In: Proceedings of the 17th Intl Conf on Pattern Recognition, 2004. ICPR 2004, vol 4, pp 323–326 Vol.4

  • Nakajim K, Matsumoto Y, Tamura T (2001) Development of real-time image sequence analysis for evaluating posture change and respiratory rate of a subject in bed. Physiol Meas 22(3):N21–28

    Article  Google Scholar 

  • Nakajima K, Matsumoto Y, Tamura T (2000) A monitor for posture changes and respiration in bed using real time image sequence analysis. In: Proc. of the 22nd Annual Intl. Conf. of the IEEE Engineering in Medicine and Biology Society, 2000, vol 1, pp 51–54 vol.1

  • Nanni L, Brahnam S, Lumini A (2010) A local approach based on a local binary patterns variant texture descriptor for classifying pain states. Expert Syst Appl 37(12):7888–7894

    Article  Google Scholar 

  • Nasution A, Emmanuel S (2007) Intelligent Video Surveillance for Monitoring Elderly in Home Environments. In: IEEE 9th Workshop on Multimedia Signal Processing, 2007. MMSP 2007, pp 203–206

  • Naufal Mansor M, Yaacob S, Nagarajan R, Che LS, Hariharan M, Ezanuddin M (2010) Detection of facial changes for ICU patients using KNN classifier. In: 2010 Intl. Conf. on Intelligent and Advanced Systems (ICIAS), pp 1–5

  • Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K (2002) Nurse-staffing levels and the quality of care in hospitals. N Engl J Med 346(22):1715–1722

    Article  Google Scholar 

  • Neubauer DN (1999) Sleep problems in the elderly. Am Fam Phys 59(9):2551–2558, 2559–2560

  • Ni B, Dat NC, Moulin P (2012) RGBD-camera based get-up event detection for hospital fall prevention. In: 2012 IEEE Intl Conf on Acoustics, Speech and Signal Processing (ICASSP), pp 1405–1408

  • Nicolas Thome SM (2008) A real-time, multi-view fall detection system: a LHMM-based approach. IEEE Trans Circuits Syst Video Technol 18:1522–1532

    Article  Google Scholar 

  • Noachtar S, Peters AS (2009) Semiology of epileptic seizures: a critical review. Epilepsy Behav 15(1):2–9

    Article  Google Scholar 

  • Noury N, Fleury A, Rumeau P, Bourke A, Laighin G, Rialle V, Lundy JE (2007) Fall detection—Principles and Methods. In: 29th Annual Intl Conf of the IEEE Engineering in Medicine and Biology Society, 2007. EMBS 2007, pp 1663–1666

  • Obdrzalek S, Kurillo G, Ofli F, Bajcsy R, Seto E, Jimison H, Pavel M (2012) Accuracy and robustness of Kinect pose estimation in the context of coaching of elderly population. In: 2012 Annual Intl. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 1188–1193

  • Olivieri DN, Gmez Conde I, Vila Sobrino XA (2012) Eigenspace-based fall detection and activity recognition from motion templates and machine learning. Expert Syst Appl 39(5):5935–5945

    Article  Google Scholar 

  • Pantelopoulos A, Bourbakis NG (2010) A survey on wearable sensor-based systems for health monitoring and prognosis. IEEE Trans Syst Man Cybern Part C-Appl Rev 40(1):1–12

    Article  Google Scholar 

  • Paradiso R (2003) Wearable health care system for vital signs monitoring. In: 4th Intl. IEEE EMBS Special Topic Conf. on Information Technology Applications in Biomedicine, 2003, pp 283–286

  • Pediaditis M, Tsiknakis M, Vorgia P, Kafetzopoulos D, Danilatou V, Fotiadis D (2010) Vision-based human motion analysis in epilepsy—Methods and challenges. In: 2010 10th IEEE Intl. Conf. on Information Technology and Applications in Biomedicine (ITAB), pp 1–5

  • Pediaditis M, Tsiknakis M, Koumakis L, Karachaliou M, Voutoufianakis S, Vorgia P (2012) Vision-based absence seizure detection. Conf Proc IEEE Eng Med Biol Soc 2012:65–68

    Google Scholar 

  • Peng YT, Lin CY, Sun MT, Feng MW (2006) Sleep condition inferencing using simple multimodality sensors. In: Circuits and Systems, 2006. ISCAS 2006. Proceedings. 2006 IEEE International Symposium on, IEEE, pp 4–pp

  • Planinc R, Kampel M (2013) Introducing the use of depth data for fall detection. Pers Ubiquit Comput 17(6):1063–1072

    Article  Google Scholar 

  • Poh MZ, McDuff DJ, Picard RW (2010) Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt Express 18(10):10762–10774

    Article  Google Scholar 

  • Prkachin KM (1992) The consistency of facial expressions of pain: a comparison across modalities. Pain 51(3):297–306

    Article  Google Scholar 

  • Prkachin KM (2009) Assessing pain by facial expression: facial expression as nexus. Pain Res Manag 14(1):53–58

    Article  Google Scholar 

  • Rashidi P, Mihailidis A (2013) A survey on ambient-assisted living tools for older adults. IEEE J Biomed Health Inf 17(3):579–590

    Article  Google Scholar 

  • Rodrigues J, Estevao M, Malaquias J, Santos P, Gouveia G, Simoes J (2007) SleepAtHome—Portable Home Based System for Pediatric Sleep Apnoea Diagnosis. In: IEEE Intl. Conf. on Portable Information Devices, 2007. PORTABLE 07, pp 1–4

  • Rogers CR, Schmidt KL, VanSwearingen JM, Cohn JF, Wachtman GS, Manders EK, Deleyiannis FWB (2007) Automated facial image analysis: detecting improvement in abnormal facial movement after treatment with botulinum toxin A. Ann Plast Surg 58(1):39–47

    Article  Google Scholar 

  • Rougier C, Auvinet E, Rousseau J, Mignotte M, Meunier J (2011a) Fall Detection from Depth Map Video Sequences. In: Abdulrazak B, Giroux S, Bouchard B, Pigot H, Mokhtari M (eds) Toward Useful Services for Elderly and People with Disabilities, no. 6719 in Lecture Notes in Computer Science, Springer Berlin Heidelberg, pp 121–128

  • Rougier C, Meunier J, St-Arnaud A, Rousseau J (2011b) Robust video surveillance for fall detection based on human shape deformation. IEEE Trans Circuits Syst Video Technol 21(5):611–622

    Article  Google Scholar 

  • Rougier C, Meunier J, St-Arnaud A, Rousseau J (2013) 3d head tracking for fall detection using a single calibrated camera. Image Vis Comput 31(3):246–254

    Article  Google Scholar 

  • Sathyanarayana S, Satzoda RK, Sathyanarayana S, Thambipillai S (2014) Block-Based Search Space Reduction Technique for Face Detection Using Shoulder and Head Curves. In: Image and Video Technology, no. 8333 in Lecture Notes in Computer Science, Springer Berlin Heidelberg, pp 385–396

  • Sathyanarayana S, Satzoda R, Sathyanarayana S, Thambipillai S (2015a) Identifying epileptic seizures based on a template-based eyeball detection technique. In: 2015 IEEE Intl Conf on Image Processing (ICIP), p [in Press]

  • Sathyanarayana S, Satzoda R, Sathyanarayana S, Thambipillai S (2015b) WellCam: Dataset for Vision-based Patient Wellness Monitoring. CVPR Workshop on The Future of Datasets in Vision

  • Sato I, Nakajima M (2005) Non-contact Breath Motion Monitor ing System in Full Automation. In: Engineering in Medicine and Biology Society, 2005. IEEE-EMBS 2005. 27th Annual Intl. Conf. of the, pp 3448–3451

  • Seki H (2009) Fuzzy inference based non-daily behavior pattern detection for elderly people monitoring system. Conf Proc IEEE Eng Med Biol Soc 2009:6187–6192

    Google Scholar 

  • Sikka K, Dhall A, Bartlett MS (2014) Classification and weakly supervised pain localization using multiple segment representation. Image Vis Comput 32(10):659–670

    Article  Google Scholar 

  • Somers VK, White DP, Amin R, Abraham WT, Costa F, Culebras A, Daniels S, Floras JS, Hunt CE, Olson LJ, Pickering TG, Russell R, Woo M, Young T (2008) Sleep apnea and cardiovascular disease. J Am Coll Cardiol 52(8):686–717

    Article  Google Scholar 

  • Stevens S, Bharucha A (2003) Caremedia: Automated video and sensor analysis for geriatric care. Carnegie Mellon University

  • Sun N, Garbey M, Merla A, Pavlidis I (2005) Imaging the cardiovascular pulse. In: IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol 2, pp 416–421 vol. 2

  • Tabar AM, Keshavarz A, Aghajan H (2006) Smart Home Care Network Using Sensor Fusion and Distributed Vision-based Reasoning. In: Proceedings of the 4th ACM Intl Workshop on Video Surveillance and Sensor Networks, ACM, New York, NY, USA, VSSN ’06, pp 145–154

  • Takemura Y, Jy Sato, Nakajima M (2005) A respiratory movement monitoring system using fiber-grating vision sensor for diagnosing sleep apnea syndrome. OPT REV 12(1):46–53

    Article  Google Scholar 

  • Toreyin B, Dedeoglu Y, Cetin A (2006) HMM Based Falling Person Detection Using Both Audio and Video. In: Signal Processing and Communications Applications, 2006 IEEE 14th, pp 1–4

  • UN (2012) World population ageing 1950–2050. Department of Economic and Social Affairs, Population Division, United Nations, http://www.un.org/esa/population/publications/worldageing19502050/

  • Van Kasteren TLM (2011) Activity Recognition for Health Monitoring Elderly Using Temporal Probabilistic Models. ASCI

  • Viola P, Jones M (2001) Robust real-time object detection. Int J Comput Vis 4:51–52

    Google Scholar 

  • Vishwakarma S, Agrawal A (2012) A survey on activity recognition and behavior understanding in video surveillance. Vis Comput 29(10):983–1009

    Article  Google Scholar 

  • Wang CW, Ahmed A, Hunter A (2006) Vision analysis in detecting abnormal breathing activity in application to diagnosis of obstructive sleep apnoea. Conf Proc IEEE Eng Med Biol Soc 1:4469–4473

    Google Scholar 

  • Wang CW, Ahmed A, Hunter A (2007) Locating the upper body of covered humans in application to diagnosis of obstructive sleep apnea. In: World Congress on Engineering, pp 662–667

  • Wang CW, Hunter A, Gravill N, Matusiewicz S (2010) Real time pose recognition of covered human for diagnosis of sleep apnoea. Comput Med Imaging Graph 34(6):523–533

    Article  Google Scholar 

  • Wang CW, Hunter A, Gravill N, Matusiewicz S (2014) Unconstrained video monitoring of breathing behavior and application to diagnosis of sleep apnea. IEEE Trans Biomed Eng 61(2):396–404

    Article  Google Scholar 

  • Wang F, Stone E, Dai W, Banerjee T, Giger J, Krampe J, Rantz M, Skubic M (2009) Testing an in-home gait assessment tool for older adults. Conf Proc IEEE Eng Med Biol Soc 2009:6147–6150

    Google Scholar 

  • Wang P, Barrett F, Martin E, Milanova M, Gur RE, Gur RC, Kohler C, Verma R (2008) Automated video based facial expression analysis of neuropsychiatric disorders. J Neurosci Methods 168(1):224–238

    Article  Google Scholar 

  • Wang S, Xu Z, Yang Y, Li X, Pang C, Haumptmann AG (2013) Fall Detection in Multi-camera Surveillance Videos: Experimentations and Observations. In: Proceedings of the 1st ACM Intl Workshop on Multimedia Indexing and Information Retrieval for Healthcare, ACM, New York, NY, USA, MIIRH ’13, pp 33–38

  • Werner P, Al-Hamadi A, Niese R (2012) Pain recognition and intensity rating based on Comparative Learning. In: 2012 19th IEEE Intl. Conf. on Image Processing (ICIP), pp 2313–2316

  • Werner P, Al-Hamadi A, Niese R, Walter S, Gruss S, Traue HC (2013) Towards Pain Monitoring: Facial Expression, Head Pose, a new Database, an Automatic System and Remaining Challenges. In: Proceedings of the British Machine Vision Conference

  • Wiesner S, Yaniv Z (2007) Monitoring patient respiration using a single optical camera. Conf Proc IEEE Eng Med Biol Soc 2007:2740–2743

    Google Scholar 

  • Yang FC, Kuo CH, Tsai MY, Huang SC (2003) Image-based sleep motion recognition using artificial neural networks. In: 2003 Intl. Conf on Machine Learning and Cybernetics, vol 5, pp 2775–2780 Vol.5

  • Yu M, Rhuma A, Naqvi S, Wang L, Chambers J (2012) A posture recognition-based fall detection system for monitoring an elderly person in a smart home environment. IEEE Trans Inf Technol Biomed 16(6):1274–1286

    Article  Google Scholar 

  • Yu MC, Wu H, Liou JL, Lee MS, Hung YP (2013) Multiparameter Sleep Monitoring Using a Depth Camera. In: Gabriel J, Schier J, Huffel SV, Conchon E, Correia C, Fred A, Gamboa H (eds) Biomedical Engineering Systems and Technologies, no. 357 in Communications in Computer and Information Science, Springer Berlin Heidelberg, pp 311–325

  • Yu X (2008) Approaches and principles of fall detection for elderly and patient. In: 10th Intl Conf on e-health Networking, Applications and Services, 2008. HealthCom 2008, pp 42–47

  • Zambanini S, Machajdik J, Kampel M (2010) Detecting falls at homes using a network of low-resolution cameras. In: 2010 10th IEEE Intl Conf on Information Technology and Applications in Biomedicine (ITAB), pp 1–4

  • Zhan K, Ramos F, Faux S (2012) Activity recognition from a wearable camera. In: 2012 12th Intl. Conf. on Control Automation Robotics Vision (ICARCV), pp 365–370

  • Zhang C, Tian Y (2012) Rgb-d camera-based daily living activity recognition. J Comput Vis Image Process 2(4):12

    Google Scholar 

  • Zhang Z, Kapoor U, Narayanan M, Lovell N, Redmond S (2011) Design of an unobtrusive wireless sensor network for nighttime falls detection. In: 2011 Annual Intl Conf of the IEEE Engineering in Medicine and Biology Society,EMBC, pp 5275–5278

  • Zhong H, Shi J, Visontai M (2004) Detecting unusual activity in video. In: Proc. of the 2004 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 2004. CVPR 2004, vol 2, pp II–819–II–826 Vol.2

  • Zhou Z, Dai W, Eggert J, Giger J, Keller J, Rantz M, He Z (2009) A real-time system for in-home activity monitoring of elders. In: Annual Intl. Conf. of the IEEE Engineering in Medicine and Biology Society, 2009. EMBC 2009, pp 6115–6118

  • Zhu Z, Fei J, Pavlidis I (2005) Tracking human breath in infrared imaging. In: Fifth IEEE Symposium on Bioinformatics and Bioengineering, 2005. BIBE 2005, pp 227–231

  • Zouba N, Bremond F, Thonnat M, Vu VT (2007) Multi-sensors Analysis for Everyday Activity Monitoring. Proc of SETIT pp 25–29

Download references

Author information

Authors and Affiliations

  1. Nanyang Technological University, Singapore, Singapore

    Supriya Sathyanarayana & Srikanthan Thambipillai

  2. University of California, San Diego, USA

    Ravi Kumar Satzoda & Suchitra Sathyanarayana

Authors
  1. Supriya Sathyanarayana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ravi Kumar Satzoda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Suchitra Sathyanarayana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Srikanthan Thambipillai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Supriya Sathyanarayana.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sathyanarayana, S., Satzoda, R., Sathyanarayana, S. et al. Vision-based patient monitoring: a comprehensive review of algorithms and technologies. J Ambient Intell Human Comput 9, 225–251 (2018). https://doi.org/10.1007/s12652-015-0328-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-015-0328-1

Keywords

Search

Navigation