Skip to main content
SpringerLink
Log in

Semantic event fusion of computer vision and ambient sensor data for activity recognition to support dementia care

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Although many Ambient Intelligence frameworks either address heterogeneous ambient sensing or computer vision techniques, very limited work integrates both techniques in the scope of activity recognition in pervasive environments. This paper presents such a framework that integrates both a computer vision component and heterogeneous sensors with unanimous semantic representation and interpretation, while it also addresses challenges for realistic applications, such as fast, efficient image analysis and ontology-based temporal interpretation models. The framework is validated through an application in clinical dementia assessment yielding positive results and fruitful conclusions for the proposed semantic fusion of vision and sensor observations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Sennheiser FreePORT—http://en-de.sennheiser.com/.

  2. Philips DTI-2 non-commercial wristwatch kindly provided by Philips Research NL—http://www.philips.nl/.

  3. Circle, Cirlce + and Stealth products by Plugwise.nl—https://www.plugwise.nl/.

  4. The ZigBee Alliance—http://www.zigbee.org/.

  5. Tags, PIR KumoSensor, Reed KumoSensor of the Wireless Sensor Tag System—http://wirelesstag.net/.

  6. http://www.loa.istc.cnr.it/ontologies/DUL.owl.

  7. http://www.w3.org/TR/owl-time/.

  8. The native OWL semantics do not support temporal reasoning. However, it can be simulated using custom property assertions, as described in (Riboni et al. 2011).

  9. The ontologies, sample dataset and implementation can be found online at: http://www.demcare.eu/results/ontologies.

References

  • Avgerinakis K, Briassouli A, Kompatsiaris I (2016) Activity detection using sequential statistical boundary detection (ssbd). Comput Vis Image 144:46–61

    Article  Google Scholar 

  • Avgerinakis K, Briassouli A, Kompatsiaris I (2013) Recognition of activities of daily living for smart home environments. In: 2013 9th International Conference on Intelligent Environments. IEEE, pp 173–180

  • Bettini C, Brdiczka O, Henricksen K et al (2010) A survey of context modelling and reasoning techniques. Pervasive Mob Comput 6:161–180.

    Article  Google Scholar 

  • Bonner SG (1998) Assisted interactive dwelling house. In: Proc. 3rd TIDE congress: technology for inclusive design and equality improving the quality of life for the European citizen. p 25

  • Capone A, Barros M, Hrasnica H, Tompros S (2009) A new architecture for reduction of energy consumption of home appliances. In: TOWARDS eENVIRONMENT, European conference of the Czech Presidency of the Council of the EU. pp 1–8

  • Chang YJ, Chen CH, Lin LF, Han RP, Huang WT, Lee GC (2012) Wireless sensor networks for vital signs monitoring: application in a nursing home. Int J Distrib Sensor Netw. doi:10.1155/2012/685107

  • Chatfield K, Lempitsky VS, Vedaldi A, Zisserman A (2011) The devil is in the details: an evaluation of recent feature encoding methods. In: BMVC, vol 2, no 4. p 8

  • Chen CY, Grauman K (2012) Efficient activity detection with max-subgraph search. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 1274–1281

  • Chen L, Nugent C (2009) Ontology-based activity recognition in intelligent pervasive environments. Int J Web Inf Syst 5:410–430

    Article  Google Scholar 

  • Chen L, Nugent CD, Wang H (2012) A knowledge-driven approach to activity recognition in smart homes. Knowl Data Eng IEEE Trans 24:961–974.

    Article  Google Scholar 

  • Dawadi PN, Cook DJ, Schmitter-Edgecombe M, Parsey C (2013) Automated assessment of cognitive health using smart home technologies. Technol Health Care 21(4):323–343

    Article  Google Scholar 

  • Demongeot J, Virone G, Duchêne F et al (2002) Multi-sensors acquisition, data fusion, knowledge mining and alarm triggering in health smart homes for elderly people. C R Biol 325:673–682

    Article  Google Scholar 

  • Derpanis KG, Sizintsev M, Cannons K, Wildes RP (2010) Efficient action spotting based on a spacetime oriented structure representation. In: IEEE conference on computer vision and pattern recognition

  • De Paola A, Gaglio S, Lo Re G, Ortolani M (2012) Sensor9k: a testbed for designing and experimenting with WSN-based ambient intelligence applications. Pervasive Mob Comput 8:448–466.

    Article  Google Scholar 

  • Duchenne O, Laptev I, Sivic J, Bach F (2009) IEEE international conference in computer vision. In: Automatic annotation of human actions in video

  • Eisenhauer M, Rosengren P, Antolin P (2010) Hydra: a development platform for integrating wireless devices and sensors into ambient intelligence systems. In: The Internet of Things. Springer, pp 367–373

  • Friedewald M, Costa O Da, Punie Y et al (2005) Perspectives of ambient intelligence in the home environment. Telemat Inf 22:221–238

    Article  Google Scholar 

  • Gauzere B, Greco C, Ritrovato P et al (2015) Semantic web technologies for object tracking and video analytics. Lect Notes Comput Sci (Subser Lect Notes Artif Intell Lect Notes Bioinform) 9475:574–585. doi:10.1007/978-3-319-27863-6_53

    Article  Google Scholar 

  • Greco L, Ritrovato P, Saggese A, Vento M (2016) Abnormal event recognition: a hybrid approach using semantic web. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 58–65

  • Gu T, Wang XH, Pung HK, Zhang DQ (2004) An ontology-based context model in intelligent environments. In: Proceedings of communication networks and distributed systems modeling and simulation conference, pp 270–275

  • Gámez N, Fuentes L (2011) FamiWare: a family of event-based middleware for ambient intelligence. Pers Ubiquitous Comput 15:329–339.

    Article  Google Scholar 

  • Helal S, Mann W, King J et al (2005) The gator tech smart house: a programmable pervasive space. Computer (Long Beach Calif) 38:50–60

    Google Scholar 

  • Jain M, Van Gemert J, Jegou H et al (2014) Action localization with tubelets from motion. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 740–747. doi:10.1109/CVPR.2014.100

  • Kerssens C, Kumar R, Adams AE, Knott CC, Matalenas L, Sanford JA, Rogers WA (2015) Personalized technology to support older adults with and without cognitive impairment living at home. Am J Alzheimers Dis Other Demen 30:85–97

  • Kleinberger T, Becker M, Ras E et al (2007) Ambient intelligence in assisted living: enable elderly people to handle future interfaces. In: Universal access in human–computer interaction. Ambient interaction. Springer, pp 103–112

  • Kläser A, Marszałek M, Schmid C, Zisserman A (2012) Human focused action localization in video. In: Kutulakos KN (ed) Trends and topics in computer vision. ECCV 2010. Lecture notes in computer science, vol 6553. Springer, Berlin

  • Lan T, Wang Y, Mori G (2011) Discriminative figure-centric models for joint action localization and recognition. In: IEEE International conference in computer vision

  • Laptev I, Patrick P (2007) Retrieving actions in movies. In: IEEE international conference in computer vision

  • Le Gal C, Martin J, Lux A, Crowley JL (2001) Smart office: design of an intelligent environment. IEEE Intell Syst 16:60–66

    Article  Google Scholar 

  • Ma S, Zhang J, Ikizler-Cinbis N, Sclaroff S (2013) Action recognition and localization by hierarchical space–time segments. 2013 IEEE Int Conf Comput Vis 2744–2751. doi:10.1109/ICCV.2013.341

  • Meditskos G, Dasiopoulou S, Efstathiou V, Kompatsiaris I (2013) Ontology patterns for complex activity modelling. In: International workshop on rules and rule markup languages for the semantic web. Springer, Berlin, pp 144–157

  • Nevatia R, Hobbs J, Bolles B (2004) An ontology for video event representation. In: Conference on computer vision and pattern recognition workshop, 2004. CVPRW’04. IEEE, pp 119–119

  • Okeyo G, Chen L, Wang H, Sterritt R (2014) Dynamic sensor data segmentation for real-time knowledge-driven activity recognition. Pervasive Mob Comput 10:155–172.

    Article  Google Scholar 

  • Oneata D, Verbeek J, Schmid C (2014) Efficient action localization with approximately normalized fisher vectors. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2545–2552. doi:10.1109/CVPR.2014.326

  • Patkos T, Chrysakis I, Bikakis A et al (2010) A reasoning framework for ambient intelligence. In: Artificial intelligence: theories, models and applications. Springer, pp 213–222

  • Prince M, Bryce R, Albanese E, Wimo A, Ribeiro W, Ferri CP (2013) The global prevalence of dementia: a systematic review and metaanalysis. Alzheimers Dement 9(1):63–75

    Article  Google Scholar 

  • Riboni D, Bettini C (2011) COSAR: hybrid reasoning for context-aware activity recognition. Pers Ubiquitous Comput 15:271–289

    Article  Google Scholar 

  • Riboni D, Pareschi L, Radaelli L, Bettini C (2011) Is ontology-based activity recognition really effective? In: Pervasive computing and communications workshops. IEEE, pp 427–431

  • Shaw R, Troncy R, Hardman L (2009) Lode: linking open descriptions of events. In: The semantic web. Springer, pp 153–167

  • Sirin E, Parsia B, Grau BC, Kalyanpur A, Katz Y (2007) Pellet: a practical owl-dl reasoner. J Web Semant 5(2):51–53

    Article  Google Scholar 

  • Stavropoulos TG, Gottis K, Vrakas D, Vlahavas I (2013) aWESoME: a web service middleware for ambient intelligence. Expert Syst Appl 40:4380–4392.

    Article  Google Scholar 

  • Stavropoulos TG, Kontopoulos E, Bassiliades N et al (2014a) Rule-based approaches for energy savings in an ambient intelligence environment. Pervasive Mob Comput. doi:10.1016/j.pmcj.2014.05.001

  • Stavropoulos TG, Meditskos G, Kontopoulos E, Kompatsiaris I (2014b) The DemaWare service-oriented AAL platform for people with dementia. Artif Intell Assist Med (AI-AM/NetMed 2014) 11

  • Stevenson G, Knox S, Dobson S, Nixon P (2009) Ontonym: a collection of upper ontologies for developing pervasive systems. In: Proceedings of the 1st workshop on context, information and ontologies. ACM, p 9

  • Stevenson GT, Ye J, Dobson SA, Nixon P (2010) Loc8: a location model and extensible framework for programming with location. IEEE Pervasive Comput 9(1):28–37. doi:10.1109/MPRV.2009.90

  • Sun C, Tappen M, Foroosh H (2014) Feature-independent action spotting without human localization, segmentation, or frame-wise tracking. In: IEEE conference on computer vision and pattern recognition, pp 2689–2696

  • Suzuki R, Otake S, Izutsu T, Yoshida M, Iwaya T (2006) Monitoring daily living activities of elderly people in a nursing home using an infrared motion-detection system. Telemed J Health 12(2):146–155

  • Tamura T, Togawa T, Ogawa M, Yoda M (1998) Fully automated health monitoring system in the home. Med Eng Phys 20:573–579

    Article  Google Scholar 

  • Tran D, Yuan J (2012) Max-margin structured output regression for spatio-temporal action localization. Adv Neural Inf Process Syst 25 359–367

    Google Scholar 

  • Weiser M (1991) The computer for the 21st century. Sci Am 265:94–104

    Article  Google Scholar 

  • Willems G, Becker JH, Tuytelaars T, Van Gool L (2009) Exemplar-based action recognition in video. Br Mach Vis Conf 1–11. doi:10.5244/C.23.90

  • Wolf P, Schmidt A, Otte JP et al (2010) openAAL-the open source middleware for ambient-assisted living (AAL). In: AALIANCE conference, Malaga, Spain. pp 1–5

  • Ye J, Coyle L, Dobson S, Nixon P (2007) Ontology-based models in pervasive computing systems. Knowl Eng Rev 22:315–347

    Article  Google Scholar 

  • Yuan J, Liu Z, Research M, Wu Y (2009) Discriminative subvolume search for efficient action detection. 2009 IEEE Conf Comput Vis Pattern Recognit, pp 2442–2449. doi:10.1109/CVPRW.2009.5206671

Download references

Author information

Authors and Affiliations

  1. Center for Research and Technologies, Information Technologies Institute, Hellas, Greece

    Thanos G. Stavropoulos, Georgios Meditskos, Stelios Andreadis, Konstantinos Avgerinakis, Katerina Adam & Ioannis Kompatsiaris

Authors
  1. Thanos G. Stavropoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Georgios Meditskos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stelios Andreadis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Konstantinos Avgerinakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Katerina Adam
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ioannis Kompatsiaris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thanos G. Stavropoulos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stavropoulos, T.G., Meditskos, G., Andreadis, S. et al. Semantic event fusion of computer vision and ambient sensor data for activity recognition to support dementia care. J Ambient Intell Human Comput 11, 3057–3072 (2020). https://doi.org/10.1007/s12652-016-0437-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-016-0437-5

Keywords

Search

Navigation