Skip to main content
SpringerLink
Log in

Recognizing multi-modal sensor signals using evolutionary learning of dynamic Bayesian networks

  • Original Article
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Multi-modal context-aware systems can provide user-adaptive services, but it requires complicated recognition models with larger resources. The limitations to build optimal models and infer the context efficiently make it difficult to develop practical context-aware systems. We developed a multi-modal context-aware system with various wearable sensors including accelerometers, gyroscopes, physiological sensors, and data gloves. The system used probabilistic models to handle the uncertain and noisy time-series sensor data. In order to construct the efficient probabilistic models, this paper uses an evolutionary algorithm to model structure and EM algorithm to determine parameters. The trained models are selectively inferred based on a semantic network which describes the semantic relations of the contexts and sensors. Experiments with the real data collected show the usefulness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Lester J, Choudhury T, Kern N, Borriello G, Hannaford B (2005) A hybrid discriminative/generative approach for modeling human activities. Paper presented at the proceedings of the 19th international joint conference on artificial intelligence, Edinburgh, Scotland

  2. Choudhury T, Borriello G, Consolvo S, Haehnel D, Harrison B, Hemingway B, Hightower J, Klasnja PP, Koscher K, LaMarca A, Landay JA, LeGrand L, Lester J, Rahimi A, Rea A, Wyatt D (2008) The mobile sensing platform: an embedded activity recognition system. Pervasive Comput IEEE 7(2):32–41. doi:10.1109/MPRV.2008.39

    Google Scholar 

  3. Ogris G, Stiefmeier T, Lukowicz P, Troster G (2008) Using a complex multi-modal on-body sensor system for activity spotting. Paper presented at the proceedings of the 2008 12th IEEE international symposium on wearable computers

  4. Roy N, Gu T, Das SK (2010) Supporting pervasive computing applications with active context fusion and semantic context delivery. Pervasive Mobile Comput 6(1):21–42. doi:10.1016/j.pmcj.2009.10.002

    Article  Google Scholar 

  5. Van Laerhoven K, Aidoo KA, Lowette S (2001) Real-time analysis of data from many sensors with neural networks. In: Proceedings of the fifth international symposium on wearable computers, 2001, pp 115–122. doi:10.1109/iswc.2001.962112

  6. Wang Y, Lin J, Annavaram M, Jacobson QA, Hong J, Krishnamachari B, Sadeh N (2009) A framework of energy efficient mobile sensing for automatic user state recognition. Paper presented at the proceedings of the 7th international conference on mobile systems, applications, and services, Kraków, Poland

  7. Krause A, Ihmig M, Rankin E, Leong D, Gupta S, Siewiorek D, Smailagic A, Deisher M, Sengupta U (2005) Trading off prediction accuracy and power consumption for context-aware wearable computing. Paper presented at the proceedings of the ninth IEEE international symposium on wearable computers

  8. Murao K, Terada T, Takegawa Y, Nishio S (2008) A context-aware system that changes sensor combinations considering energy consumption. Paper presented at the proceedings of the 6th international conference on pervasive computing, Sydney, Australia

  9. Hwang K-S, Cho S-B (2009) Landmark detection from mobile life log using a modular Bayesian network model. Expert Syst Appl 36(10):12065–12076. doi:10.1016/j.eswa.2009.03.002

    Article  Google Scholar 

  10. Hong J-H, Yang S-I, Cho S-B (2010) ConaMSN: a context-aware messenger using dynamic Bayesian networks with wearable sensors. Expert Syst Appl 37(6):4680–4686. doi:10.1016/j.eswa.2009.12.040

    Article  Google Scholar 

  11. Bao L, Intille S (2004) Activity recognition from user-annotated acceleration data. doi:http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.68.5133

  12. Brezmes T, Gorricho J-L, Cotrina J (2009) Activity recognition from accelerometer data on a mobile phone. Paper presented at the proceedings of the 10th international work-conference on artificial neural networks: part II: distributed computing, artificial intelligence, bioinformatics, soft computing, and ambient assisted living, Salamanca, Spain

  13. Huỳnh T, Blanke U, Schiele B (2007) Scalable recognition of daily activities with wearable sensors. Paper presented at the Proceedings of the 3rd international conference on Location-and context-awareness, Oberpfaffenhofen, Germany

  14. Jatoba LC, Grossmann U, Kunze C, Ottenbacher J, Stork W (2008) Context-aware mobile health monitoring: evaluation of different pattern recognition methods for classification of physical activity. In: 30th annual international conference of the IEEE on Engineering in Medicine and Biology Society. EMBS 2008, 20–25 August 2008, pp 5250–5253. doi:10.1109/iembs.2008.4650398

  15. Godfrey A, Conway R, Meagher D, ÓLaighin G (2008) Direct measurement of human movement by accelerometry. Med Eng Phys 30(10):1364–1386. doi:10.1016/j.medengphy.2008.09.005

    Article  Google Scholar 

  16. Blum M, Pentland A, Troster G (2006) InSense: interest-based life logging. Multimedia IEEE 13(4):40–48. doi:10.1109/mmul.2006.87

    Article  Google Scholar 

  17. Maurer U, Rowe A, Smailagic A, Siewiorek D (2006) Location and activity recognition using ewatch: a wearable sensor platform. In: Yang C, Julio A (eds) Ambient intelligence in everyday Life. Springer, Berlin, pp 86–102

  18. Oliver N, Garg A, Horvitz E (2004) Layered representations for learning and inferring office activity from multiple sensory channels. Comput Vis Image Underst 96(2):163–180. doi:10.1016/j.cviu.2004.02.004

    Article  Google Scholar 

  19. Harrison BL, Consolvo S, Choudhury T (2010) Using multi-modal sensing for human activity modeling in the real world handbook of ambient intelligence and smart environments. In: Nakashima H, Aghajan H, Augusto JC (eds), Springer, Berlin, pp 463–478. doi:10.1007/978-0-387-93808-0_17

  20. Ermes M, Parkka J, Mantyjarvi J, Korhonen I (2008) Detection of daily activities and sports with wearable sensors in controlled and uncontrolled conditions. IEEE Trans Inf Technol Biomed 12(1):20–26. doi:10.1109/titb.2007.899496

    Article  Google Scholar 

  21. Oliver N, Horvitz E (2005) S-SEER: selective perception in a multimodal office activity recognition system. Paper presented at the proceedings of the first international conference on machine learning for multimodal interaction, Martigny, Switzerland

  22. Schmidt A, Aidoo KA, Takaluoma A, Tuomela U, Laerhoven KV, Velde WVd (1999) Advanced Interaction in Context. Paper presented at the proceedings of the 1st international symposium on handheld and ubiquitous computing, Karlsruhe, Germany

  23. Ming L, Rozgic V, Thatte G, Sangwon L, Emken A, Annavaram M, Mitra U, Spruijt-Metz D, Narayanan S (2010) Multimodal physical activity recognition by fusing temporal and cepstral information. IEEE Trans Neural Syst Rehabil Eng 18(4):369–380. doi:10.1109/tnsre.2010.2053217

    Article  Google Scholar 

  24. Marinov M, Zheliazkova I (2005) An interactive tool based on priority semantic networks. Know Based Syst 18(2–3):71–77. doi:10.1016/j.knosys.2004.08.001

    Article  Google Scholar 

  25. Kofod-Petersen A, Cassens J (2006) Using activity theory to model context awareness. In: Second international workshop on modeling and retrieval of context. MRC 2005, Edinburgh, July 31–August 1, pp 1–17. doi:10.1007/11740674_1

  26. Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning, 1st edn. Addison-Wesley Longman Publishing Co., Inc., Boston

    MATH  Google Scholar 

  27. Ling CX, Sheng VS (2008) Cost-sensitive learning and the class imbalance problem. Encyclopedia of machine learning. Springer, Berlin

  28. Thai-Nghe N, Gantner Z, Schmidt-Thieme L (2010) Cost-sensitive learning methods for imbalanced data. In: The 2010 international joint conference on neural networks (IJCNN), 18–23 July 2010, pp 1–8. doi:10.1109/ijcnn.2010.5596486

  29. Tapia EM (2008) Using machine learning for real-time activity recognition and estimation of energy expenditure. Massachusetts Institute of Technology, Cambridge

Download references

Acknowledgments

The authors would like to thank Dr. J.-K. Min, Mr. S.-I. Yang and Dr. J.-H. Hong for their help to implement the system used in this paper. This research is supported by Ministry of Culture, Sports and Tourism (MCST) and Korea Creative Content Agency (KOCCA) in the Culture Technology (CT) Research & Development Program.

Author information

Authors and Affiliations

  1. Department of Computer Science, Yonsei University, 50 Yonsei-ro, Sudaemoon-gu, Seoul, 120-749, Korea

    Young-Seol Lee & Sung-Bae Cho

Authors
  1. Young-Seol Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sung-Bae Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sung-Bae Cho.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lee, YS., Cho, SB. Recognizing multi-modal sensor signals using evolutionary learning of dynamic Bayesian networks. Pattern Anal Applic 17, 695–707 (2014). https://doi.org/10.1007/s10044-012-0300-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-012-0300-z

Keywords

Search

Navigation