Skip to main content
SpringerLink
Log in

Continuous activity recognition in a maintenance scenario: combining motion sensors and ultrasonic hands tracking

  • Industrial and Commercial Application
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

We describe the design and evaluation of pattern analysis methods for the recognition of maintenance-related activities. The presented work focuses on the spotting of subtle hand actions in a continuous stream of data based on a combination of body-mounted motion sensors and ultrasonic positioning. The spotting and recognition approach is based on three core ideas: (1) the use of location information to compensate for the ambiguity of hand motions, (2) the use of motion data to compensate for the slow sampling rate and unreliable signal of the low cost ultrasonic positioning system, and (3) an incremental, multistage spotting methodology. The proposed methods are evaluated in an elaborate bicycle repair experiment containing nearly 10 h of data from six subjects. The evaluation compares different strategies and system variants and shows that precision and recall rates around 90% can be achieved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. http://www.sonitor.com/.

  2. http://www.hexamite.com/.

  3. http://www.ascension-tech.com/realtime/.

  4. http://www.lukotronic.com/.

  5. The numbers are slightly bigger than those given in Ref. [52] because additional data were used for the presented evaluations.

  6. For an exhaustive survey on how to solve positioning problems using LSQ see [15].

  7. Refer to Ref. [19] or [56] for such an approach.

  8. The x-axis is defined in such a way that it is always pointing into the same direction the user is facing.

  9. A quantitative comparison of the two results would only be possible in case the absolute position of the subject would have been recorded. In such a case the relative trajectories could be transformed into the global reference system.

  10. Refer to [17] for a fast polynomial least-squares approximation. Applying such a method on the cost function remains future work.

  11. One-class SVMs with homogeneous polynomial kernel functions of degree 10.

References

  1. Addlesee M, Curwen R, Hodges S, Newman J, Steggles P, Ward A, Hopper A (2001) Implementing a sentient computing system. IEEE Comput Mag 34(8):50–56

    Google Scholar 

  2. Amft O (2005) MARKER—a Matlab time-series labeling toolbox. http://www2.ife.ee.ethz.ch/∼oamft/projects/marker/

  3. Amft O, Junker H, Tröster G (2005) Detection of eating and drinking arm gestures using inertial body-worn sensors. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05). Osaka, Japan, pp 160–163. doi:10.1109/ISWC.2005.17

  4. Andrew AH, Anokwa Y, Koscher K, Lester J, Borriello G (2007) Context to make you more aware. In: ICDCS Workshops, p 49. doi:10.1109/ICDCSW.2007.30

  5. Antifakos S, Michahelles F, Schiele B (2002) Proactive instructions for furniture assembly. In: Proceedings of the 4th international conference on ubiquitous computing, Gothenborg, Sweden, pp 351–360

  6. Arulampalam S, Maskell S, Gordon N, Clapp T (2001) A tutorial on particle filters for on-line non-linear/non-gaussian Bayesian tracking. IEEE Trans Signal Process 50:174–188

    Article  Google Scholar 

  7. Ashbrook D, Starner T (2003) Using GPS to learn significant locations and predict movement across multiple users. Pers Ubiquitous Comput 7(5):275–286. doi:10.1007/s00779-003-0240-0

    Google Scholar 

  8. Bannach D, Amft O, Lukowicz P (2008) Rapid prototyping of activity recognition applications. IEEE Pervasive Comput 7(2):22–31. doi:10.1109/MPRV.2008.36

    Google Scholar 

  9. Bannach D, Kunze K, Lukowicz P (2006) Distributed modular toolbox for multi-modal context recognition. In: Proceedings of 19th international conference on architecture of computing systems (ARCS 2006), Frankfurt/Main, Germany, pp 99–113. doi:10.1007/11682127

  10. Bao L, Intille SS (2004) Activity recognition from user-annotated acceleration data. In: Proceedings of the 2nd international conference on pervasive computing (Pervasive 2004), Linz, Austria, pp 1–17

  11. Bilmes JA (1998) A gentle tutorial of the EM algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. Tech. Rep. TR-97-021, Computer Science Division, Department of Electrical Engineering and Computer Science, U.C. Berkeley

  12. Brand M, Kettnaker V (2000) Discovery and segmentation of activities in video. IEEE Trans Pattern Anal Mach Intell 22(8):844–851. doi:10.1109/34.868685

    Google Scholar 

  13. Brashear H, Henderson V, Park KH, Hamilton H, Lee S, Starner T (2006) American sign language recognition in game development for deaf children. In: Proceedings of the 8th international ACM SIGACCESS conference on computers and accessibility (Assets 2006), Portland, OR, USA, pp 79–86

  14. Brown RG, Hwang PY (1997) Introduction to random signals and applied Kalman filtering, 3rd edn. Wiley, New York

  15. Cheung KW, So HC, Ma WK, Chan YT (2006) A constrained least squares approach to mobile positioning: algorithms and optimality. EURASIP J Appl Signal Process 2006:1–23

  16. Dey A, Salber D, Abowd G (2001) A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum Comput Interact Spec Issue Context Aware Comput 16(2–4):97–166

    Google Scholar 

  17. Elhay S, Golub GH, Kautsky J (1991) Updating and downdating of orthogonal polynomials with data fitting applications. SIAM J Matrix Anal Appl 12(2):327–353

    Article  MathSciNet  MATH  Google Scholar 

  18. Gellersen HW, Schmidt A, Beigl M (2002) Multi-sensor context-awareness in mobile devices and smart artifacts. MONET 7(5):341–351

    MATH  Google Scholar 

  19. Gupta N, Hauser R (2007) Kalman filtering with equality and inequality state constraints. Tech. rep. Oxford University Computing Laboratory, Oxford, UK

  20. Harter A, Hopper A, Steggles P, Ward A, Webster P (2002) The anatomy of a context-aware application. Wireless Netw 8(2–3):187–197

    Article  MATH  Google Scholar 

  21. Hazas M, Kray C, Gellersen H, Agbota H, Kortuem G, Krohn A (2005) A relative positioning system for co-located mobile devices. In: Proceedings of the 3rd international conference on mobile systems, applications, and services (MobiSys 2005), Seattle, pp 177–190. doi:10.1145/1067170.1067190

  22. Jason I, Simon F (2004) Optimized wavelet hand pose estimation for american sign language recognition. In: Proceedings of the congress on evolutionary computation (CEC 2004), Portland, OR, pp 797–802

  23. Junker H, Lukowicz P, Tröster G (2004) Continuous recognition of arm activities with wearable motion sensors. In: Proceedings of the 8th IEEE international symposium on wearable computers (ISWC’04), Arlington, VA

  24. Krause A, Smailagic A, Siewiorek D (2006) Context-aware mobile computing: learning context-dependent personal preferences from a wearable sensor array. IEEE Trans Mobile Comput 5(2):113–127

    Article  Google Scholar 

  25. Kunze K, Wagner F, Kartal E, Kluge EM, Lukowicz P (2009) Does context matter? A quantitative evaluation in a real world maintenance scenario. In: Proceedings of 7th international conference on pervasive computing, Nara, Japan, pp 372–389

  26. Lawo M, Herzog O, Witt H (2007) An industrial case study on wearable computing applications. In: Proceedings of the 9th ACM international conference on human computer interaction with mobile devices and services (MobileHCI 2007), Singapore, pp 448–451. doi:10.1145/1377999.1378052

  27. Lee C, Xu Y (1996) Online, interactive learning of gestures for human/robot interfaces. In: Proceedings of the IEEE international conference on robotics and automation (ICRA 1996), Minneapolis, MN, pp 2982–2987. doi:10.1109/ROBOT.1996.509165

  28. Lee HK, Kim JH (1998) Gesture spotting from continuous hand motion. Pattern Recognit Lett 19(5–6):513–520

    Article  Google Scholar 

  29. Lester J, Choudhury T, Borriello G (2006) A practical approach to recognizing physical activities. In: Proceedings of the 4th international conference on pervasive computing (Pervasive 2006), Dublin, pp 1–16

  30. Li W, Jonsson M (2006) Providing user self-contained location information using mobile phones. In: Advances in pervasive computing, adjunct proceedings of pervasive 2006, pp 33–36

  31. Lin J, Keogh E, Lonardi S, Chiu B (2003) A symbolic representation of time series, with implications for streaming algorithms. In: Proceedings of the 8th ACM SIGMOD workshop on research issues in data mining and knowledge discovery (DMKD 2003), San Diego, CA

  32. Lin J, Wei EKL, Lonardi S (2007) Experiencing SAX: a novel symbolic representation of time series. Data Min Knowl Discov J 15(2):107–144

    Article  MathSciNet  Google Scholar 

  33. Lukowicz P, Kirstein T, Tröster G (2004) Wearable systems for health care applications. Methods Inf Med 43(3):232–230

    Google Scholar 

  34. Lukowicz P, Timm-Giel A, Lawo M, Herzog O (2007) WearIT@work: toward real-world industrial wearable computing. IEEE Pervasive Comput 6(4):8–13

    Article  Google Scholar 

  35. Minnen D, Isbell C, Essa I, Starner T (2007) Detecting subdimensional motifs: An efficient algorithm for generalized multivariate pattern discovery. In: Proceedings of the 7th IEEE international conference on data mining (ICDM 2007), Omaha, NE, pp 601–606. doi:10.1109/ICDM.2007.52

  36. Mitra S, Acharya T (2007) Gesture recognition: A survey. IEEE Trans Syst Man Cybern C 37(3):311–324

    Article  Google Scholar 

  37. Müller M, Röder T, Clausen M (2005) Efficient content-based retrieval of motion capture data. In: Proceedings of the 35th international conference on computer graphics and interactive techniques (SIGGRAPH 2005), Los Angeles, CA, pp 677–685. doi:10.1145/1073204.1073247

  38. Naoum-Sawaya J, Slim M, Khawam S, Al-Alaoui MA (2006) Dynamic system design for american sign language recognition. In: Proceedings of the 2nd international symposium on communications, control, and signal processing (ISCCSP 2006), Marakesh, Morocco

  39. Ogris G (2009) Multi-modal on-body sensing of human activities. PhD thesis, University of Passau, Passau

  40. Ogris G, Stiefmeier T, Junker H, Lukowicz P, Tröster G (2005) Using ultrasonic hand tracking to augment motion analysis based recognition of manipulative gestures. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), Osaka, Japan, pp 152–159. doi:10.1109/ISWC.2005.54

  41. Patterson DJ, Fox D, Kautz H, Philipose M (2005) Fine-grained activity recognition by aggregating abstract object usage. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), Osaka, Japan, pp 44–51. doi:10.1109/ISWC.2005.22

  42. Poppe R (2010) A survey on vision-based human action recognition. Image Vis Comput 28(6):976–990

    Article  Google Scholar 

  43. Priyantha NB (2005) The cricket indoor location system. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA

  44. Roetenberg D (2006) Inertial and magnetic sensing of human motion. PhD thesis, University of Twente, Enschede

  45. Roetenberg D, Veltink PH (2005) Camera-marker and inertial sensor fusion for improved motion tracking. In: Proceedings of the European Society of Movement Analysis for Adults and Children (ESMAC 2005), Barcelona, pp 51–52

  46. Schölkopf B, Smola AJ (2002) Learning with kernels: support vector machines, regularization, optimization, and beyond (Adaptive computation and machine learning). MIT Press, Cambridge

  47. Siewiorek D, Smailagic A, Starner T (2008) Application design for wearable computing. Synthesis lectures on mobile and pervasive computing. Morgan and Claypool Publishers

  48. Smith A, Balakrishnan H, Goraczko M, Priyantha N (2004) Tracking moving devices with the Cricket location system. In: Proceedings of the 2nd international conference on mobile systems, applications, and services (MobiSys 2004), Boston, MA, pp 190–202. doi:10.1145/990064.990088

  49. Starner T, Schiele B, Pentland A (1998) Visual contextual awareness in wearable computing. In: Proceedings of the 2nd international symposium on wearable computers (ISWC’98), Pittsburgh, PA, pp 50–57. doi:10.1109/ISWC.1998.729529

  50. Starner T, Weaver J, Pentland A (1998) Real-time American sign language recognition using desk and wearable computer based video. IEEE Trans Pattern Anal Mach Intell 20(12):1371–1375. doi:10.1109/34.735811

  51. Stiefmeier T (2008) Real-time spotting of human activities in industrial environments. PhD thesis, ETH Zürich, Zurich

  52. Stiefmeier T, Ogris G, Junker H, Lukowicz P, Tröster G (2006) Combining motion sensors and ultrasonic hands tracking for continuous activity recognition in a maintenance scenario. In: Proceedings of the 10th IEEE international symposium on wearable computers (ISWC’06), Montreux, pp 97–104. doi:10.1109/ISWC.2006.286350

  53. Stiefmeier T, Roggen D, Ogris G, Lukowicz P, Tröster G (2008) Wearable activity tracking in car manufacturing. IEEE Pervasive Comput Spec Issue Activity Based Comput 7(2):42–50

    Google Scholar 

  54. Stiefmeier T, Roggen D, Tröster G (2007) Fusion of string-matched templates for continuous activity recognition. In: Proceedings of the 11th IEEE international symposium on wearable computers (ISWC’07), Boston, MA, pp 41–44

  55. Stiefmeier T, Roggen D, Tröster G (2007) Gestures are strings: Efficient online gesture spotting and classification using string matching. In: Proceedings of the 2nd international conference on body area networks (BodyNets 2007), Florence, Italy

  56. Ungarala S, Dolence E, Li K (2007) Constrained extended Kalman filter for nonlinear state estimation. In: Proceedings of the 8th international IFAC symposium on dynamics and control of process systems (DYCOPS 2007), Cancún, pp 63–68

  57. Ward A (1998) Sensor-driven computing. PhD thesis, University of Cambridge, Cambridge, UK

  58. Ward A, Jones A, Hopper A (1997) A new location technique for the active office. IEEE Personal Commun 4(5):42–47

    Article  Google Scholar 

  59. Ward JA (2006) Activity monitoring: continuous recognition and performance evaluation. PhD thesis, ETH Zürich, Zurich

  60. Ward JA, Lukowicz P, Tröster G, Starner TE (2006) Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans Pattern Anal Mach Intell 28(10):1553–1567. doi:10.1109/TPAMI.2006.197

    Google Scholar 

  61. Westeyn T, Vadas K, Bian X, Starner T, Abowd G (2005) Recognizing mimicked autistic self-stimulatory behaviors using HMMs. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), pp 164–169

  62. Wijnalda G, Pauws S, Vignoli F, Stuckenschmidt H (2005) A personalized music system for motivation in sport performance. IEEE Pervasive Comput 4(3):26–32

    Article  Google Scholar 

  63. Yamato J, Ohya J, Ishii K (1992) Recognizing human action in time-sequential images using Hidden Markov Models. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR 1992), Champaign, IL, pp 379–385. doi:10.1109/CVPR.1992.223161

  64. Yoon HS, Soh J, Bae YJ, Yang HS (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognit 34(7):1491–1501. doi:10.1016/S0031-3203(00)00096-0

  65. Zinnen A, Laerhoven KV, Schiele B (2007) Toward recognition of short and non-repetitive activities from wearable sensors. In: Proceedings of the European conference on ambient intelligence (AMI 2007), Darmstadt, pp 142–158

  66. Zinnen A, Wojek C, Schiele B (2009) Multi activity recognition based on bodymodel-derived primitives. In: 4th international symposium on location and context awareness (LoCA 2009), Tokyo

Download references

Acknowledgments

This work was supported by the European Union under contract EC IP 004216.

Author information

Authors and Affiliations

  1. Department of Biomechanics, Kinesiology and Applied Computer Science, University of Vienna, Vienna, Austria

    Georg Ogris

  2. Embedded Systems Lab. (ESL), University of Passau, Passau, Germany

    Paul Lukowicz

  3. Wearable Computing Lab. (WearLab), ETH Zürich, Zurich, Switzerland

    Thomas Stiefmeier & Gerhard Tröster

Authors
  1. Georg Ogris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Lukowicz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Stiefmeier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gerhard Tröster
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Georg Ogris.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ogris, G., Lukowicz, P., Stiefmeier, T. et al. Continuous activity recognition in a maintenance scenario: combining motion sensors and ultrasonic hands tracking. Pattern Anal Applic 15, 87–111 (2012). https://doi.org/10.1007/s10044-011-0216-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-011-0216-z

Keywords

Search

Navigation