Abstract
We describe the design and evaluation of pattern analysis methods for the recognition of maintenance-related activities. The presented work focuses on the spotting of subtle hand actions in a continuous stream of data based on a combination of body-mounted motion sensors and ultrasonic positioning. The spotting and recognition approach is based on three core ideas: (1) the use of location information to compensate for the ambiguity of hand motions, (2) the use of motion data to compensate for the slow sampling rate and unreliable signal of the low cost ultrasonic positioning system, and (3) an incremental, multistage spotting methodology. The proposed methods are evaluated in an elaborate bicycle repair experiment containing nearly 10 h of data from six subjects. The evaluation compares different strategies and system variants and shows that precision and recall rates around 90% can be achieved.
Similar content being viewed by others
Notes
The numbers are slightly bigger than those given in Ref. [52] because additional data were used for the presented evaluations.
For an exhaustive survey on how to solve positioning problems using LSQ see [15].
The x-axis is defined in such a way that it is always pointing into the same direction the user is facing.
A quantitative comparison of the two results would only be possible in case the absolute position of the subject would have been recorded. In such a case the relative trajectories could be transformed into the global reference system.
Refer to [17] for a fast polynomial least-squares approximation. Applying such a method on the cost function remains future work.
One-class SVMs with homogeneous polynomial kernel functions of degree 10.
References
Addlesee M, Curwen R, Hodges S, Newman J, Steggles P, Ward A, Hopper A (2001) Implementing a sentient computing system. IEEE Comput Mag 34(8):50–56
Amft O (2005) MARKER—a Matlab time-series labeling toolbox. http://www2.ife.ee.ethz.ch/∼oamft/projects/marker/
Amft O, Junker H, Tröster G (2005) Detection of eating and drinking arm gestures using inertial body-worn sensors. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05). Osaka, Japan, pp 160–163. doi:10.1109/ISWC.2005.17
Andrew AH, Anokwa Y, Koscher K, Lester J, Borriello G (2007) Context to make you more aware. In: ICDCS Workshops, p 49. doi:10.1109/ICDCSW.2007.30
Antifakos S, Michahelles F, Schiele B (2002) Proactive instructions for furniture assembly. In: Proceedings of the 4th international conference on ubiquitous computing, Gothenborg, Sweden, pp 351–360
Arulampalam S, Maskell S, Gordon N, Clapp T (2001) A tutorial on particle filters for on-line non-linear/non-gaussian Bayesian tracking. IEEE Trans Signal Process 50:174–188
Ashbrook D, Starner T (2003) Using GPS to learn significant locations and predict movement across multiple users. Pers Ubiquitous Comput 7(5):275–286. doi:10.1007/s00779-003-0240-0
Bannach D, Amft O, Lukowicz P (2008) Rapid prototyping of activity recognition applications. IEEE Pervasive Comput 7(2):22–31. doi:10.1109/MPRV.2008.36
Bannach D, Kunze K, Lukowicz P (2006) Distributed modular toolbox for multi-modal context recognition. In: Proceedings of 19th international conference on architecture of computing systems (ARCS 2006), Frankfurt/Main, Germany, pp 99–113. doi:10.1007/11682127
Bao L, Intille SS (2004) Activity recognition from user-annotated acceleration data. In: Proceedings of the 2nd international conference on pervasive computing (Pervasive 2004), Linz, Austria, pp 1–17
Bilmes JA (1998) A gentle tutorial of the EM algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. Tech. Rep. TR-97-021, Computer Science Division, Department of Electrical Engineering and Computer Science, U.C. Berkeley
Brand M, Kettnaker V (2000) Discovery and segmentation of activities in video. IEEE Trans Pattern Anal Mach Intell 22(8):844–851. doi:10.1109/34.868685
Brashear H, Henderson V, Park KH, Hamilton H, Lee S, Starner T (2006) American sign language recognition in game development for deaf children. In: Proceedings of the 8th international ACM SIGACCESS conference on computers and accessibility (Assets 2006), Portland, OR, USA, pp 79–86
Brown RG, Hwang PY (1997) Introduction to random signals and applied Kalman filtering, 3rd edn. Wiley, New York
Cheung KW, So HC, Ma WK, Chan YT (2006) A constrained least squares approach to mobile positioning: algorithms and optimality. EURASIP J Appl Signal Process 2006:1–23
Dey A, Salber D, Abowd G (2001) A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum Comput Interact Spec Issue Context Aware Comput 16(2–4):97–166
Elhay S, Golub GH, Kautsky J (1991) Updating and downdating of orthogonal polynomials with data fitting applications. SIAM J Matrix Anal Appl 12(2):327–353
Gellersen HW, Schmidt A, Beigl M (2002) Multi-sensor context-awareness in mobile devices and smart artifacts. MONET 7(5):341–351
Gupta N, Hauser R (2007) Kalman filtering with equality and inequality state constraints. Tech. rep. Oxford University Computing Laboratory, Oxford, UK
Harter A, Hopper A, Steggles P, Ward A, Webster P (2002) The anatomy of a context-aware application. Wireless Netw 8(2–3):187–197
Hazas M, Kray C, Gellersen H, Agbota H, Kortuem G, Krohn A (2005) A relative positioning system for co-located mobile devices. In: Proceedings of the 3rd international conference on mobile systems, applications, and services (MobiSys 2005), Seattle, pp 177–190. doi:10.1145/1067170.1067190
Jason I, Simon F (2004) Optimized wavelet hand pose estimation for american sign language recognition. In: Proceedings of the congress on evolutionary computation (CEC 2004), Portland, OR, pp 797–802
Junker H, Lukowicz P, Tröster G (2004) Continuous recognition of arm activities with wearable motion sensors. In: Proceedings of the 8th IEEE international symposium on wearable computers (ISWC’04), Arlington, VA
Krause A, Smailagic A, Siewiorek D (2006) Context-aware mobile computing: learning context-dependent personal preferences from a wearable sensor array. IEEE Trans Mobile Comput 5(2):113–127
Kunze K, Wagner F, Kartal E, Kluge EM, Lukowicz P (2009) Does context matter? A quantitative evaluation in a real world maintenance scenario. In: Proceedings of 7th international conference on pervasive computing, Nara, Japan, pp 372–389
Lawo M, Herzog O, Witt H (2007) An industrial case study on wearable computing applications. In: Proceedings of the 9th ACM international conference on human computer interaction with mobile devices and services (MobileHCI 2007), Singapore, pp 448–451. doi:10.1145/1377999.1378052
Lee C, Xu Y (1996) Online, interactive learning of gestures for human/robot interfaces. In: Proceedings of the IEEE international conference on robotics and automation (ICRA 1996), Minneapolis, MN, pp 2982–2987. doi:10.1109/ROBOT.1996.509165
Lee HK, Kim JH (1998) Gesture spotting from continuous hand motion. Pattern Recognit Lett 19(5–6):513–520
Lester J, Choudhury T, Borriello G (2006) A practical approach to recognizing physical activities. In: Proceedings of the 4th international conference on pervasive computing (Pervasive 2006), Dublin, pp 1–16
Li W, Jonsson M (2006) Providing user self-contained location information using mobile phones. In: Advances in pervasive computing, adjunct proceedings of pervasive 2006, pp 33–36
Lin J, Keogh E, Lonardi S, Chiu B (2003) A symbolic representation of time series, with implications for streaming algorithms. In: Proceedings of the 8th ACM SIGMOD workshop on research issues in data mining and knowledge discovery (DMKD 2003), San Diego, CA
Lin J, Wei EKL, Lonardi S (2007) Experiencing SAX: a novel symbolic representation of time series. Data Min Knowl Discov J 15(2):107–144
Lukowicz P, Kirstein T, Tröster G (2004) Wearable systems for health care applications. Methods Inf Med 43(3):232–230
Lukowicz P, Timm-Giel A, Lawo M, Herzog O (2007) WearIT@work: toward real-world industrial wearable computing. IEEE Pervasive Comput 6(4):8–13
Minnen D, Isbell C, Essa I, Starner T (2007) Detecting subdimensional motifs: An efficient algorithm for generalized multivariate pattern discovery. In: Proceedings of the 7th IEEE international conference on data mining (ICDM 2007), Omaha, NE, pp 601–606. doi:10.1109/ICDM.2007.52
Mitra S, Acharya T (2007) Gesture recognition: A survey. IEEE Trans Syst Man Cybern C 37(3):311–324
Müller M, Röder T, Clausen M (2005) Efficient content-based retrieval of motion capture data. In: Proceedings of the 35th international conference on computer graphics and interactive techniques (SIGGRAPH 2005), Los Angeles, CA, pp 677–685. doi:10.1145/1073204.1073247
Naoum-Sawaya J, Slim M, Khawam S, Al-Alaoui MA (2006) Dynamic system design for american sign language recognition. In: Proceedings of the 2nd international symposium on communications, control, and signal processing (ISCCSP 2006), Marakesh, Morocco
Ogris G (2009) Multi-modal on-body sensing of human activities. PhD thesis, University of Passau, Passau
Ogris G, Stiefmeier T, Junker H, Lukowicz P, Tröster G (2005) Using ultrasonic hand tracking to augment motion analysis based recognition of manipulative gestures. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), Osaka, Japan, pp 152–159. doi:10.1109/ISWC.2005.54
Patterson DJ, Fox D, Kautz H, Philipose M (2005) Fine-grained activity recognition by aggregating abstract object usage. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), Osaka, Japan, pp 44–51. doi:10.1109/ISWC.2005.22
Poppe R (2010) A survey on vision-based human action recognition. Image Vis Comput 28(6):976–990
Priyantha NB (2005) The cricket indoor location system. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA
Roetenberg D (2006) Inertial and magnetic sensing of human motion. PhD thesis, University of Twente, Enschede
Roetenberg D, Veltink PH (2005) Camera-marker and inertial sensor fusion for improved motion tracking. In: Proceedings of the European Society of Movement Analysis for Adults and Children (ESMAC 2005), Barcelona, pp 51–52
Schölkopf B, Smola AJ (2002) Learning with kernels: support vector machines, regularization, optimization, and beyond (Adaptive computation and machine learning). MIT Press, Cambridge
Siewiorek D, Smailagic A, Starner T (2008) Application design for wearable computing. Synthesis lectures on mobile and pervasive computing. Morgan and Claypool Publishers
Smith A, Balakrishnan H, Goraczko M, Priyantha N (2004) Tracking moving devices with the Cricket location system. In: Proceedings of the 2nd international conference on mobile systems, applications, and services (MobiSys 2004), Boston, MA, pp 190–202. doi:10.1145/990064.990088
Starner T, Schiele B, Pentland A (1998) Visual contextual awareness in wearable computing. In: Proceedings of the 2nd international symposium on wearable computers (ISWC’98), Pittsburgh, PA, pp 50–57. doi:10.1109/ISWC.1998.729529
Starner T, Weaver J, Pentland A (1998) Real-time American sign language recognition using desk and wearable computer based video. IEEE Trans Pattern Anal Mach Intell 20(12):1371–1375. doi:10.1109/34.735811
Stiefmeier T (2008) Real-time spotting of human activities in industrial environments. PhD thesis, ETH Zürich, Zurich
Stiefmeier T, Ogris G, Junker H, Lukowicz P, Tröster G (2006) Combining motion sensors and ultrasonic hands tracking for continuous activity recognition in a maintenance scenario. In: Proceedings of the 10th IEEE international symposium on wearable computers (ISWC’06), Montreux, pp 97–104. doi:10.1109/ISWC.2006.286350
Stiefmeier T, Roggen D, Ogris G, Lukowicz P, Tröster G (2008) Wearable activity tracking in car manufacturing. IEEE Pervasive Comput Spec Issue Activity Based Comput 7(2):42–50
Stiefmeier T, Roggen D, Tröster G (2007) Fusion of string-matched templates for continuous activity recognition. In: Proceedings of the 11th IEEE international symposium on wearable computers (ISWC’07), Boston, MA, pp 41–44
Stiefmeier T, Roggen D, Tröster G (2007) Gestures are strings: Efficient online gesture spotting and classification using string matching. In: Proceedings of the 2nd international conference on body area networks (BodyNets 2007), Florence, Italy
Ungarala S, Dolence E, Li K (2007) Constrained extended Kalman filter for nonlinear state estimation. In: Proceedings of the 8th international IFAC symposium on dynamics and control of process systems (DYCOPS 2007), Cancún, pp 63–68
Ward A (1998) Sensor-driven computing. PhD thesis, University of Cambridge, Cambridge, UK
Ward A, Jones A, Hopper A (1997) A new location technique for the active office. IEEE Personal Commun 4(5):42–47
Ward JA (2006) Activity monitoring: continuous recognition and performance evaluation. PhD thesis, ETH Zürich, Zurich
Ward JA, Lukowicz P, Tröster G, Starner TE (2006) Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans Pattern Anal Mach Intell 28(10):1553–1567. doi:10.1109/TPAMI.2006.197
Westeyn T, Vadas K, Bian X, Starner T, Abowd G (2005) Recognizing mimicked autistic self-stimulatory behaviors using HMMs. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), pp 164–169
Wijnalda G, Pauws S, Vignoli F, Stuckenschmidt H (2005) A personalized music system for motivation in sport performance. IEEE Pervasive Comput 4(3):26–32
Yamato J, Ohya J, Ishii K (1992) Recognizing human action in time-sequential images using Hidden Markov Models. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR 1992), Champaign, IL, pp 379–385. doi:10.1109/CVPR.1992.223161
Yoon HS, Soh J, Bae YJ, Yang HS (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognit 34(7):1491–1501. doi:10.1016/S0031-3203(00)00096-0
Zinnen A, Laerhoven KV, Schiele B (2007) Toward recognition of short and non-repetitive activities from wearable sensors. In: Proceedings of the European conference on ambient intelligence (AMI 2007), Darmstadt, pp 142–158
Zinnen A, Wojek C, Schiele B (2009) Multi activity recognition based on bodymodel-derived primitives. In: 4th international symposium on location and context awareness (LoCA 2009), Tokyo
Acknowledgments
This work was supported by the European Union under contract EC IP 004216.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ogris, G., Lukowicz, P., Stiefmeier, T. et al. Continuous activity recognition in a maintenance scenario: combining motion sensors and ultrasonic hands tracking. Pattern Anal Applic 15, 87–111 (2012). https://doi.org/10.1007/s10044-011-0216-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10044-011-0216-z