research-article

Let's (not) stick together: pairwise similarity biases cross-validation in activity recognition

Authors:

Nils Y. Hammerla,

Thomas PlötzAuthors Info & Claims

UbiComp '15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing

Pages 1041 - 1051

https://doi.org/10.1145/2750858.2807551

Published: 07 September 2015 Publication History

Abstract

The ability to generalise towards either new users or unforeseen behaviours is a key requirement for activity recognition systems in ubiquitous computing. Differences in recognition performance for the two application cases can be significant, and user-dependent performance is typically assumed to be an upper bound on performance. We demonstrate that this assumption does not hold for the widely used cross-validation evaluation scheme that is typically employed both during system bootstrapping and for reporting results. We describe how the characteristics of segmented time-series data render random cross-validation a poor fit, as adjacent segments are not statistically independent. We develop an alternative approach -- meta-segmented cross validation -- that explicitly circumvents this issue and evaluate it on two data-sets. Results indicate a significant drop in performance across a variety of feature extraction and classification methods if this bias is removed, and that prolonged, repetitive activities are particularly affected.

References

[1]

Alshurafa, N., Xu, W., Liu, J. J., Huang, M.-C., Mortazavi, B., Roberts, C. K., and Sarrafzadeh, M. Designing a robust activity recognition framework for health and exergaming using wearable sensors. Biomedical and Health Informatics, IEEE Journal of 18, 5 (2014), 1636--1646.

[2]

Anjum, A., and Ilyas, M. U. Activity recognition using smartphone sensors. In Consumer Communications and Networking Conference (CCNC), 2013 IEEE, IEEE (2013), 914--919.

[3]

Berlin, E., and Van Laerhoven, K. Detecting leisure activities with dense motif discovery. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, ACM (2012), 250--259.

Digital Library

[4]

Bishop, C. M., et al. Pattern recognition and machine learning, vol. 4. springer New York, 2006.

Digital Library

[5]

Blum, A., Kalai, A., and Langford, J. Beating the hold-out: Bounds for k-fold and progressive cross-validation. In Proceedings of the twelfth annual conference on Computational learning theory, ACM (1999), 203--208.

Digital Library

[6]

Bogomolov, A., Lepri, B., Ferron, M., Pianesi, F., and Pentland, A. S. Pervasive stress recognition for sustainable living. In Pervasive Computing and Communications Workshops (PERCOM Workshops), 2014 IEEE International Conference on, IEEE (2014), 345--350.

[7]

Bulling, A., Blanke, U., and Schiele, B. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys (CSUR) 46, 3 (2014), 33.

Digital Library

[8]

Chang, C.-C., and Lin, C.-J. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2, 3 (2011), 27.

Digital Library

[9]

Figo, D., Diniz, P. C., Ferreira, D. R., and Cardoso, J. M. Preprocessing techniques for context recognition from accelerometer data. Personal and Ubiquitous Computing 14, 7 (2010), 645--662.

Digital Library

[10]

Gjoreski, H., Kaluža, B., Gams, M., Milić, R., and Luštrek, M. Ensembles of multiple sensors for human energy expenditure estimation. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, ACM (2013), 359--362.

Digital Library

[11]

Gu, W., Yang, Z., Shangguan, L., Sun, W., Jin, K., and Liu, Y. Intelligent sleep stage mining service with smartphones. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM (2014), 649--660.

Digital Library

[12]

Gupta, P., and Dallas, T. Feature selection and activity recognition system using a single tri-axial accelerometer.

[13]

Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I. H. The weka data mining software: an update. ACM SIGKDD explorations newsletter 11, 1 (2009), 10--18.

Digital Library

[14]

Hammerla, N. Y., Fisher, J. M., Andras, P., Rochester, L., Walker, R., and Plötz, T. Pd disease state assessment in naturalistic environments using deep learning.

[15]

Hammerla, N. Y., Kirkham, R., Andras, P., and Ploetz, T. On preserving statistical characteristics of accelerometry data using their empirical cumulative distribution. In Proceedings of the 2013 International Symposium on Wearable Computers, ACM (2013), 65--68.

Digital Library

[16]

Hirano, T., and Maekawa, T. A hybrid unsupervised/supervised model for group activity recognition. In Proceedings of the 2013 International Symposium on Wearable Computers, ACM (2013), 21--24.

Digital Library

[17]

Hung, H., Englebienne, G., and Kools, J. Classifying social actions with a single accelerometer. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, ACM (2013), 207--210.

Digital Library

[18]

Keogh, E., Chu, S., Hart, D., and Pazzani, M. Segmenting time series: A survey and novel approach. Data mining in time series databases 57 (2004), 1--22.

[19]

Kim, J.-H. Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Computational Statistics & Data Analysis 53, 11 (2009), 3735--3745.

Digital Library

[20]

Kose, M., Incel, O. D., and Ersoy, C. Online human activity recognition on smart phones. In Workshop on Mobile Sensing: From Smartphones and Wearables to Big Data (2012), 11--15.

[21]

Ladha, C., Hammerla, N., Hughes, E., Olivier, P., and Plötz, T. Dog's life: wearable activity recognition for dogs. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, ACM (2013), 415--418.

Digital Library

[22]

Ladha, C., Hammerla, N. Y., Olivier, P., and Plötz, T. Climbax: Skill assessment for climbing enthusiasts. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, ACM (2013), 235--244.

Digital Library

[23]

Lane, N. D., Pengyu, L., Zhou, L., and Zhao, F. Connecting personal-scale sensing and networked community behavior to infer human activities. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM (2014), 595--606.

Digital Library

[24]

Li, C.-Y., Chen, Y.-C., Chen, W.-J., Huang, P., and Chu, H.-h. Sensor-embedded teeth for oral activity recognition. In Proceedings of the 2013 International Symposium on Wearable Computers, ACM (2013), 41--44.

Digital Library

[25]

Park, J.-g., Patel, A., Curtis, D., Teller, S., and Ledlie, J. Online pose classification and walking speed estimation using handheld devices. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, ACM (2012), 113--122.

Digital Library

[26]

Peterek, T., Penhaker, M., Gajdoš, P., and Dohnálek, P. Comparison of classification algorithms for physical activity recognition. In Innovations in Bio-inspired Computing and Applications. Springer, 2014, 123--131.

[27]

Plötz, T., Hammerla, N. Y., Rozga, A., Reavis, A., Call, N., and Abowd, G. D. Automatic assessment of problem behavior in individuals with developmental disabilities. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, ACM (2012), 391--400.

Digital Library

[28]

Plötz, T., Moynihan, P., Pham, C., and Olivier, P. Activity recognition and healthier food preparation. In Activity Recognition in Pervasive Intelligent Environments. Springer, 2011, 313--329.

[29]

Reiss, A., Hendeby, G., and Stricker, D. Confidence-based multiclass adaboost for physical activity monitoring. In Proceedings of the 2013 International Symposium on Wearable Computers, ACM (2013), 13--20.

Digital Library

[30]

Reiss, A., and Stricker, D. Introducing a new benchmarked dataset for activity monitoring. In Wearable Computers (ISWC), 2012 16th International Symposium on, IEEE (2012), 108--109.

Digital Library

[31]

Reiss, A., and Stricker, D. Personalized mobile physical activity recognition. In Proceedings of the 2013 International Symposium on Wearable Computers, ACM (2013), 25--28.

Digital Library

[32]

Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. Collecting complex activity datasets in highly rich networked sensor environments. In Networked Sensing Systems (INSS), 2010 Seventh International Conference on, IEEE (2010), 233--240.

[33]

Shoaib, M., Scholten, H., and Havinga, P. J. Towards physical activity recognition using smartphone sensors. In Ubiquitous Intelligence and Computing, 2013 IEEE 10th International Conference on and 10th International Conference on Autonomic and Trusted Computing (UIC/ATC), IEEE (2013), 80--87.

Digital Library

[34]

Velloso, E., Bulling, A., Gellersen, H., Ugulino, W., and Fuks, H. Qualitative activity recognition of weight lifting exercises. In Proceedings of the 4th Augmented Human International Conference, ACM (2013), 116--123.

Digital Library

[35]

Ward, J. A., Lukowicz, P., and Gellersen, H. W. Performance metrics for activity recognition. ACM Transactions on Intelligent Systems and Technology (TIST) 2, 1 (2011), 6.

Digital Library

[36]

Wu, W., Dasgupta, S., Ramirez, E. E., Peterson, C., and Norman, G. J. Classification accuracies of physical activities using smartphone motion sensors. Journal of medical Internet research 14, 5 (2012), e130.

Cited By

Kirsch KStrutzke SKlitzing LPilger FThöne-Reineke CHoffmann G(2025)Validation of a Time-Distributed residual LSTM–CNN and BiLSTM for equine behavior recognition using collar-worn sensorsComputers and Electronics in Agriculture10.1016/j.compag.2025.109999231(109999)Online publication date: Apr-2025
https://doi.org/10.1016/j.compag.2025.109999
P SPlötz T(2024)Using Graphs to Perform Effective Sensor-Based Human Activity Recognition in Smart HomesSensors10.3390/s2412394424:12(3944)Online publication date: 18-Jun-2024
https://doi.org/10.3390/s24123944
Zeng FGuo MTan LGuo FLiu X(2024)Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity RecognitionSensors10.3390/s2403075824:3(758)Online publication date: 24-Jan-2024
https://doi.org/10.3390/s24030758
Show More Cited By

Index Terms

Let's (not) stick together: pairwise similarity biases cross-validation in activity recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object recognition
  2. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis

Recommendations

Let the objects tell what you are doing
UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct

Recognition of activities of daily living (ADLs) performed in smart homes proved to be very effective when the interaction of the inhabitant with household items is considered. Analyzing how objects are manipulated can be particularly useful, in ...
Generic performance measure for multiclass-classifiers

Generic, compact and meaningful performance measure for arbitrary classifiers.Includes confidence indicators in interval [0; 1].Robust towards class imbalance, sensitive towards class separation.Demo implementation available. The evaluation of ...
Confidence-based multiclass AdaBoost for physical activity monitoring
ISWC '13: Proceedings of the 2013 International Symposium on Wearable Computers

Physical activity monitoring has recently become an important topic in wearable computing, motivated by e.g. healthcare applications. However, new benchmark results show that the difficulty of the complex classification problems exceeds the potential of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp '15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 2015

1302 pages

ISBN:9781450335744

DOI:10.1145/2750858

General Chairs:
Kenji Mase
Nagoya University, JP
,
Marc Langheinrich
Università della Svizzera italiana (USI), CH
,
Daniel Gatica-Perez
IDIAP-EPFL, CH
,
Program Chairs:
Hans Gellersen
Lancaster University, UK
,
Tanzeem Choudhury
Cornell University
,
Koji Yatani
Univerisity of Tokyo, JP

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Yahoo! Japan: Yahoo! Japan
SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing
FX Palo Alto Laboratory, Inc.: FX Palo Alto Laboratory, Inc.
Google Inc.
ACM: Association for Computing Machinery
Rakuten Institute of Technology: Rakuten Institute of Technology
Microsoft: Microsoft
Bell Labs: Bell Laboratories
SIGCHI: ACM Special Interest Group on Computer-Human Interaction
KDDI
Panasonic: Panasonic Corporation
NTT DoCoMo
Telefónica: Telefónica
ISTC-PC: Intel Science and Technology Center for Pervasive Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 September 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council

Conference

UbiComp '15

Sponsor:

Yahoo! Japan
SIGMOBILE
FX Palo Alto Laboratory, Inc.
ACM
Rakuten Institute of Technology
Microsoft
Bell Labs
SIGCHI
Panasonic
Telefónica
ISTC-PC

UbiComp '15: The 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 7 - 11, 2015

Osaka, Japan

Acceptance Rates

UbiComp '15 Paper Acceptance Rate 101 of 394 submissions, 26%;

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

68
Total Citations
View Citations
1,215
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kirsch KStrutzke SKlitzing LPilger FThöne-Reineke CHoffmann G(2025)Validation of a Time-Distributed residual LSTM–CNN and BiLSTM for equine behavior recognition using collar-worn sensorsComputers and Electronics in Agriculture10.1016/j.compag.2025.109999231(109999)Online publication date: Apr-2025
https://doi.org/10.1016/j.compag.2025.109999
P SPlötz T(2024)Using Graphs to Perform Effective Sensor-Based Human Activity Recognition in Smart HomesSensors10.3390/s2412394424:12(3944)Online publication date: 18-Jun-2024
https://doi.org/10.3390/s24123944
Zeng FGuo MTan LGuo FLiu X(2024)Wearable Sensor-Based Residual Multifeature Fusion Shrinkage Networks for Human Activity RecognitionSensors10.3390/s2403075824:3(758)Online publication date: 24-Jan-2024
https://doi.org/10.3390/s24030758
Zhao DYang YCao CLiu B(2024)Prediction Models and Feature Importance Analysis for Service State of Tunnel Sections Based on Machine LearningApplied Sciences10.3390/app1420916714:20(9167)Online publication date: 10-Oct-2024
https://doi.org/10.3390/app14209167
Rehman SAli AKhan AOkpala C(2024)Human Activity Recognition: A Comparative Study of Validation Methods and Impact of Feature Extraction in Wearable SensorsAlgorithms10.3390/a1712055617:12(556)Online publication date: 5-Dec-2024
https://doi.org/10.3390/a17120556
Leng ZJung MHwang SOh SZhang LPlötz TKim KKostakos VKay JHoang T(2024)Emotion Recognition on the Go: Utilizing Wearable IMUs for Personalized Emotion RecognitionCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678452(537-544)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678452
Hiremath SPlötz TKostakos VKay JHoang T(2024)Game of LLMs: Discovering Structural Constructs in Activities using Large Language ModelsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678444(487-492)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678444
Kapoor SCantrell EPeng KPham TBail CGundersen OHofman JHullman JLones MMalik MNanayakkara PPoldrack RRaji IRoberts MSalganik MSerra-Garcia MStewart BVandewiele GNarayanan A(2024)REFORMS: Consensus-based Recommendations for Machine-learning-based ScienceScience Advances10.1126/sciadv.adk345210:18Online publication date: 3-May-2024
https://doi.org/10.1126/sciadv.adk3452
Chen LLong HXu JWu BZhou HTang XPeng L(2024)Deep Citywide Multisource Data Fusion-Based Air Quality EstimationIEEE Transactions on Cybernetics10.1109/TCYB.2023.324561854:1(111-122)Online publication date: Jan-2024
https://doi.org/10.1109/TCYB.2023.3245618
Tello ADegeler VLazovik A(2024)Too Good To Be True: accuracy overestimation in (re)current practices for Human Activity Recognition2024 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)10.1109/PerComWorkshops59983.2024.10503465(511-517)Online publication date: 11-Mar-2024
https://doi.org/10.1109/PerComWorkshops59983.2024.10503465
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten