skip to main content
research-article

The Impact of Walking and Resting on Wrist Motion for Automated Detection of Meals

Published: 30 September 2020 Publication History

Abstract

This article considers detecting eating in free-living humans by tracking wrist motion. We are specifically interested in the effect of secondary activities that people conduct while simultaneously eating, such as walking, watching television, or working. These secondary activities cause wrist motions that obfuscate those associated with eating, increasing the difficulty of detecting periods of eating. We collected a large dataset of 4,680 hours of wrist motion from 351 participants during free living. Participants reported secondary activities in 72% of meals. Analysis of wrist motion data revealed that the wrist was resting 12.8% of the time during self-reported meals compared to only 6.8% of the time in a cafeteria dataset, whereas walking motion was found 5.5% of the time during meals in free living compared to 0% in a cafeteria. Augmenting an eating detection classifier to include walking and resting detection improved accuracy from 74% to 77% on our free-living dataset (t[353] = 7.86, p < 0.001). Although eating detection could be improved using more sophisticated machine learning methods or sensor modalities, all approaches would be affected by secondary activities, as they affect the labeling of data itself. Our work suggests that future work should collect detailed ground truth on secondary activities being conducted during eating, as these activities could hold insights into when an eating activity starts or stops in the absence of video-based ground truth.

References

[1]
Rawan Alharbi, Tammy Stump, Nilofar Vafaie, Angela Pfammatter, Bonnie Spring, and Nabil Alshurafa. 2018. I can’t be myself: Effects of wearable cameras on the capture of authentic behavior in the wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 90.
[2]
Nabil Alshurafa, Haik Kalantarian, Mohammad Pourhomayoun, Jason J. Liu, Shruti Sarin, Behnam Shahbazi, and Majid Sarrafzadeh. 2015. Recognition of nutrition intake using time-frequency decomposition in a wearable necklace using a piezoelectric sensor. IEEE Sensors Journal 15, 7 (2015), 3909--3916.
[3]
Oliver Amft, Martin Kusserow, and Gerhard Troster. 2007. Automatic identification of temporal sequences in chewing sounds. In Proceedings of the 2007 IEEE International Conference on Bioinformatics and Biomedicine (BIBM’07). IEEE, Los Alamitos, CA, 194--201.
[4]
Oliver Amft and Gerhard Tröster. 2008. Recognition of dietary activity events using on-body sensors. Artificial Intelligence in Medicine 42, 2 (2008), 121--136.
[5]
Abdelkareem Bedri, Richard Li, Malcolm Haynes, Raj Prateek Kosaraju, Ishaan Grover, Temiloluwa Prioleau, Min Yan Beh, Mayank Goel, Thad Starner, and Gregory Abowd. 2017. EarBit: Using wearable sensors to detect eating episodes in unconstrained environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 37.
[6]
Shengie Bi, Tao Wang, Nicole Tobias, Josephine Nordrum, Shang Wang, George Halvorsen, Sougata Sen, et al. 2018. Auracle: Detecting eating episodes with an ear-mounted sensor. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 92.
[7]
Box Inc.2019. Box: Content Management, Workflow, and Collaboration. Retrieved December 2, 2019 from https://www.box.com/home [Online; accessed 2-December-2019].
[8]
Keum San Chun, Sarnab Bhattacharya, and Edison Thomaz. 2018. Detecting eating episodes by tracking jawbone movements with a non-contact wearable sensor. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 4.
[9]
Guang Deng and L. W. Cahill. 1993. An adaptive Gaussian filter for noise reduction and edge detection. In Proceedings of the 1993 IEEE Conference Record Nuclear Science Symposium and Medical Imaging Conference. IEEE, Los Alamitos, CA, 1615--1619.
[10]
Bo Dong, Subir Biswas, Robert Gernhardt, and Janik Schlemminger. 2013. A mobile food intake monitoring system based on breathing signal analysis. In Proceedings of the 8th International Conference on Body Area Networks. 165--168.
[11]
Yujie Dong, Jenna Scisco, Mike Wilson, Eric Muth, and Adam Hoover. 2014. Detecting periods of eating during free-living by tracking wrist motion. IEEE Journal of Biomedical and Health Informatics 18, 4 (2014), 1253--1260.
[12]
Abul Doulah, Xin Yang, Jason Parton, Janine A. Higgins, Megan A. McCrory, and Edward Sazonov. 2018. The importance of field experiments in testing of sensors for dietary assessment and eating behavior monitoring. In Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC’18). IEEE, Los Alamitos, CA, 5759--5762.
[13]
Muhammad Farooq, Abul Doulah, Jason Parton, Megan A. McCrory, Janine A. Higgins, and Edward Sazonov. 2019. Validation of sensor-based food intake detection by multicamera video observation in an unconstrained environment. Nutrients 11, 3 (2019), 609.
[14]
Juan M. Fontana, Muhammad Farooq, and Edward Sazonov. 2014. Automatic ingestion monitor: A novel wearable device for monitoring of ingestive behavior. IEEE Transactions on Biomedical Engineering 61, 6 (2014), 1772--1779.
[15]
Sebastian O. H. Madgwick, Andrew J. L. Harrison, and Ravi Vaidyanathan. 2011. Estimation of IMU and MARG orientation using a gradient descent algorithm. In Proceedings of the 2011 IEEE International Conference on Rehabilitation Robotics. IEEE, Los Alamitos, CA, 1--7.
[16]
Ryan Mattfeld, Elliot Jesch, and Adam Hoover. 2017. A new dataset for evaluating pedometer performance. In Proceedings of the 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM’17). IEEE, Los Alamitos, CA, 865--869.
[17]
Mark Mirtchouk, Drew Lustig, Alexandra Smith, Ivan Ching, Min Zheng, and Samantha Kleinberg. 2017. Recognizing eating from body-worn sensors: Combining free-living and laboratory data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 85.
[18]
Eric Muth, Ryan Mattfeld, and Adam Hoover. 2017. Bites as a unit of measurement. In Advances in the Assessment of Dietary Intake.CRC Press, Boca Raton, FL, 149--161.
[19]
Eric R. Muth and Adam Hoover. 2017. Designing mobile health technologies for self-monitoring: The bite counter as a case study. In Mobile Health. Springer, 101--120.
[20]
Raul I. Ramos-Garcia, Eric R. Muth, John N. Gowdy, and Adam W. Hoover. 2015. Improving the recognition of eating gestures using intergesture sequential dependencies. IEEE Journal of Biomedical and Health Informatics 19, 3 (2015), 825--831.
[21]
James N. Salley, Adam W. Hoover, Michael L. Wilson, and Eric R. Muth. 2016. Comparison between human and bite-based methods of estimating caloric intake. Journal of the Academy of Nutrition and Dietetics 116, 10 (2016), 1568--1577.
[22]
Edward Sazonov, Stephanie Schuckers, Paulo Lopez-Meyer, Oleksandr Makeyev, Nadezhda Sazonova, Edward L. Melanson, and Michael Neuman. 2008. Non-invasive monitoring of chewing and swallowing for objective quantification of ingestive behavior. Physiological Measurement 29, 5 (2008), 525.
[23]
Giovanni Schiboni and Oliver Amft. 2018. Sparse natural gesture spotting in free living to monitor drinking with wrist-worn inertial sensors. In Proceedings of the 2018 ACM International Symposium on Wearable Computers. ACM, New York, NY, 140--147.
[24]
Surya Sharma and Adam Hoover. 2019. A study on linear acceleration of the wrist during free-living. In Proceedings of the 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM’19). IEEE, Los Alamitos, CA.
[25]
Yiru Shen, James Salley, Eric Muth, and Adam Hoover. 2017. Assessing the accuracy of a wrist motion tracking method for counting bites across demographic and food variables. IEEE Journal of Biomedical and Health Informatics 21, 3 (2017), 599--606.
[26]
Julius Sim and Chris C. Wright. 2005. The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy 85, 3 (2005), 257--268.
[27]
TDK Invensense. 2018. MPU-9150. Retrieved August 31, 2020 from https://www.invensense.com/products/motion-tracking/9-axis/mpu-9150.
[28]
Edison Thomaz, Irfan Essa, and Gregory D. Abowd. 2015. A practical approach for recognizing eating moments with wrist-mounted inertial sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, New York, NY, 1029--1040.
[29]
Edison Thomaz, Aman Parnami, Jonathan Bidwell, Irfan Essa, and Gregory D. Abowd. 2013. Technological approaches for addressing privacy concerns when recognizing eating behaviors with wearable cameras. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, New York, NY, 739--748.
[30]
U.S. Department of Labor. 2016. Bureau of Labor Statistics Data for Activity: Eating and Drinking. Retrieved August 31, 2020 from https://data.bls.gov/timeseries/TUU10101AA01013237.
[31]
Wikipedia Contributors. 2019. Confusion Matrix—Table of Confusion. Retrieved August 6, 2019 from https://en.wikipedia.org/wiki/Confusion_matrix#Table_of_confusion [Online; accessed 6-August-2019].
[32]
Wikipedia Contributors. 2019. Precision and Recall—Imbalanced Data. Retrieved August 6, 2019 from https://en.wikipedia.org/wiki/Precision_and_recall#Imbalanced_Data [Online; accessed 6-August-2019].
[33]
Rui Zhang and Oliver Amft. 2017. Monitoring chewing and eating in free-living using smart eyeglasses. IEEE Journal of Biomedical and Health Informatics 22, 1 (2017), 23--32.
[34]
Shibo Zhang, Rawan Alharbi, Matthew Nicholson, and Nabil Alshurafa. 2017. When generalized eating detection machine learning models fail in the field. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2017 ACM International Symposium on Wearable Computers. ACM, New York, NY, 613--622.

Cited By

View all
  • (2024)A Novel Audio-Visual Information Fusion System for Mental Disorders Detection2024 27th International Conference on Information Fusion (FUSION)10.23919/FUSION59988.2024.10706499(1-7)Online publication date: 8-Jul-2024
  • (2024)Eating Speed Measurement Using Wrist-Worn IMU Sensors Towards Free-Living EnvironmentsIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.342287528:10(5816-5828)Online publication date: Oct-2024
  • (2024)Detecting Eating Episodes From Wrist Motion Using Daily Pattern AnalysisIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2023.334107728:2(1054-1065)Online publication date: Feb-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computing for Healthcare
ACM Transactions on Computing for Healthcare  Volume 1, Issue 4
Special Issue on Wearable Technologies for Smart Health: Part 1
October 2020
184 pages
EISSN:2637-8051
DOI:10.1145/3427421
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 September 2020
Accepted: 01 April 2020
Revised: 01 February 2020
Received: 01 August 2019
Published in HEALTH Volume 1, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Automated dietary monitoring
  2. eating detecting
  3. gesture recognition
  4. m-health
  5. obesity
  6. resting detection
  7. walking detection
  8. wearables

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)36
  • Downloads (Last 6 weeks)3
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)A Novel Audio-Visual Information Fusion System for Mental Disorders Detection2024 27th International Conference on Information Fusion (FUSION)10.23919/FUSION59988.2024.10706499(1-7)Online publication date: 8-Jul-2024
  • (2024)Eating Speed Measurement Using Wrist-Worn IMU Sensors Towards Free-Living EnvironmentsIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.342287528:10(5816-5828)Online publication date: Oct-2024
  • (2024)Detecting Eating Episodes From Wrist Motion Using Daily Pattern AnalysisIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2023.334107728:2(1054-1065)Online publication date: Feb-2024
  • (2023)Rationale and design of the SenseWhy project: A passive sensing and ecological momentary assessment study on characteristics of overeating episodesDIGITAL HEALTH10.1177/205520762311583149Online publication date: 27-Apr-2023
  • (2022)Top-Down Detection of Eating Episodes by Analyzing Large Windows of Wrist Motion Using a Convolutional Neural NetworkBioengineering10.3390/bioengineering90200709:2(70)Online publication date: 11-Feb-2022
  • (2022)EatingTrakProceedings of the ACM on Human-Computer Interaction10.1145/35467496:MHCI(1-22)Online publication date: 20-Sep-2022
  • (2022)Combining passive eating monitoring and ecological momentary assessment to characterize dietary lapses from a lifestyle modification interventionAppetite10.1016/j.appet.2022.106090175(106090)Online publication date: Aug-2022
  • (2021)Optimizing a Just-in-Time Adaptive Intervention to Improve Dietary Adherence in Behavioral Obesity Treatment: Protocol for a Microrandomized TrialJMIR Research Protocols10.2196/3356810:12(e33568)Online publication date: 6-Dec-2021
  • (2021)Combining ecological momentary assessment, wrist-based eating detection, and dietary assessment to characterize dietary lapse: A multi-method study protocolDIGITAL HEALTH10.1177/20552076209882127Online publication date: 2-Feb-2021
  • (2020)The Behavioral Intervention with Technology for E-Weight Loss Study (BITES): Incorporating Energy Balance Models and the Bite Counter into an Online Behavioral Weight Loss ProgramJournal of Technology in Behavioral Science10.1007/s41347-020-00181-46:2(406-418)Online publication date: 27-Nov-2020

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media