research-article

FMT: A Wearable Camera-Based Object Tracking Memory Aid for Older Adults

Authors:

Franklin Mingzhe Li,

Khai N. TruongAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 3, Issue 3

Article No.: 95, Pages 1 - 25

https://doi.org/10.1145/3351253

Published: 09 September 2019 Publication History

Abstract

Older adults sometimes forget about whether or not they have completed routine actions and the states of objects that they have interacted with (e.g., the kitchen stove is on or off). In this work, we explore whether video clips captured from a body-worn camera every time objects of interest are found within its field of view can help older adults determine if they have completed certain actions with these objects and what their states are. We designed FMT ("Fiducial Marker Tracker")---a real-time capture and access application that opportunistically captures video clips of objects the user interacts with. To do this, the user places fiducial markers close to objects which would be captured when the marker enters the user's body-worn camera's field of view. We examine and discuss what objects this system would be best suited to track, and the usefulness and usability of this approach. FMT successfully captured direct interactions with an object at an average rate of 75.6% across all participants (SD = 9.9%). Our results also reveal how, what, and why users would use such a system for help.

References

[1]

Anita L. Allen. 2008. Dredging up the past: Lifelogging, Memory, and Surveillance. Univ. Chicago Law Rev. 75, 1 (2008), 47--74.

[2]

H Aoki and S. Matsushita. 2002. Balloon Tag: (in)visible marker which tells who's who. In ieeexplore.ieee.org, 181--182.

Digital Library

[3]

Ido Arev, Hyun Soo Park, Yaser Sheikh, Jessica Hodgins, and Ariel Shamir. 2014. Automatic editing of footage from multiple social cameras. ACM Trans. Graph. 33, 4 (July 2014), 1--11.

Digital Library

[4]

Lauren G. Block and Vicki G. Morwitz. 1999. Shopping lists as an external memory aid for grocery shopping: Influences on list writing and list fulfillment. J. Consum. Psychol. 8, 4 (January 1999), 343--375.

[5]

Robin Boldt. 2013. The Last-Seen Image: an Image-Based Approach for Finding Lost Objects Using a Head-Mounted Display. 91 (2013).

[6]

W F Brewer. 1988. Qualitative analysis of the recalls of randomly sampled autobiographical events. In Practical aspects of memory: Current research and issues, Vol. 1: Memory in everyday life. 263--268.

[7]

William F Brewer. 1986. What is autobiographical memory? Autobiographical Mem. c (1986), 25--49.

[8]

Georgina Browne, Emma Berry, Narinder Kapur, Steve Hodges, Gavin Smyth, Peter Watson, and Ken Wood. 2011. SenseCam improves memory for recent events and quality of life in a patient with memory retrieval difficulties. Memory 19, 7 (2011), 713--722.

[9]

John C. Cavanaugh, Jayne G. Grady, and Marion Perlmutter. 1983. Forgetting and Use of Memory Aids in 20 to 70 Year Olds Everyday Life. Int. J. Aging Hum. Dev. 17, 2 (September 1983), 113--122.

[10]

Centers for Disease Control and Prevention. 2012. Identifying Vulnerable Older Adults and Legal Options for Increasing Their Protection During All-Hazards Emergencies: A Cross-Sector Guide for States and Communities. U.S. Dep. Heal. Hum. Serv. (2012). Retrieved from http://www.cdc.gov/aging/emergency/pdf/guide.pdf

[11]

Tim Dalgleish and Sally G Cox. 2002. Memory and emotional disorder. Handb. Mem. Disord. (2002), 437--457.

[12]

Daniel P.W. Ellis and Keansub Lee. 2004. Minimal-impact audio-based personal archives. In Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences - CARPE'04, 39.

Digital Library

[13]

G R Hayes. 2004. The Personal Audio Loop: Designing a Ubiquitous AudioBased Memory Aid,? BT - Proc. 6th Int?l Conf. Human Computer Interaction with Mobile Devices and Services. (2004), 168--179.

[14]

Gillian R. Hayes and Khai N. Truong. 2009. Selective Archiving: A Model for Privacy Sensitive Capture and Access Technologies. In Protecting Privacy in Video Surveillance. Springer London, London, 165--184.

Digital Library

[15]

R.A. Hertzog, C. & Dixon. 1994. Metacognitive development in adulthood and old age. In Metacognition: Knowing about Knowing, J. Metcalfe & A. Shimamura (ed.). 227--251.

[16]

Keita Higuchi, Ryo Yonetani, Yoichi Sato, and Keita Higuch. 2017. EgoScanning: Quickly Scanning First-Person Videos with Egocentric Elastic Timelines. (2017).

Digital Library

[17]

Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. 2006. SenseCam: A Retrospective Memory Aid. (2006), 177--193.

Digital Library

[18]

Canadian Institute for Health Information. 2007. Health Care in Canada, 2011. React. Wkly. NA;, 1157 (2007), 3.

[19]

T Jebara, B Schiele, N Oliver, and A Pentland. 1998. DyPERS: Dynamic personal enhanced reality system. Proc. 1998 Image Underst. Work. (1998). Retrieved September 16, 2018 from http://www.media.mit.edu/vismod/demos/dypers

[20]

Shiqi Jiang, Pengfei Zhou, M O Li, Zhenjiang Li, and Mo Li. 2019. Memento: An Emotion-driven Lifelogging System with Wearables. ACM Trans. Sen. Netw 15, 8 (2019).

Digital Library

[21]

Walter S. Lasecki, Young Chol Song, Henry Kautz, and Jeffrey P. Bigham. 2013. Real-time crowd labeling for deployable activity recognition. In Proceedings of the 2013 conference on Computer supported cooperative work - CSCW '13, 1203.

Digital Library

[22]

Huy Viet Le, Sarah Clinch, Corina Sas, Tilman Dingler, Niels Henze, and Nigel Davies. 2016. Impact of Video Summary Viewing on Episodic Memory Recall: Design Guidelines for Video Summarizations. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4793--4805.

Digital Library

[23]

Matthew L. Lee and Anind K. Dey. 2007. Providing good memory cues for people with episodic memory impairment. In Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility - Assets '07, 131.

Digital Library

[24]

Matthew L. Lee and Anind K. Dey. 2008. Lifelogging memory appliance for people with episodic memory impairment. In Proceedings of the 10th international conference on Ubiquitous computing - UbiComp '08, 44.

Digital Library

[25]

YJ Lee, J Ghosh, K Grauman - Computer Vision and Pattern, and Undefined 2012. Discovering Important People and Objects for Egocentric Video Summarization. ieeexplore.ieee.org. https://ieeexplore.ieee.org/abstract/document/6247820/

[26]

Yong Jae Lee and Kristen Grauman. 2015. Predicting Important Objects for Egocentric Video Summarization. Int. J. Comput. Vis. 114, 1 (August 2015), 38--55.

Digital Library

[27]

Catherine Loveday and Martin A Conway. 2011. Using SenseCam with an amnesic patient: Accessing inaccessible everyday memories. Memory 19, 7 (2011), 697--704.

[28]

Zheng Lu and Kristen Grauman. 2013. Story-driven summarization for egocentric video. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2714--2721.

Digital Library

[29]

Elizabeth A Maylor. 1993. Aging and forgetting in prospective and retrospective memory tasks. {erratum appears in Psychol Aging 1995 Jun;10(2):190.}. Psychol. Aging 8, 3 (1993), 420--428. https://journals-scholarsportal-info.myaccess.library.utoronto.ca/pdf/08827974/v08i0003/420_aafiparmt.xml

[30]

Elizabeth D Mynatt and Wendy A Rogers. 2001. Developing technology to support the functional independence of older adults. Ageing Int. 27, 1 (2001), 24--41.

[31]

Evangelos Niforatos, Caterina Cinel, Mack Cortis Cathleen, Marc Langheinrich, and Geoff Ward. 2017. Can Less be More? Contrasting Limited, Unlimited, and Automatic Picture Capture for Augmenting Memory Recall. IMWUT 1, 2 (2017).

Digital Library

[32]

Beth A.D. Nolan, R. Mark Mathews, and Melanie Harrison. 2001. Using external memory aids to increase room finding by older adults with dementia. Am. J. Alzheimers. Dis. Other Demen. 16, 4 (July 2001), 251--254.

[33]

Shwetak N. Patel and Gregory D. Abowd. 2010. The ContextCam: Automated Point of Capture Video Annotation. In Springer. 301--318.

[34]

Shwetak N. Patel, Jun Rekimoto, and Gregory D. Abowd. 2006. iCam: Precise at-a-Distance Interaction in the Physical Environment. In Springer. 272--287.

Digital Library

[35]

Donald J Patterson, Dieter Fox, Henry Kautz, and Matthai Philipose. Fine-Grained Activity Recognition by Aggregating Abstract Object Usage.

Digital Library

[36]

Jun Rekimoto and Katashi Nagao. The World through the Computer: Computer Augmented Interaction with Real World Environments. Retrieved May 14, 2019 from http://www.csl.sony.co.jp/person/frekimoto,nagaog.html

[37]

Etsuko Sugawara and Hiroshi Nikaido. 2014. World Report on Ageing and Health.

[38]

Desney Tan, Emma Berry, Mary Czerwinski, Gordon Bell, Jim Gemmell, Steve Hodges, Narinder Kapur, Brian Meyers, Nuria Oliver, George Robertson, and Ken Wood. 2007. Save everything: supporting human memory with a personal digital lifetime store. In Personal information management. 90--107.

[39]

Khai N Truong, Gregory D Abowd, and Jason A Brotherton. 2001. Who, What, When, Where, How: Design Issues of Capture & Access Applications. 209--224.

[40]

Lois M. Verbrugge. 1980. Health diaries. Med. Care 18, 1 (1980), 73--95.

[41]

Emma Woodberry, Georgina Browne, Steve Hodges, Peter Watson, Narinder Kapur, and Ken Woodberry. 2015. The use of a wearable camera improves autobiographical memory in patients with Alzheimer's disease. Memory 23, 3 (2015), 340--349.

[42]

Jia Xu, Lopamudra Mukherjee, Yin Li, Jamieson Warner, James M Rehg, and Vikas Singh. Gaze-enabled Egocentric Video Summarization via Constrained Submodular Maximization. cv-foundation.org.

[43]

MT Yasami, T Dua, M Harper, and Saxena S. 2013. Mental Health of Older Adults, Addressing a Growing Concern. In Mental health and older people, 4--9.

Cited By

Qin YSu JQin HTian Y(2024)Exploring the Role of Video Playback Visual Cues in Object Retrieval TasksSensors10.3390/s2410314724:10(3147)Online publication date: 15-May-2024
https://doi.org/10.3390/s24103147
Zhu DAl Mahmud ALiu W(2024)Digital Storytelling Intervention for Enhancing the Social Participation of People With Mild Cognitive Impairment: Co-Design and Usability StudyJMIR Aging10.2196/541387(e54138)Online publication date: 17-Jan-2024
https://doi.org/10.2196/54138
Mamish JAlharbi RSen SHolla SKamath PSangar YAlshurafa NHester J(2024)NIR-sighted: A Programmable Streaming Architecture for Low-Energy Human-Centric Vision ApplicationsACM Transactions on Embedded Computing Systems10.1145/367207623:6(1-26)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3672076
Show More Cited By

Index Terms

FMT: A Wearable Camera-Based Object Tracking Memory Aid for Older Adults
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing

Recommendations

GO-Finder: A Registration-Free Wearable System for Assisting Users in Finding Lost Objects via Hand-Held Object Discovery
IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

People spend an enormous amount of time and effort looking for lost objects. To help remind people of the location of lost objects, various computational systems that provide information on their locations have been developed. However, prior systems for ...
GO-Finder: A Registration-free Wearable System for Assisting Users in Finding Lost Hand-held Objects
People spend an enormous amount of time and effort looking for lost objects. To help remind people of the location of lost objects, various computational systems that provide information on their locations have been developed. However, prior systems for ...
Agile life: addressing knowledge and social motivations for active aging
OzCHI '12: Proceedings of the 24th Australian Computer-Human Interaction Conference

Despite the inter-relationship between physical, cognitive and social factors for older people, the frequency of physical activity typically decreases with age. In this paper, we focus on two specific issues related to physical activity and older people ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 3, Issue 3

September 2019

1415 pages

EISSN:2474-9567

DOI:10.1145/3361560

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2019

Published in IMWUT Volume 3, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Natural Sciences and Engineering Research Council of Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
785
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)8

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Qin YSu JQin HTian Y(2024)Exploring the Role of Video Playback Visual Cues in Object Retrieval TasksSensors10.3390/s2410314724:10(3147)Online publication date: 15-May-2024
https://doi.org/10.3390/s24103147
Zhu DAl Mahmud ALiu W(2024)Digital Storytelling Intervention for Enhancing the Social Participation of People With Mild Cognitive Impairment: Co-Design and Usability StudyJMIR Aging10.2196/541387(e54138)Online publication date: 17-Jan-2024
https://doi.org/10.2196/54138
Mamish JAlharbi RSen SHolla SKamath PSangar YAlshurafa NHester J(2024)NIR-sighted: A Programmable Streaming Architecture for Low-Energy Human-Centric Vision ApplicationsACM Transactions on Embedded Computing Systems10.1145/367207623:6(1-26)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3672076
Wang ZGuo BWang QZhang DYu Z(2023)MobiEye: An Efficient Shopping-Assistance System for the Visually Impaired With Mobile Phone SensingIEEE Transactions on Human-Machine Systems10.1109/THMS.2023.330556653:5(865-874)Online publication date: Oct-2023
https://doi.org/10.1109/THMS.2023.3305566
Wang XTalavera EKarastoyanova DAzzopardi G(2023)Fall Detection With a Nonintrusive and First-Person Vision ApproachIEEE Sensors Journal10.1109/JSEN.2023.331482823:22(28304-28317)Online publication date: 15-Nov-2023
https://doi.org/10.1109/JSEN.2023.3314828
Kang CYeom IAshtari AWoo WNoh (2023)ARbility: re-inviting older wheelchair users to in-store shopping via wearable augmented realityVirtual Reality10.1007/s10055-023-00769-027:3(1919-1936)Online publication date: 22-Mar-2023
https://dl.acm.org/doi/10.1007/s10055-023-00769-0
Yagi TNishiyasu TKawasaki KMatsuki MSato Y(2022)GO-Finder: A Registration-free Wearable System for Assisting Users in Finding Lost Hand-held ObjectsACM Transactions on Interactive Intelligent Systems10.1145/351926812:4(1-29)Online publication date: 4-Nov-2022
https://dl.acm.org/doi/10.1145/3519268
Harrington CMartin-Hammond ABray K(2022)Examining Identity as a Variable of Health Technology Research for Older Adults: A Systematic ReviewProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517621(1-24)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517621
Liu MKittur AMyers B(2022)Crystalline: Lowering the Cost for Developers to Collect and Organize Information for Decision MakingProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501968(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501968
Mannheim IWouters EKöttl Hvan Boekel LBrankaert Rvan Zaalen Y(2022)Ageism in the Discourse and Practice of Designing Digital Technology for Older Persons: A Scoping ReviewThe Gerontologist10.1093/geront/gnac14463:7(1188-1200)Online publication date: 21-Sep-2022
https://doi.org/10.1093/geront/gnac144
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents