research-article

VIMES: A Wearable Memory Assistance System for Automatic Information Retrieval

Authors:

Carlos Bermejo,

Shayan Mirjafari,

Pan HuiAuthors Info & Claims

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Pages 3191 - 3200

https://doi.org/10.1145/3394171.3413663

Published: 12 October 2020 Publication History

Abstract

The advancement of artificial intelligence and wearable computing triggers the radical innovation of cognitive applications. In this work, we propose VIMES, an augmented reality-based memory assistance system that helps recall declarative memory, such as whom the user meets and what they chat. Through a collaborative method with 20 participants, we design VIMES, a system that runs on smartglasses, takes the first-person audio and video as input, and extracts personal profiles and event information to display on the embedded display or a smartphone. We perform an extensive evaluation with 50 participants to show the effectiveness of VIMES for memory recall. VIMES outperforms (90% memory accuracy) other traditional methods such as self-recall (34%) while offering the best memory experience (Vividness, Coherence, and Visual Perspective all score over 4/5). The user study results show that most participants find VIMES useful (3.75/5) and easy to use (3.46/5).

Supplementary Material

MP4 File (3394171.3413663.mp4)

In this video, we present VIMES, a wearable memory assistant for memory retrieval. We present the main findings of a collaborative design study that has been used in the process of building VIMES. Moreover, we discuss our design's main contributions to help future designers of wearable assistant systems, social acceptance, privacy, and intended adoption.

Download
18.96 MB

References

[1]

Mousa Ahmadi, Cristian Borcea, and Quentin Jones. 2019. Collaborative lifelogging through the integration of machine and human computation. In Proceedings of the 24th International Conference on Intelligent User Interfaces: Companion. 23--24.

Digital Library

[2]

Mélissa C Allé, Liliann Manning, Jevita Potheegadoo, Romain Coutelle, Jean-Marie Danion, and Fabrice Berna. 2017. Wearable cameras are useful tools to investigate and remediate autobiographical memory impairment: A systematic PRISMA review. Neuropsychology review, Vol. 27, 1 (2017), 81--99.

[3]

Marc B"achlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jeffrey M Hausdorff, Nir Giladi, and Gerhard Tröster. 2010. Wearable assistant for Parkinson's disease patients with the freezing of gait symptom. Information Technology in Biomedicine, IEEE Transactions on, Vol. 14, 2 (2010), 436--446.

[4]

Michelle S Bourgeois. 1993. Effects of memory aids on the dyadic conversations of individuals with dementia. Journal of Applied Behavior Analysis, Vol. 26, 1 (1993), 77--87.

[5]

Tristan Braud, Farshid Hassani Bijarbooneh, Dimitris Chatzopoulos, and Pan Hui. 2017. Future networking challenges: The case of mobile augmented reality. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS). IEEE, 1796--1807.

[6]

Georgina Browne, Emma Berry, Narinder Kapur, Steve Hodges, Gavin Smyth, Peter Watson, and Ken Wood. 2011. SenseCam improves memory for recent events and quality of life in a patient with memory retrieval difficulties. Memory, Vol. 19, 7 (2011), 713--722.

[7]

Laurindo de Sousa Britto Neto, Vanessa Regina Margareth Lima Maike, Fernando Luiz Koch, Maria Cecília Calani Baranauskas, Anderson de Rezende Rocha, and Siome Klein Goldenstein. 2015. A Wearable Face Recognition System Built into a Smartwatch and the Blind and Low Vision Users.Springer International Publishing, Cham, 515--528. https://doi.org/10.1007/978-3-319-29133-8_25

[8]

Google Developers. 2015. https://developers.google.com/glass/develop/gdk/voice.

[9]

Lina Dib, Daniela Petrelli, and Steve Whittaker. 2010. Sonic souvenirs: exploring the paradoxes of recorded sound for family remembering. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 391--400.

Digital Library

[10]

Aiden R Doherty and Alan F Smeaton. 2008. Combining face detection and novelty to identify important events in a visual lifelog. In Computer and Information Technology Workshops, 2008. CIT Workshops 2008. IEEE 8th International Conference on. IEEE, 348--353.

Digital Library

[11]

Mortaza Doulaty, Oscar Saz, and Thomas Hain. 2015. Unsupervised domain discovery using latent dirichlet allocation for acoustic modelling in speech recognition. arXiv preprint arXiv:1509.02412 (2015).

[12]

eMarketer. 2017. Wearable user penetration rate in the United States, in 2017, by age. https://www.statista.com/statistics/739398/us-wearable-penetration-by-age/. [Online; accessed 4-February-2019].

[13]

Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. 2018. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. arXiv preprint arXiv:1804.03619 (2018).

[14]

Mohammad Ghafouri, Shohreh Amini, Kamel Khalili, and Bassel E Sawaya. 2006. HIV-1 associated dementia: symptoms and causes. Retrovirology, Vol. 3, 1 (2006), 28.

[15]

Google Glass. 2015. https://developers.google.com/glass.

[16]

Cathal Gurrin, Alan F. Smeaton, and Aiden R. Doherty. 2014. LifeLogging: Personal Big Data. Foundations and Trends in Information Retrieval, Vol. 8, 1 (2014), 1-125. https://doi.org/10.1561/1500000033

Digital Library

[17]

Kiryong Ha, Zhuo Chen, Wenlu Hu, Wolfgang Richter, Padmanabhan Pillai, and Mahadev Satyanarayanan. 2014. Towards wearable cognitive assistance. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services. ACM, 68--81.

Digital Library

[18]

Catrina M Hacker, Emily X Meschke, and Irving Biederman. 2019. A face in a (temporal) crowd. Vision research, Vol. 157 (2019), 55--60.

[19]

Elmar Hashimov. 2015. Qualitative Data Analysis: A Methods Sourcebook and The Coding Manual for Qualitative Researchers: Matthew B. Miles, A. Michael Huberman, and Johnny Salda na. Thousand Oaks, CA: SAGE, 2014. 381 pp. Johnny Salda na. Thousand Oaks, CA: SAGE, 2013. 303 pp.

[20]

Steve Hodges, Emma Berry, and Ken Wood. 2011. SenseCam: A wearable camera that stimulates and rehabilitates autobiographical memory. Memory, Vol. 19, 7 (2011), 685--696.

[21]

Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. 2006. SenseCam: A retrospective memory aid. In UbiComp 2006: Ubiquitous Computing. Springer, 177--193.

[22]

Lei Hou and Xiangyu Wang. 2013. A study on the benefits of augmented reality in retaining working memory in assembly tasks: A focus on differences in gender. Automation in Construction, Vol. 32 (2013), 38--45.

[23]

Zhanpeng Huang, Weikai Li, and Pan Hui. 2015. Ubii: Towards Seamless Interaction between Digital and Physical Worlds. In Proceedings of the 23rd Annual ACM Conference on Multimedia Conference. ACM, 341--350.

Digital Library

[24]

Sinh Huynh, Rajesh Krishna Balan, and Youngki Lee. 2015. Demo: Towards Recognition of Rich Non-Negative Emotions Using Daily Wearable Devices. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. ACM, 471--472.

Digital Library

[25]

Ellen Isaacs, Artie Konrad, Alan Walendowski, Thomas Lennig, Victoria Hollis, and Steve Whittaker. 2013. Echoes from the past: how technology mediated reflection improves well-being. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1071--1080.

Digital Library

[26]

Katherine Isbister, Kaho Abe, and Michael Karlesky. 2017. Interdependent Wearables (for Play): A Strong Concept for Design. In CHI. 465--471.

[27]

Shiqi Jiang, Zhenjiang Li, Pengfei Zhou, and Mo Li. 2019. Memento: An emotion-driven lifelogging system with wearables. ACM Transactions on Sensor Networks (TOSN), Vol. 15, 1 (2019), 1--23.

Digital Library

[28]

Vaiva Kalnikaite, Abigail Sellen, Steve Whittaker, and David Kirk. 2010. Now let me see where i was: understanding how lifelogs mediate memory. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2045--2054.

Digital Library

[29]

Marion Koelle, Swamy Ananthanarayan, Simon Czupalla, Wilko Heuten, and Susanne Boll. 2018a. Your smart glasses' camera bothers me! exploring opt-in and opt-out gestures for privacy mediation. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction. 473--481.

Digital Library

[30]

Marion Koelle, Katrin Wolf, and Susanne Boll. 2018b. Beyond LED status lights-design requirements of privacy notices for body-worn cameras. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction. 177--187.

Digital Library

[31]

Martina Luchetti and Angelina R Sutin. 2016. Measuring the phenomenology of autobiographical memory: A short form of the Memory Experiences Questionnaire. Memory, Vol. 24, 5 (2016), 592--602.

[32]

Naresh K Malhotra, Sung S Kim, and James Agarwal. 2004. Internet users' information privacy concerns (IUIPC): The construct, the scale, and a causal model. Information systems research, Vol. 15, 4 (2004), 336--355.

[33]

Steve Mann. 1997. Wearable computing: A first step toward personal imaging. Computer, Vol. 30, 2 (1997), 25--32.

Digital Library

[34]

Daniel McDuff, Amy Karlson, Ashish Kapoor, Asta Roseway, and Mary Czerwinski. 2012. AffectAura: an intelligent system for emotional memory. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 849--858.

Digital Library

[35]

A McPherson, FG Furniss, C Sdogati, F Cesaroni, B Tartaglini, and J Lindesay. 2001. Effects of individualized memory aids on the conversation of persons with severe dementia: a pilot study. Aging & Mental Health, Vol. 5, 3 (2001), 289--294.

[36]

Pranav Misra, Alyssa Marconi, Matthew Peterson, and Gabriel Kreiman. 2018. Minimal memory for details in real life events. Scientific reports, Vol. 8, 1 (2018), 1--11.

[37]

Vivian Genaro Motti and Kelly Caine. 2015. Users? privacy concerns about wearables. In International Conference on Financial Cryptography and Data Security. Springer, 231--244.

[38]

OpenCV4Android. 2016. http://opencv.org/platforms/android.html.

[39]

Apache OpenNLP. 2010. https://opennlp.apache.org.

[40]

Mike Owens and Grant Allen. 2010. The Definitive Guide to SQLite. Springer.

[41]

Giambattista Parascandolo, Heikki Huttunen, and Tuomas Virtanen. 2016. Recurrent neural networks for polyphonic sound event detection in real life recordings. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 6440--6444.

[42]

Trevor Pering, Yuvraj Agarwal, Rajesh Gupta, and Roy Want. 2006. Coolspots: reducing the power consumption of wireless mobile devices with multiple radio interfaces. In Proceedings of the 4th international conference on Mobile systems, applications and services. ACM, 220--232.

Digital Library

[43]

Anuj Rathore, Pravin Nagar, Chetan Arora, and CV Jawahar. 2019. Generating 1 Minute Summaries of Day Long Egocentric Videos. In Proceedings of the 27th ACM International Conference on Multimedia. 2305--2313.

Digital Library

[44]

Philipp A Rauschnabel, Alexander Brem, and Bjoern S Ivens. 2015. Who will buy smart glasses? Empirical results of two pre-market-entry studies on the role of personality in individual awareness and intended adoption of Google Glass wearables. Computers in Human Behavior, Vol. 49 (2015), 635--647.

Digital Library

[45]

Philipp A Rauschnabel and Young K Ro. 2016. Augmented reality smart glasses: An investigation of technology acceptance drivers. International Journal of Technology Marketing, Vol. 11, 2 (2016), 123--148.

[46]

Elizabeth B-N Sanders and Pieter Jan Stappers. 2008. Co-creation and the new landscapes of design. Co-design, Vol. 4, 1 (2008), 5--18.

[47]

Corina Sas, Scott Challioner, Christopher Clarke, Ross Wilson, Alina Coman, Sarah Clinch, Mike Harding, and Nigel Davies. 2015. Self-defining memory cues: creative expression and emotional meaning. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2013--2018.

Digital Library

[48]

Corina Sas, Tomasz Fratczak, Matthew Rees, Hans Gellersen, Vaiva Kalnikaite, Alina Coman, and Kristina Höök. 2013. AffectCam: arousal-augmented sensecam for richer recall of episodic memories. In CHI'13 Extended Abstracts on Human Factors in Computing Systems. ACM, 1041--1046.

Digital Library

[49]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition. 815--823.

[50]

Abigail J Sellen, Andrew Fogg, Mike Aitken, Steve Hodges, Carsten Rother, and Ken Wood. 2007. Do life-logging technologies support memory for the past?: an experimental study using sensecam. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 81--90.

Digital Library

[51]

Khaled Shaalan. 2014. A survey of arabic named entity recognition and classification. Computational Linguistics, Vol. 40, 2 (2014), 469--510.

Digital Library

[52]

Yiran Shen, Chengwen Luo, Weitao Xu, and Wen Hu. 2015. Poster: An Online Approach for Gait Recognition on Smart Glasses. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. ACM, 389--390.

Digital Library

[53]

Zhiqi Shen, Shaojing Fan, Yongkang Wong, Tian-Tsong Ng, and Mohan Kankanhalli. 2019. Human-imperceptible Privacy Protection Against Machines. In Proceedings of the 27th ACM International Conference on Multimedia. 1119--1128.

Digital Library

[54]

Jiayu Shu, Rui Zheng, and Pan Hui. 2016. Cardea: Context-Aware Visual Privacy Protection from Pervasive Cameras. eprint arXiv:1610.00889 (Oct 2016), 1--10.

[55]

Enrico Tanuwidjaja, Derek Huynh, Kirsten Koa, Calvin Nguyen, Churen Shao, Patrick Torbett, Colleen Emmenegger, and Nadir Weibel. 2014. Chroma: A wearable augmented-reality solution for color blindness. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 799--810.

Digital Library

[56]

Ying-Chao Tung, Chun-Yen Hsu, Han-Yu Wang, Silvia Chyou, Jhe-Wei Lin, Pei-Jung Wu, Andries Valstar, and Mike Y Chen. 2015. User-defined game input for smart glasses in public space. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3327--3336.

Digital Library

[57]

Dapper Vision. 2014. http://www.openshades.com.

[58]

Willem A Wagenaar. 1986. My memory: A study of autobiographical memory over six years. Cognitive psychology, Vol. 18, 2 (1986), 225--252.

[59]

Emma Woodberry, Georgina Browne, Steve Hodges, Peter Watson, Narinder Kapur, and Ken Woodberry. 2015. The use of a wearable camera improves autobiographical memory in patients with Alzheimer's disease. Memory, Vol. 23, 3 (2015), 340--349.

[60]

WHO World Health Organization. 2017. Dementia. http://www.who.int/news-room/fact-sheets/detail/dementia. [Online; accessed 15-October-2018].

[61]

Y. Wu, X. Shen, T. Mei, X. Tian, N. Yu, and Y. Rui. 2016. Monet: A System for Reliving Your Memories by Theme-Based Photo Storytelling. IEEE Transactions on Multimedia, Vol. 18, 11 (Nov 2016), 2206--2216. https://doi.org/10.1109/TMM.2016.2614185

Digital Library

[62]

Yingying Zhu, Nandita M Nayak, and Amit K Roy-Chowdhury. 2013. Context-aware modeling and recognition of activities in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2491--2498.

Digital Library

Cited By

Gamage B(2024)AI-Enabled Smart Glasses for People with Severe Vision ImpairmentsACM SIGACCESS Accessibility and Computing10.1145/3654768.3654771(1-1)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1145/3654768.3654771
Zulfikar WChan SMaes P(2024)Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory AugmentationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642450(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642450
Gamage BDo TPrice NLowery AMarriott K(2023)What do Blind and Low-Vision People Really Want from Assistive Smart Devices? Comparison of the Literature with a Focus StudyProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608955(1-21)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3608955
Show More Cited By

Recommendations

Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory Augmentation
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

People have to remember an ever-expanding volume of information. Wearables that use information capture and retrieval for memory augmentation can help but can be disruptive and cumbersome in real-world tasks, such as in social settings. To address this, ...
Memento: An Emotion-driven Lifelogging System with Wearables

Due to the increasing popularity of mobile devices, the usage of lifelogging has dramatically expanded. People collect their daily memorial moments and share with friends on the social network, which is an emerging lifestyle. We see great potential of ...
Haven't we met before?: a realistic memory assistance system to remind you of the person in front of you
AH '14: Proceedings of the 5th Augmented Human International Conference

This paper presents a perceived real-time system for memory augmentation. We propose a realistic approach to realize a memory assistance system, focusing on retrieving the person in front of you. The proposed system is capable of fully automatic ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

October 2020

4889 pages

ISBN:9781450379885

DOI:10.1145/3394171

General Chairs:
Chang Wen Chen
Chinese University of Hong Kong, Shenzhen, China
,
Rita Cucchiara
UNIMORE, Italy
,
Xian-Sheng Hua
Alibaba Group, China
,
Program Chairs:
Guo-Jun Qi
Futurewei Technologies, USA
,
Elisa Ricci
UNITN & Fondazione Bruno Kessler, Italy
,
Zhengyou Zhang
Tencent, China
,
Roger Zimmermann
National University of Singapore, Singapore

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Research Grants Council of Hong Kongand the 5GEAR and FIT projects from Academy of Finland

Conference

MM '20

Sponsor:

SIGMM

MM '20: The 28th ACM International Conference on Multimedia

October 12 - 16, 2020

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
357
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)5

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gamage B(2024)AI-Enabled Smart Glasses for People with Severe Vision ImpairmentsACM SIGACCESS Accessibility and Computing10.1145/3654768.3654771(1-1)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1145/3654768.3654771
Zulfikar WChan SMaes P(2024)Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory AugmentationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642450(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642450
Gamage BDo TPrice NLowery AMarriott K(2023)What do Blind and Low-Vision People Really Want from Assistive Smart Devices? Comparison of the Literature with a Focus StudyProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608955(1-21)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3608955
Sun YXu QWang ZHuang QEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)When Measures are Unreliable: Imperceptible Adversarial Perturbations toward Top-k Multi-Label LearningProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611846(1515-1526)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611846
Khan MKhusro S(2023)SmartLog: A Smart TV-Based Lifelogging System for Capturing, Storing, and Visualizing Watching BehaviorInternational Journal of Human–Computer Interaction10.1080/10447318.2023.225005440:20(6232-6251)Online publication date: 29-Aug-2023
https://doi.org/10.1080/10447318.2023.2250054
Shatilov KChatzopoulos DLee LHui P(2021)Emerging ExG-based NUI Inputs in Extended Realities: A Bottom-up SurveyACM Transactions on Interactive Intelligent Systems10.1145/345795011:2(1-49)Online publication date: 21-Jul-2021
https://dl.acm.org/doi/10.1145/3457950
Santana RRossi GMéndez GRodríguez ACajas V(2021)Smart Glasses User Experience in STEM Students: A Systematic Mapping StudyTrends and Applications in Information Systems and Technologies10.1007/978-3-030-72657-7_44(455-467)Online publication date: 23-Apr-2021
https://doi.org/10.1007/978-3-030-72657-7_44

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten