research-article

Face Recognition Assistant for People with Visual Impairments

Authors:

Mohammad Kianpisheh,

Franklin Mingzhe Li,

Khai N. TruongAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 3, Issue 3

Article No.: 90, Pages 1 - 24

https://doi.org/10.1145/3351248

Published: 09 September 2019 Publication History

Abstract

Although there are many face recognition systems to help individuals with visual impairments (VIPs) recognize other people, almost all require a database with the pictures and names of the people who should be tracked. These solutions would not be able to help VIPs recognize people they might not know well. In this work, we investigate the requirements and challenges that must be addressed in the design of a face recognition system for helping VIPs recognize people with whom they have weak-ties. We first conducted a formative study with eight visually impaired people. Using insights learned from the formative study, we developed a research prototype that runs on a mobile phone worn around the user's neck. The developed prototype is a wearable face recognition system that opportunistically captures and stores undistorted face images and contextual information about the user's interaction with each person to a database, without the user intervention, as she interacts with new people. We then used this prototype application as a technology probe---asking VIP participants to use the device in a realistic scenario in which they meet and re-encounter several new people. We analyze and report feedback collected from VIPs about the design and use of such a service.

References

[1]

S. Panchanathan, S. Chakraborty, & T. McDaniel. (2016). Social Interaction Assistant: A Person-Centered Approach to Enrich Social Interactions for Individuals with Visual Impairments. IEEE Journal on Selected Topics in Signal Processing, 10(5), 942--951.

[2]

W. Wiener, G. Lawson, Audition for the traveler who is visually impaired. Foundations of orientation and mobility 2 (1997) 104--169

[3]

L. E. Dreer, T.R. Elliott, D. C. Fletcher, & M. Swanson, (2005). Social problem-solving abilities and psychological adjustment of persons in low vision rehabilitation. Rehabilitation Psychology, 50(3), 232--238.

[4]

K. Shinohara, & J. O. Wobbrock, (2011). In the shadow of misperception. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI '11 (p. 705). New York, New York, USA: ACM Press.

Digital Library

[5]

Claudine McCreadie, and Anthea Tinker. "The acceptability of assistive technology to older people." Ageing & Society 25.1 (2005): 91--110.

[6]

Weiyang Liu, et al. "Sphereface: Deep hypersphere embedding for face recognition." The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Vol. 1. 2017.

[7]

Mark S. Granovetter, "The strength of weak ties." Social networks. 1977. 347--367.

[8]

Elizabeth Wolfe. Morrison, "Newcomers' relationships: The role of social network ties during socialization."Academy of management Journal 45.6 (2002): 1149--1160.

[9]

Caroline Haythornthwaite, "Strong, weak, and latent ties and the impact of new media." The information society 18.5 (2002): 385--401.

[10]

Mark S. Granovetter, "The Strength of Weak Ties." American Journal of Sociology, vol. 78, no. 6, 1973, pp. 1360--1380. JSTOR, JSTOR, www.jstor.org/stable/2776392.

[11]

G. M. Sandstrom, & E. W. Dunn. (2014). Social interactions and well-being: The surprising power of weak ties. Personality and Social Psychology Bulletin, 40(7), 910--922.

[12]

C. N. Crittenden, M. L. M. Murphy, & S. Cohen, (2018). Social integration and age-related decline in lung function. Health Psychology.

[13]

A. Dhand, D. Luke, M. Tsiaklides, C. Lang, & J. M. Lee, (2016, February). Patients' Social Networks Influence Timing of Hospital Arrival After Acute Ischemic Stroke: a Mixed Methods Study. In STROKE (Vol. 47). TWO COMMERCE SQ, 2001 MARKET ST, PHILADELPHIA, PA 19103 USA: LIPPINCOTT WILLIAMS & WILKINS.

[14]

Y. Zhao, S. Wu, L. Reynolds, S. Azenkot, Science, I., Tech, C., & Park, M. (2018). A Face Recognition Application for People with Visual Impairments: Understanding Use Beyond the Lab, 1--14.

Digital Library

[15]

K. M. Kramer, D. S. Hedin, & D. J. Rolkosky, (2010, August). Smartphone based face recognition tool for the blind. In Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE (pp. 4538--4541). IEEE.

[16]

L. B. Neto, F. Grijalva, V. R. M. L. Maike, L.D. Martini, D. Florencio, M. C. C. Baranauskas, S. Goldenstein, (2017). A Kinect-Based Wearable Face Recognition System to Aid Visually Impaired Users. IEEE Transactions on Human-Machine Systems, 47(1), 52--64.

[17]

S. Krishna, G. Little, J. Black, & S. Panchanathan. (2005). A wearable face recognition system for individuals with visual impairments. In Proceedings of the 7th international ACM SIGACCESS conference on Computers and accessibility - Assets '05 (p. 106). New York, New York, USA: ACM Press.

Digital Library

[18]

S. Chaudhry, & R. Chandra, (2015). Design of a mobile face recognition system for visually impaired persons. ArXiv Preprint ArXiv:1502.00756. Retrieved from http://arxiv.org/abs/1502.00756

[19]

S. Panchanathan, S. Chakraborty, & T. McDaniel, (2016). Social Interaction Assistant: A Person-Centered Approach to Enrich Social Interactions for Individuals with Visual Impairments. IEEE Journal on Selected Topics in Signal Processing, 10(5), 942--951.

[20]

Thad E. Starner, "Attention, memory, and wearable interfaces." IEEE pervasive computing 1, no. 4 (2002): 88--91.

Digital Library

[21]

Weidong Shi, Xi Wang, Xi Zhao, Varun Prakash, and Omprakash Gnawali. 2013. "Computerized-Eyewear Based Face Recognition System for Improving Social Lives of Prosopagnosics." In Proceedings of the ICTs for Improving Patients Rehabilitation Research Techniques. IEEE.

Digital Library

[22]

Masakazu Iwamura, Kai Kunze, Yuya Kato, Yuzuko Utsumi, and Koichi Kise. 2014. "Haven't We Met before? - A Realistic Memory Assistance System to Remind You of The Person in Front of You." In Augmented Human International Conference 2014, 1--4.

Digital Library

[23]

Y. Utsumi, Y. Kato, K. Kunze, M. Iwamura, and K. Kise. 2013. "Who Are You?: A Wearable Face Recognition System to Support Human Memory." Proceedings of Augmented Human 2013, 150--53.

Digital Library

[24]

Martin Kurze and Axel Roselius. 2011. Smart glasses linking real live and social network's contacts by face recognition. In Proceedings of the 2nd Augmented Human International Conference (AH '11). ACM, New York, NY, USA, Article 31, 2 pages.

Digital Library

[25]

Edward T. Hall, et. al. "The hidden dimension, vol. 1990." NY: Anchor Books (1969).

[26]

Massimiliano Patacchiola, and Angelo Cangelosi. "Head pose estimation in the wild using convolutional neural networks and adaptive gradient methods." Pattern Recognition 71 (2017): 132--143.

[27]

Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. "Joint face detection and alignment using multitask cascaded convolutional networks." IEEE Signal Processing Letters 23, no. 10 (2016): 1499--1503.

[28]

Voykinska, Violeta, Shiri Azenkot, Shaomei Wu, and Gilly Leshed. "How blind people interact with visual content on social networking services." In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, pp. 1584--1595. ACM, 2016.

Digital Library

[29]

Marynel Vázquez, and Aaron Steinfeld. "Helping visually impaired users properly aim a camera." In Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility, pp. 95--102. ACM, 2012.

Digital Library

[30]

M. Hassaballah, and Saleh Aly. "Face recognition: challenges, achievements and future directions." IET Computer Vision 9, no. 4 (2015): 614--626.

Digital Library

[31]

J. Beveridge, P. Ross, Jonathon Phillips, David S. Bolme, Bruce A. Draper, Geof H. Givens, Yui Man Lui, Mohammad Nayeem Teli et al. "The challenge of face recognition from digital point-and-shoot cameras." In Biometrics: Theory, Applications and Systems (BTAS), 2013 IEEE Sixth International Conference on, pp. 1--8. IEEE, 2013.

[32]

José Luis Pech-Pacheco, et. al. "Diatom autofocusing in brightfield microscopy: a comparative study." In Pattern Recognition, 2000. Proceedings. 15th International Conference on, vol. 3, pp. 314--317. IEEE, 2000.

Digital Library

[33]

Fenwick, Kent, Michael Massimi, Ronald Baecker, Sandra Black, Kevin Tonon, Cosmin Munteanu, Elizabeth Rochon, and David Ryan. "Cell phone software aiding name recall." In CHI'09 Extended Abstracts on Human Factors in Computing Systems, pp. 4279--4284. ACM, 2009.

Digital Library

[34]

Anja Kintsch, and Rogerio DePaula. "A framework for the adoption of assistive technology." SWAAAC 2002: Supporting learning through assistive technology (2002): 1--10.1

[35]

H. Day, et. al. "The stability of impact of assistive devices." Disability and rehabilitation 23, no. 9 (2001): 400--404.

[36]

Betsy Phillips, and Hongxin Zhao. "Predictors of assistive technology abandonment." Assistive technology 5, no. 1 (1993): 36--45.

[37]

Anja Kintsch, and Rogerio DePaula. "A framework for the adoption of assistive technology." SWAAAC 2002: Supporting learning through assistive technology (2002): 1--10.

[38]

Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. "Quantized convolutional neural networks for mobile devices." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4820--4828. 2016.

[39]

Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. "Compression of deep convolutional neural networks for fast and low power mobile applications." arXiv preprint arXiv:1511.06530 (2015).

[40]

Andrew G Howard, Zhu Menglong, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. "Mobilenets: Efficient convolutional neural networks for mobile vision applications." arXiv preprint arXiv:1704.04861 (2017).

[41]

https://www.zdnet.com/article/huawei-unveils-ai-ascend-chips/

[42]

Laurindo de Sousa Britto Neto, et. al. "A Wearable Face Recognition System Built into a Smartwatch and the Visually Impaired User." In ICEIS (3), pp. 5--12. 2015.

Digital Library

[43]

Ning Zhang, Manohar Paluri, Yaniv Taigman, Rob Fergus, and Lubomir Bourdev. "Beyond frontal faces: Improving person recognition using multiple cues." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4804--4813. 2015.

[44]

Teresa E. Seeman, "Social ties and health: The benefits of social integration." Annals of epidemiology 6, no. 5 (1996): 442--451.

[45]

B. Sarason, et. al. "Close personal relationships and health outcomes: A key to the role of social support." Handbook of personal relationships: Theory, research and interventions (2nd ed) Chichester, UK: Wiley(1997): 547--573.

[46]

OrCam. OrCam - See for Yourself. Retrieved May 10, 2019 from https://www.orcam.com/en/

[47]

H. Bredin, G. Gelly, Improving speaker diarization of TV series using talking-face detection and clustering. InProceedings of the 24th ACM international conference on Multimedia 2016 Oct 1 (pp. 157--161). ACM.

Digital Library

[48]

I. D. Gebru, et. al. Audio-visual speaker diarization based on spatiotemporal bayesian fusion. IEEE transactions on pattern analysis and machine intelligence. 2018 May 1;40(5):1086--99.

Cited By

Li FWang ACarrington PKane S(2024)A Recipe for Success? Exploring Strategies for Improving Non-Visual Access to Cooking InstructionsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675662(1-15)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675662
Bian SLiu MZhou BLukowicz PMagno M(2024)Body-Area Capacitive or Electric Field Sensing for Human Activity Recognition and Human-Computer InteractionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435558:1(1-49)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643555
Bandukda MWang YPerusquia-Hernandez MLi FHolloway C(2024)Context matters: Investigating information sharing in mixed-visual ability social interactionsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651121(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651121
Show More Cited By

Index Terms

Face Recognition Assistant for People with Visual Impairments
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing

Recommendations

The challenges in adopting assistive technologies in the workplace for people with visual impairments
OzCHI '18: Proceedings of the 30th Australian Conference on Computer-Human Interaction

There are many barriers to employment for people with visual impairments. Assistive technologies (ATs), such as computer screen readers and enlarging software, are commonly used to help overcome employment barriers and enable people with visual ...
LêRótulos: A Mobile Application Based on Text Recognition in Images to Assist Visually Impaired People
Universal Access in Human-Computer Interaction. Methods, Technologies, and Users
Abstract
The autonomy of the visual impaired person can be evaluated in day to day activities like recognizing objects, identifying textual information, among others. This paper features the OCR technology-based LêRótulos application, with the objective of ...
The Emerging Professional Practice of Remote Sighted Assistance for People with Visual Impairments
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

People with visual impairments (PVI) must interact with a world they cannot see. Remote sighted assistance (RSA) has emerged as a conversational assistive technology. We interviewed RSA assistants ("agents") who provide assistance to PVI via a ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 3, Issue 3

September 2019

1415 pages

EISSN:2474-9567

DOI:10.1145/3361560

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2019

Published in IMWUT Volume 3, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
688
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li FWang ACarrington PKane S(2024)A Recipe for Success? Exploring Strategies for Improving Non-Visual Access to Cooking InstructionsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675662(1-15)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675662
Bian SLiu MZhou BLukowicz PMagno M(2024)Body-Area Capacitive or Electric Field Sensing for Human Activity Recognition and Human-Computer InteractionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435558:1(1-49)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643555
Bandukda MWang YPerusquia-Hernandez MLi FHolloway C(2024)Context matters: Investigating information sharing in mixed-visual ability social interactionsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651121(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3651121
Li FLiu MKane SCarrington P(2024)A Contextual Inquiry of People with Vision Impairments in CookingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642233(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642233
Li FCarrington P(2023)Empowering Autonomy and Agency: Exploring and Augmenting Accessible Cyber-Physical SystemsAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610770(263-266)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610770
Li FZhang LBandukda MStangl AShinohara KFindlater LCarrington P(2023)Understanding Visual Arts Experiences of Blind PeopleProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580941(1-21)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580941
Singh ASingh AKaur J(2023)Using Self-Driving Techniques for Blind Assistance2023 International Conference in Advances in Power, Signal, and Information Technology (APSIT)10.1109/APSIT58554.2023.10201728(520-525)Online publication date: 9-Jun-2023
https://doi.org/10.1109/APSIT58554.2023.10201728
Mauro NCena FPutnam CPera MRoldán Álvarez D(2023)Introduction to this special issue on intelligent systems for people with diverse cognitive abilitiesHuman–Computer Interaction10.1080/07370024.2023.225196939:1-2(1-7)Online publication date: 6-Sep-2023
https://doi.org/10.1080/07370024.2023.2251969
Li FLu CLu ZCarrington PTruong K(2022)An Exploration of Captioning Practices and Challenges of Individual Content Creators on YouTube for People with Hearing ImpairmentsProceedings of the ACM on Human-Computer Interaction10.1145/35129226:CSCW1(1-26)Online publication date: 7-Apr-2022
https://dl.acm.org/doi/10.1145/3512922
Akter TAhmed TKapadia ASwaminathan M(2022)Shared Privacy Concerns of the Visually Impaired and Sighted Bystanders with Camera-Based Assistive TechnologiesACM Transactions on Accessible Computing10.1145/350685715:2(1-33)Online publication date: 19-May-2022
https://dl.acm.org/doi/10.1145/3506857
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents