extended-abstract

Superpower glass: delivering unobtrusive real-time social cues in wearable systems

Authors:

Peter Washington,

Carl Feinstein,

Terry Winograd,

Dennis WallAuthors Info & Claims

UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct

Pages 1218 - 1226

https://doi.org/10.1145/2968219.2968310

Published: 12 September 2016 Publication History

Abstract

We have developed a system for automatic facial expression recognition, which runs on Google Glass and delivers real-time social cues to the wearer. We evaluate the system as a behavioral aid for children with Autism Spectrum Disorder (ASD), who can greatly benefit from real-time non-invasive emotional cues and are more sensitive to sensory input than neurotypically developing children. In addition, we present a mobile application that enables users of the wearable aid to review their videos along with auto-curated emotional information on the video playback bar. This integrates our learning aid into the context of behavioral therapy. Expanding on our previous work describing in-lab trials, this paper presents our system and application-level design decisions in depth as well as the interface learnings gathered during the use of the system by multiple children with ASD in an at-home iterative trial.

References

[1]

Alt, Florian, et al. "GravitySpot: Guiding Users in Front of Public Displays Using On-Screen Visual Cues." Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 2015.

Digital Library

[2]

Battocchi, Alberto, et al. "Collaborative Puzzle Game: a tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD)." Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, 2009.

Digital Library

[3]

Bosseler, Alexis, and Dominic W. Massaro. "Development and evaluation of a computer-animated tutor for vocabulary and language learning in children with autism." Journal of autism and developmental disorders 33.6 (2003): 653--672.

[4]

Costanza, Enrico, et al. "Eye-q: Eyeglass peripheral display for subtle intimate notifications." Proceedings of the 8th conference on Human-computer interaction with mobile devices and services. ACM, 2006.

Digital Library

[5]

Dalal, Navneet, and Bill Triggs. "Histograms of oriented gradients for human detection." Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 1. IEEE, 2005.

Digital Library

[6]

el Kaliouby, Rana, Rosalind Picard, and S. I. M. O. N. BARONCOHEN. "Affective computing and autism." Annals of the New York Academy of Sciences 1093.1 (2006): 228--248.

[7]

Feil-Seifer, David, and Maja J. Matarić. "Toward socially assistive robotics for augmenting interventions for children with autism spectrum disorders. "Experimental robotics. Springer Berlin Heidelberg, 2009.

[8]

Gal, Eynat, et al. "Enhancing social communication of children with high-functioning autism through a co-located interface." Ai & Society 24.1 (2009): 75--84.

Digital Library

[9]

Giusti, Leonardo, et al. "Dimensions of collaboration on a tabletop interface for children with autism spectrum disorder." Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2011.

Digital Library

[10]

Golan, O., Ashwin, E., Granader, Y., McClintock, S., Day, K., Leggett, V. and Baron-Cohen, S., 2010. Enhancing emotion recognition in children with autism spectrum conditions: An intervention using animated vehicles with real emotional faces. Journal of autism and developmental disorders, 40(3), pp.269--279.

[11]

Goodrich, Michael A., et al. "A case for low-dose robotics in autism therapy." Proceedings of the 6th international conference on Human-robot interaction. ACM, 2011.

Digital Library

[12]

Gross, Ralph, et al. "Multi-pie." Image and Vision Computing 28.5 (2010): 807--813.

Digital Library

[13]

Haber, Nick, et al. "A practical approach to real-time neutral feature subtraction for facial expression recognition." 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2017.

[14]

Hernandez, Javier, and Rosalind W. Picard. "SenseGlass: Using Google Glass to sense daily emotions." Proceedings of the adjunct publication of the 27th annual ACM symposium on User interface software and technology. ACM, 2014.

Digital Library

[15]

Hetzroni, Orit E., and Juman Tannous. "Effects of a computer-based intervention program on the communicative functions of children with autism." Journal of autism and developmental disorders 34.2 (2004): 95--113.

[16]

Kandalaft, Michelle R., et al. "Virtual reality social cognition training for young adults with high-functioning autism." Journal of autism and developmental disorders 43.1 (2013): 34--44.

[17]

Kientz, J.A., Hayes, G.R., Arriaga, R.I., & Abowd, G.D. (2008). Designing and Developing Technology for Caregivers of Individuals with Autism. Presented the Technology and Mental Health Workshop at CHI 2008. Florence, Italy.

[18]

Krishna, Sreekar, et al. "VibroGlove: an assistive technology aid for conveying facial expressions." CHI'10 Extended Abstracts on Human Factors in Computing Systems. ACM, 2010.

Digital Library

[19]

Kumar, Karthik, and Yung-Hsiang Lu. "Cloud computing for mobile users: Can offloading computation save energy?." Computer 4 (2010): 51--56.

Digital Library

[20]

Kurata, Takeshi, et al. "VizWear: Toward human-centered interaction through wearable vision and visualization." Advances in Multimedia Information Processing---PCM 2001. Springer Berlin Heidelberg, 2001. 40--47.

Digital Library

[21]

Lerman, Dorothea C., et al. "A rapid assessment of skills in young children with autism." Journal of Applied Behavior Analysis 37.1 (2004): 11.

[22]

LiKamWa, Robert, et al. "Energy characterization and optimization of image sensing toward continuous mobile vision." Proceeding of the 11th annual international conference on Mobile systems, applications, and services. ACM, 2013.

Digital Library

[23]

Lucey, Patrick, et al. "The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression." Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on. IEEE, 2010.

[24]

Lyons, Michael, et al. "Coding facial expressions with gabor wavelets." Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on. IEEE, 1998.

Digital Library

[25]

Madsen, Miriam, et al. "Technology for just-in-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder." Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility. ACM, 2008.

Digital Library

[26]

Myles, Brenda Smith, et al. "Sensory issues in children with Asperger syndrome and autism." Education and Training in Developmental Disabilities(2004): 283--290.

[27]

Pantic, Maja, et al. "Web-based database for facial expression analysis." Multimedia and Expo, 2005. ICME 2005. IEEE International Conference on. IEEE, 2005.

[28]

Park, Ji Hyun, Bretagne Abirached, and Yan Zhang. "A framework for designing assistive technologies for teaching children with ASDs emotions." CHI'12 Extended Abstracts on Human Factors in Computing Systems. ACM, 2012.

Digital Library

[29]

Pavel, Amy, et al. "Video digests: a browsable, skimmable format for informational lecture videos." UIST. 2014.

Digital Library

[30]

Picard, Rosalind W. Affective computing. Vol. 252. Cambridge: MIT press, 1997.

[31]

Picard, Rosalind W. "Affective Computing for HCI." HCI (1). 1999.

Digital Library

[32]

Picard, Rosalind W. "Affective computing: from laughter to IEEE." Affective Computing, IEEE Transactions on 1.1 (2010): 11--17.

Digital Library

[33]

Sakunkoo, Nathan, and Pattie Sakunkoo. "GliFlix: Using Movie Subtitles for Language Learning." UIST, 2013.

[34]

Savran, Arman, et al. "Bosphorus database for 3D face analysis." Biometrics and Identity Management. Springer Berlin Heidelberg, 2008. 47--56.

Digital Library

[35]

Sim, Terence, Simon Baker, and Maan Bsat. "The CMU pose, illumination, and expression (PIE) database." Automatic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on. IEEE, 2002.

Digital Library

[36]

Strickland, Dorothy. "Virtual reality for the treatment of autism." Studies in health technology and informatics (1997): 81--86.

[37]

Tan, Xiaoyang, and Bill Triggs. "Enhanced local texture feature sets for face recognition under difficult lighting conditions." Image Processing, IEEE Transactions on 19.6 (2010): 1735--1750.

Digital Library

[38]

Teeters, Alea Chandler. Use of a wearable camera system in conversation: Toward a companion tool for social-emotional learning in autism. Diss. Massachusetts Institute of Technology, 2007.

[39]

Valdez, Patricia, and Albert Mehrabian. "Effects of color on emotions." Journal of Experimental Psychology: General 123.4 (1994): 394.

[40]

Washington, Peter, et al. A Wearable Social Interaction Aid for Children with Autism. ACM annual conference on Human Factors in Computing Systems (CHI) Late-Breaking Work, 2016.

Digital Library

[41]

Wilson, Jeff, et al. "Swan: System for wearable audio navigation." Wearable Computers, 2007 11th IEEE International Symposium on. IEEE, 2007.

Digital Library

[42]

Yin, Lijun, et al. "A high-resolution 3D dynamic facial expression database." Automatic Face & Gesture Recognition, 2008. FG'08. 8th IEEE International Conference On. IEEE, 2008.

Cited By

Khang AJadhav BSayyed M(2024)Role of Cutting-Edge Technologies and Deep Learning Frameworks in the Digital Healthcare SectorAI-Driven Innovations in Digital Healthcare10.4018/979-8-3693-3218-4.ch001(1-22)Online publication date: 5-Jan-2024
https://doi.org/10.4018/979-8-3693-3218-4.ch001
Kargarandehkordi AKaisti MWashington P(2024)Personalization of Affective Models Using Classical Machine Learning: A Feasibility StudyApplied Sciences10.3390/app1404133714:4(1337)Online publication date: 6-Feb-2024
https://doi.org/10.3390/app14041337
Jaiswal AWashington P(2024)Using #ActuallyAutistic on Twitter for Precision Diagnosis of Autism Spectrum Disorder: Machine Learning StudyJMIR Formative Research10.2196/526608(e52660)Online publication date: 14-Feb-2024
https://doi.org/10.2196/52660
Show More Cited By

Index Terms

Superpower glass: delivering unobtrusive real-time social cues in wearable systems
1. Human-centered computing

Recommendations

A Wearable Social Interaction Aid for Children with Autism
CHI EA '16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems

Over 1 million children under the age of 17 in the US have been identified with Autism Spectrum Disorder (ASD). These children struggle to recognize facial expressions, make eye contact, and engage in social interactions. Gaining these skills requires ...
SuperpowerGlass: A Wearable Aid for the At-Home Therapy of Children with Autism

We have developed a system for automatic facial expression recognition running on Google Glass, delivering real-time social cues to children with Autism Spectrum Disorder (ASD). The system includes multiple mechanisms to engage children and their parents,...
Go go games: therapeutic video games for children with autism spectrum disorders
IDC '13: Proceedings of the 12th International Conference on Interaction Design and Children

In this paper, we describe the design of a therapeutic video game suite for early elementary children with autism spectrum disorders (ASD). The purpose of this work is to present our hypothesis that games that are both fun and faithful to evidence-based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct

September 2016

1807 pages

ISBN:9781450344623

DOI:10.1145/2968219

General Chairs:
Paul Lukowicz
DFKI and University of Kaiserslautern
,
Antonio Krüger
DFKI and Saarland University
,
Program Chairs:
Andreas Bulling
Max Planck Institute for Informatics
,
Youn-Kyung Lim
KAIST
,
Shwetak N. Patel
University of Washington

Copyright © 2016 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2016

Check for updates

Author Tags

Qualifiers

Extended-abstract

Funding Sources

Google
Dekeyser and Friends Foundation
David and Lucile Packard Foundation

Conference

UbiComp '16

Sponsor:

UbiComp '16: The 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 12 - 16, 2016

Germany, Heidelberg

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
718
Total Downloads

Downloads (Last 12 months)56
Downloads (Last 6 weeks)5

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Khang AJadhav BSayyed M(2024)Role of Cutting-Edge Technologies and Deep Learning Frameworks in the Digital Healthcare SectorAI-Driven Innovations in Digital Healthcare10.4018/979-8-3693-3218-4.ch001(1-22)Online publication date: 5-Jan-2024
https://doi.org/10.4018/979-8-3693-3218-4.ch001
Kargarandehkordi AKaisti MWashington P(2024)Personalization of Affective Models Using Classical Machine Learning: A Feasibility StudyApplied Sciences10.3390/app1404133714:4(1337)Online publication date: 6-Feb-2024
https://doi.org/10.3390/app14041337
Jaiswal AWashington P(2024)Using #ActuallyAutistic on Twitter for Precision Diagnosis of Autism Spectrum Disorder: Machine Learning StudyJMIR Formative Research10.2196/526608(e52660)Online publication date: 14-Feb-2024
https://doi.org/10.2196/52660
Jaiswal AKruiper RRasool ANandkeolyar AWall DWashington P(2024)Digitally Diagnosing Multiple Developmental Delays Using Crowdsourcing Fused With Machine Learning: Protocol for a Human-in-the-Loop Machine Learning StudyJMIR Research Protocols10.2196/5220513(e52205)Online publication date: 8-Feb-2024
https://doi.org/10.2196/52205
Neuhaus RRingfort-Felner RCourtney DKneile MHassenzahl M(2024)Virtual Unreality: Augmentation-Oriented Ideation Through Design CardsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642364(1-23)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642364
Gao XYin LTian SHuang YJi Q(2024)Wearable Technology for Signal Acquisition and Interactive Feedback in Autism Spectrum Disorder Intervention: A ReviewIEEE Sensors Journal10.1109/JSEN.2024.337476224:9(13797-13815)Online publication date: 1-May-2024
https://doi.org/10.1109/JSEN.2024.3374762
El Shemy IJaccheri LGiannakos MVulchanova M(2024)Augmented reality-enhanced language learning for children with autism spectrum disorder: a systematic literature reviewBehaviour & Information Technology10.1080/0144929X.2024.230460743:16(4097-4124)Online publication date: 19-Jan-2024
https://doi.org/10.1080/0144929X.2024.2304607
Neuhaus RRingfort-Felner RSadeghian SHassenzahl M(2024)To mimic reality or to go beyond? “Superpowers” in virtual reality, the experience of augmentation and its consequencesInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103165181:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103165
Pandya SJain SVerma J(2024)A comprehensive analysis towards exploring the promises of AI-related approaches in autism researchComputers in Biology and Medicine10.1016/j.compbiomed.2023.107801168(107801)Online publication date: Jan-2024
https://doi.org/10.1016/j.compbiomed.2023.107801
Toma MTurcu CTurcu CVlad STiliute DPascu P(2023)Extended Reality-Based Mobile Application Solutions for the Therapy of Children with Autism Spectrum Disorders: A Systematic Literature Review (Preprint)JMIR Serious Games10.2196/49906Online publication date: 13-Jun-2023
https://doi.org/10.2196/49906
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten