Rúbia E. O. Schultz Ascari
Roberto Pereira
ABSTRACT
Augmentative and Alternative Communication (AAC) involves the use of non-verbal modes as a complement or substitute for spoken language, supporting communicative abilities of people, especially people with speech limitations. Computing systems have been proposed to support AAC, applying different technology to address users' different needs. Computer vision techniques can assist people with motor impairments by using their remaining functional motions. This paper proposes a methodology to support AAC of people with motor impairments, using computer vision and machine learning techniques to enable personalized gestural interaction. The methodology was instantiated in a pilot system described in this paper and evaluated by Human-Computer Interaction experts. The evaluation results suggested improvements for the methodology and for the system, and indicated the methodology is feasible to support the design of AAC systems, and that the developed system is promising to support AAC.
- Julio Abascal. 2008. Users with disabilities: maximum control with minimum effort. Articulated Motion and Deformable Objects (2008), 449--456. Google Scholar
Digital Library
- Abdullah Al Mahmud. 2012. Iterative design to improve aphasic communication. Ph.D. Dissertation. Technische Universiteit Eindhoven.Google Scholar
- Natasha Alves, Stefanie Blain, Tiago Falk, Brian Leung, Negar Memarian, and Tom Chau. 2016. Access Technologies for Children and Youth with Severe Motor Disabilities. Paediatric Rehabilitation Engineering: From Disability to Possibility (2016), 45.Google Scholar
- Governo de Aragão. 2018. Portal Aragonês de Comunicação Aumentativa e Alternativa. http://www.arasaac.org. Acessado em 13/08/2018.Google Scholar
- Rúbia Eliza de Oliveira Schultz Ascari, Roberto Pereira, and Luciano Silva. 2018. Mobile Interaction for Augmentative and Alternative Communication: a Systematic Mapping. SBC Journal on 3D Interactive Systems 9, 2 (2018), 105--118.Google Scholar
- Behrooz Ashtiani and Ian Scott MacKenzie. 2010. BlinkWrite2: an improved text entry method using eye blinks. In Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications. ACM, 339--345. Google ScholarDigital Library
- Samy Bakheet. 2017. A Fuzzy Framework for Real-Time Gesture Spotting and Recognition. Journal of Russian Laser Research 38, 1 (2017), 61--75.Google ScholarCross Ref
- Maria Cecilia Baranauskas, Clarisse de Souza, and Roberto Pereira. 2015. GranDIHC-BR--Grand Research Challenges for Human-Computer Interaction in Brazil. Human-Computer Interaction Special Committee (CEIHC) of the Brazilian Computer Society (SBC) (2015).Google Scholar
- Cassio T Batista, Erick M Campos, and Nelson C Sampaio Neto. 2017. A Proposal of a Universal Remote Control System Based on Head Movements. In Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems. ACM, 2. Google ScholarDigital Library
- Selma Belgacem, Clément Chatelain, and Thierry Paquet. 2017. Gesture sequence recognition with one shot learned CRF/HMM hybrid model. Image and Vision Computing 61 (2017), 12--21. Google ScholarDigital Library
- Margrit Betke, James Gips, and Peter Fleming. 2002. The camera mouse: visual tracking of body features to provide computer access for people with severe disabilities. IEEE Transactions on neural systems and Rehabilitation Engineering 10, 1 (2002), 1--10.Google ScholarCross Ref
- Pradipta Biswas and Pat Langdon. 2013. A new interaction technique involving eye gaze tracker and scanning system. In Proceedings of the 2013 Conference on Eye Tracking South Africa. ACM, 67--70. Google ScholarDigital Library
- Luciana Correia Lima de Faria Borges, Lucia Filgueiras, Cristiano Maciel, and Vinicius Pereira. 2013. A customized mobile application for a cerebral palsy user. In Proceedings of the 31st ACM international conference on Design of communication. ACM, 7--16. Google ScholarDigital Library
- Dario Cazzato, Marco Leo, and Cosimo Distante. 2014. An investigation on the feasibility of uncalibrated and unconstrained gaze tracking for human assistive applications by using head pose estimation. Sensors 14, 5 (2014), 8363--8379.Google ScholarCross Ref
- Weiqin Chen. 2013. Gesture-based applications for elderly people. In International Conference on Human-Computer Interaction. Springer, 186--195. Google ScholarDigital Library
- Camilla Almeida da Silva, António Ramires Fernandes, and Ana Paula Grohmann. 2014. STAR: Speech Therapy with Augmented Reality for Children with Autism Spectrum Disorders. In International Conference on Enterprise Information Systems. Springer, 379--396.Google Scholar
- Luciana Correia Lima De Faria Borges, Lucia Vilela Leite Filgueiras, and Cristiano Maciel. 2011. Towards a participatory development technique of assistive technology for mobility and speech impaired patients. In Proceedings of the 10th Brazilian Symposium on Human Factors in Computing Systems and the 5th Latin American Conference on Human-Computer Interaction. Brazilian Computer Society, 247--256. Google ScholarDigital Library
- Luciana Correia Lima de Faria Borges, Lucia Vilela Leite Filgueiras, Cristiano Maciel, and Vinicius Carvalho Pereira. 2012. Customizing a communication device for a child with cerebral palsy using participatory design practices: contributions towards the PD4CAT method. In Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems. Brazilian Computer Society, 57--66. Google ScholarDigital Library
- Archana Vilas Dehankar, Sanjeev Jain, and Vilas M. Thakare. 2017. Using AEPI method for hand gesture recognition in varying background and blurred images. In Electronics, Communication and Aerospace Technology (ICECA), 2017 International conference of Vol. 1. IEEE, 404--409.Google Scholar
- Archana Vilas Dehankar, Vilas M. Thakare, and Sanjeev Jain. 2017. Detecting centroid for hand gesture recognition using morphological computations. In Inventive Systems and Control (ICISC), 2017 International Conference on. IEEE, 1--5.Google ScholarCross Ref
- Débora Deliberato, Maria de Jesus Gonçalves, and Eliseu Coutinho de Macedo. 2009. Comunicação alternativa: teoria, prática, tecnologias e pesquisa. São Paulo: Memnon (2009).Google Scholar
- Athanasios Drigas and Georgia Kokkalia. 2016. Mobile Learning for Special Preschool Education. International Journal of Interactive Mobile Technologies (iJIM) 10, 1 (2016), 60--67.Google ScholarCross Ref
- Sergey Yurievich Eroshkin, Nataliya Aleksandrovna Kameneva, Dzhordzh Vladimirovich Kovkov, and Alexander Ilyich Sukhorukov. 2017. Conceptual system in the modern information management. Procedia Computer Science 103 (2017), 609--612. Google ScholarDigital Library
- Guo Wei Gao and Xin Yu Duan. 2011. An overview of human-computer interaction based on the camera for disabled people. In Advanced Materials Research, Vol. 219. Trans Tech Publ, 1317--1320.Google ScholarCross Ref
- Tania Rossi Garbin. 2008. Ambientes de comunicação alternativos com base na realidade aumentada para crianças com paralisia cerebral: uma proposta de currículo em ação. (2008).Google Scholar
- Tania Rossi Garbin and Carlos Alberto Dainese. 2009. AmCARA-Ambiente e Comunicação Alternativo com Realidade Aumentada: O acesso do deficiente motor severo a softwares e Web. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 1.Google Scholar
- Francisco Gomez-Donoso and Miguel Cazorla. 2015. Recognizing Schaeffer's gestures for robot interaction. Advances in Artificial Intelligence, CAEPIA (2015).Google Scholar
- Kristen Grauman, Margrit Betke, Jonathan Lombardi, James Gips, and Gary R Bradski. 2003. Communication via eye blinks and eyebrow raises: Video-based human-computer interfaces. Universal Access in the Information Society 2, 4 (2003), 359--373. Google ScholarDigital Library
- Aashni Haria, Archanasri Subramanian, Nivedhitha Asokkumar, Shristi Poddar, and Jyothi S Nayak. 2017. Hand gesture recognition for human computer interaction. Procedia Computer Science 115 (2017), 367--374. Google ScholarDigital Library
- Chin-Pan Huang, Chaur-Heh Hsieh, Kuan-Ting Lai, and Wei-Yang Huang. 2011. Human action recognition using histogram of oriented gradient of motion history image. In Instrumentation, Measurement, Computer, Communication and Control, 2011 First International Conference on. IEEE, 353--356. Google ScholarDigital Library
- Chaudhary Muhammad Aqdus Ilyas, Mohammad A Haque, Matthias Rehm, Kamal Nasrollahi, and Thomas B Moeslund. 2017. Facial Expression Recognition for Traumatic Brain Injured Patients. In International Conference on Computer Vision Theory and Applications. SCITEPRESS Digital Library.Google Scholar
- Robert J. K. Jacob. 1991. The use of eye movements in human-computer interaction techniques: what you look at is what you get. ACM Transactions on Information Systems (TOIS) 9, 2 (1991), 152--169. Google ScholarDigital Library
- Dan Jellinek and Peter Abrahams. 2012. Moving together: mobile apps for inclusion and assistance. OneVoice for Accessible ICT (2012).Google Scholar
- Ghassan Kbar, Akshay Bhatia, Mustufa Haider Abidi, and Ibraheem Alsharawy. 2017. Assistive technologies for hearing, and speaking impaired people: a survey. Disability and Rehabilitation: Assistive Technology 12, 1 (2017), 3--20.Google ScholarCross Ref
- Tomasz Kocejko, Adam Bujnowski, and Jerzy Wtorek. 2009. Eye-mouse for disabled. In Human-computer systems interaction. Springer, 109--122.Google Scholar
- Chris Lankford. 2000. Effective eye-gaze input into windows. In Proceedings of the 2000 symposium on Eye tracking research & applications. ACM, 23--27. Google ScholarDigital Library
- Luciane Aparecida Liegel, Milena Maria Rodege Gogola, and Percy Nohama. 2008. Layout de teclado para uma prancha de comunicação alternativa e ampliada Keyboard layout for an augmentative and alternative communication board. Revista brasileira de educaçao especial 14, 3 (2008), 479--496.Google Scholar
- Li Liu, Shuo Niu, Jingjing Ren, and Jingyuan Zhang. 2012. Tongible: a non-contact tongue-based interaction technique. In Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility. ACM, 233--234. Google ScholarDigital Library
- Ian Scott MacKenzie and Behrooz Ashtiani. 2010. BlinkWrite: efficient text entry using eye blinks. Universal Access in the Information Society 10, 1 (2010), 69--80. Google ScholarDigital Library
- Cristina Manresa-Yee, Pere Ponsa, Javier Varona, and Francisco J Perales. 2010. User experience to improve the usability of a vision-based interface. Interacting with Computers 22, 6 (2010), 594--605. Google ScholarDigital Library
- Negar Memarian, Tom Chau, and Anastasios N Venetsanopoulos. 2009. Application of infrared thermal imaging in rehabilitation engineering: Preliminary results. In Science and Technology for Humanity (TIC-STH), 2009 IEEE Toronto International Conference. IEEE, 1--5.Google ScholarCross Ref
- Negar Memarian, Anastasios N Venetsanopoulos, and Tom Chau. 2009. Infrared thermography as an access pathway for individuals with severe motor impairments. Journal of neuroengineering and rehabilitation 6, 1 (2009), 11.Google ScholarCross Ref
- Karyn Moffatt, Golnoosh Pourshahid, and Ronald M. Baecker. 2015. Augmentative and alternative communication devices for aphasia: the emerging role of "smart" mobile devices. Universal Access in the Information Society (2015), 1--14. Google ScholarDigital Library
- Kenny Morrison and Stephen J McKenna. 2002. Contact-free recognition of user-defined gestures as a means of computer access for the physically disabled. In Workshop on Universal Access and Assistive Technology. 99--103.Google Scholar
- Giorgos Mountrakis, Jungho Im, and Caesar Ogole. 2011. Support vector machines in remote sensing: A review. ISPRS Journal of Photogrammetry and Remote Sensing 66, 3 (2011), 247--259.Google ScholarCross Ref
- Cosmin Munteanu, Sharon Oviatt, Gerald Penn, and Randy Gomez. 2016. Designing Speech and Multimodal Interactions for Mobile, Wearable, and Pervasive Applications. (2016), 3612--3619. Google ScholarDigital Library
- Nirmita Narasimhan. 2010. e-Accessibility Policy Handbook for Persons with Disabilities. (2010).Google Scholar
- Farhood Negin, Pau Rodriguez, Michal Koperski, Adlen Kerboua, Jordi Gonzàlez, Jeremy Bourgeois, Emmanuelle Chapoulie, Philippe Robert, and Francois Bremond. 2018. PRAXIS: Towards Automatic Cognitive Assessment Using Gesture Recognition. Expert Systems with Applications (2018).Google Scholar
- Shuo Niu, Li Liu, and D Scott McCrickard. 2018. Tongue-able interfaces: Prototyping and evaluating camera based tongue gesture input system. Smart Health (2018).Google Scholar
- Ryan O'Grady, Charles J Cohen, Glenn Beach, and Gary Moody. 2004. NaviGaze: enabling access to digital media for the profoundly disabled. In Information Theory, 2004. ISIT 2004. Proceedings. International Symposium on. IEEE, 211--216. Google ScholarDigital Library
- Nuria Oliver, Alex P Pentland, and Francois Berard. 1997. Lafter: Lips and face real time tracker. In Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on. IEEE, 123--129. Google ScholarDigital Library
- S Oprea, A Garcia-Garcia, J Garcia-Rodriguez, S Orts-Escolano, and M Cazorla. 2017. A recurrent neural network based Schaeffer gesture recognition system. In Neural Networks (IJCNN), 2017 International Joint Conference on. IEEE, 425--431.Google ScholarCross Ref
- Jasmina Ivšac Pavliša, Marta Ljubešić, and Ivana Jerečić. 2012. The use of AAC with young children in croatia--from the speech and language pathologist's view. In KES International Symposium on Agent and Multi-Agent Systems: Technologies and Applications. Springer, 221--230. Google ScholarDigital Library
- Emanuele Perini, Simone Soria, Andrea Prati, and Rita Cucchiara. 2006. Face-Mouse: A human-computer interface for tetraplegic people. In European Conference on Computer Vision. Springer, 99--108. Google ScholarDigital Library
- Thiago Pradi, Luciano Silva, Olga R. P. Bellon, and Gustavo M. S. Dória. 2016. Ferramentas de computação visual para apoio ao treinamento de expressões faciais por autistas: uma revisão de literatura. 430 Seminário Integrado de Software e Hardware (SEMISH) (2016).Google Scholar
- Maria Armanda Quintela, Mafalda Mendes, and Secundino Correia. 2013. Augmentative and alternative communication: Vox4all® presentation. In Information Systems and Technologies (CISTI), 2013 8th Iberian Conference on. IEEE, 1--6.Google Scholar
- Cristhian Rosales, Luis Jácome, Jorge Carrión, Carlos Jaramillo, and Mario Palma. 2017. Computer vision for detection of body expressions of children with cerebral palsy. In Ecuador Technical Chapters Meeting (ETCM), 2017 IEEE. IEEE, 1--6.Google ScholarCross Ref
- Sancho Salcedo-Sanz, José Luis Rojo-Álvarez, Manel Martínez-Ramón, and Gustavo Camps-Valls. 2014. Support vector machines in engineering: an overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 4, 3 (2014), 234--267. Google ScholarDigital Library
- Benson Schaeffer, Arlene Musil, and George Kollinzas. 1985. Total communication: A signed speech program for nonverbal children. Research.Google Scholar
- CALL Scotland. 2018. Wheel of AAC Apps for Communication. http://www.aacscotland.org.uk/Resources/Wheel-of-AAC-Apps-for-Communication/. Acessado em 19/07/2018.Google Scholar
- Catherine Caudle Smith. 2013. Using mobile technology to improve autonomy in students with intellectual disabilities in postsecondary education programs. (2013).Google Scholar
- Valencia University Spain and The Orange Foundation Spain. 2017. The Pictograms Room. www.pictogramas.org. Acessado em 11/05/2017.Google Scholar
- Claudia Tambascia, Ismael Ávila, Lara Piccolo, and Amanda Meincke Melo. 2008. Usabilidade, acessibilidade e inteligibilidade aplicadas em interfaces para analfabetos, idosos e pessoas com deficiência. In Proceedings of the VIII Brazilian Symposium on Human Factors in Computing Systems. Sociedade Brasileira de Computação, 354--355. Google ScholarDigital Library
- Pang-Ning Tan et al. 2007. Introduction to data mining. Pearson Education India.Google Scholar
- Ippei Torii, Kaoruko Ohtani, Nahoko Shirahama, Takahito Niwa, and Naohiro Ishii. 2012. Voice output communication aid application for personal digital assistant for autistic children. In Computer and Information Science (ICIS), 2012 IEEE/ACIS 11th International Conference on. IEEE, 329--333. Google ScholarDigital Library
- Diego Torricelli, Michela Goffredo, Silvia Conforto, and Maurizio Schmid. 2009. An adaptive blink detector to initialize and update a view-basedremote eye gaze tracking system in a natural scenario. Pattern Recognition Letters 30, 12 (2009), 1144--1150. Google ScholarDigital Library
- Kentaro Toyama. 1998. Look, ma-no hands! hands-free cursor control with real-time 3d face tracking. PUI98 (1998).Google Scholar
- Jason Vazquez-Li, Lyle Pierson Stachecki, and John Magee. 2016. Eye-gaze with predictive link following improves accessibility as a mouse pointing interface. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 297--298. Google ScholarDigital Library
- Paul Viola and Michael Jones. 2001. Rapid object detection using a boosted cascade of simple features. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, Vol. 1. IEEE, I--I.Google ScholarCross Ref
- Stephen VonTetzchner and Mogens Hygum Jensen. 1996. Augmentative and alternative communication: European perspectives. Whurr Publishers.Google Scholar
- Ikushi Yoda, Kazuyuki Ito, and Tsuyoshi Nakayama. 2017. Modular Gesture Interface for People with Severe Motor Dysfunction: Foot Recognition. Studies in health technology and informatics 242 (2017), 725--732.Google Scholar
- Shasha Zhang, Weicun Zhang, and Yunluo Li. 2016. Human Action Recognition Based on Multifeature Fusion. In Proceedings of 2016 Chinese Intelligent Systems Conference. Springer, 183--192.Google ScholarCross Ref
Index Terms
- Towards a Methodology to Support Augmentative and Alternative Communication by means of Personalized Gestural Interaction
Recommendations
Methodology based on Computer Vision and Machine Learning to guide the Design of Augmentative and Alternative Communication Systems using Personalized Gestural Interaction
IHC '21: Proceedings of the XX Brazilian Symposium on Human Factors in Computing SystemsPeople who have motor disabilities associated with some difficulty in speech demand alternative means to interact with other people and the environment in which they are inserted. Augmentative and Alternative Communication (AAC) refers to all ...
Personalized gestural interaction applied in a gesture interactive game-based approach for people with disabilities
IUI '20: Proceedings of the 25th International Conference on Intelligent User InterfacesTechnology can support people with disabilities to participate in social and economic life. Using relevant Human-Computer Interaction, as obtained through Intelligent User Interfaces, people with motor and speech impairments may be able to communicate ...
Computer Vision-based Methodology to Improve Interaction for People with Motor and Speech Impairment
Augmentative and Alternative Communication (AAC) aims to complement or replace spoken language to compensate for expression difficulties faced by people with speech impairments. Computing systems have been developed to support AAC; however, partially ...
Comments