skip to main content
10.1145/3610541.3614566acmconferencesArticle/Chapter ViewAbstractPublication Pagessiggraph-asiaConference Proceedingsconference-collections
abstract

Visual-gestural Interface for Auslan Virtual Assistant

Published: 28 November 2023 Publication History

Abstract

For Deaf people to use natural language interfaces, technologies must understand sign language input and respond in the same language. We developed a prototype smart home assistant that utilises gesture-based controls, improving accessibility and convenience for Deaf users. The prototype features Zelda, an interactive signing avatar that provides responses in Auslan (Australian Sign Language), enabling effective two-way communication. Our live demonstration includes gesture recognition and sign production.

Supplementary Material

MP4 File (cNgcn-emt_106s1-yaml5_supp_video.mp4)
Supplementary Video

References

[1]
Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudrealt, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, Christian Vogler, and Meredith Ringel Morris. 2019. Sign language recognition, generation, and translation: An interdisciplinary perspective. In ASSETS ’19. Pittsburgh, PA, USA, 16–31. https://doi.org/10.1145/3308561.3353774
[2]
Necati Cihan Camgoz, Ben Saunders, Guillaume Rochette, Marco Giovanelli, Giacomo Inches, Robin Nachtrab-Ribback, and R. Bowden. 2021. Content4All open research sign language translation datasets. In FG 2021. 1–5.
[3]
Aashaka Desai, Lauren Berger, Fyodor O Minakov, Vanessa Milan, Chinmay Singh, Kriston Pumphrey, Richard E Ladner, Hal Daumé III, Alex X Lu, Naomi Caselli, and Danielle Bragg. 2023. ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition. arXiv preprint arXiv:2304.05934 (2023).
[4]
Trevor A Johnston. 2008. Auslan Corpus. https://elar.soas.ac.uk/Collection/MPI55247
[5]
Trevor A Johnston and Adam Schembri. 2010. Variation, lexicalization and grammaticalization in signed languages. Langage et société 1, 131 (2010), 19–35.
[6]
Jessica Korte, Axel Bender, Guy Gallasch, Janet Wiles, and Andrew D. Back. 2020. A plan for developing an Auslan Communication Technologies Pipeline. Lecture Notes in Computer Science 12536 (2020), 264–277. https://doi.org/10.1007/978-3-030-66096-3_19
[7]
Jessica Korte, Julie Lyons, and Adele Vogel. 2022a. Auslan personal assistant functionality identified through interviews. Technical Report. TAS DCRC and DSTG.
[8]
Jessica Korte, Adele Vogel, and Maria Zelenskaya. 2022b. Maximising usability of an avatar for visual-gestural user interfaces. Technical Report. TAS DCRC and DSTG.
[9]
Verena Krausneker and Sandra Schügerl. 2021. Best practice protocol on the use of sign language avatars. Technical Report. University of Vienna, Vienna, Austria. 1–17 pages. https://avatar-bestpractice.univie.ac.at/
[10]
Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera. 2021. Sign language recognition: A deep survey. Expert Systems with Applications 164 (2021), 113794. https://doi.org/10.1016/j.eswa.2020.113794
[11]
Panneer Selvam Santhalingam, Al Amin Hosain, Ding Zhang, Parth Pathak, Huzefa Rangwala, and Raja Kushalnagar. 2020. MmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-Wave Signals. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 1, Article 26 (mar 2020), 30 pages. https://doi.org/10.1145/3381010
[12]
Xin Shen, Shaozu Yuan, Hongwei Sheng, Heming Du, and Xin Yu. 2023. Auslan-Daily: Australian Sign Language translation for daily communication and news. https://uq-cvlab.github.io/Auslan-Daily-Dataset/
[13]
Shashindi Vithanage. 2022. Auslan Alexa: VR "Wizard of Oz". Honours. The University of Queensland.
[14]
Shashindi Vithanage, Arindam Dey, and Jessica Korte. 2023. ’Auslan Alexa’: A case study of VR Wizard of Oz for requirements elicitation. In IEEE Conference on Virtual Reality and 3D User Interfaces. IEEE, Shanghai, 911–912. https://doi.org/10.1007/978-3-030
[15]
World Federation of the Deaf and World Association of Sign Language Interpreters. 2018. WFD and WASLI statement on use of signing avatars. https://wfdeaf.org/news/resources/wfd-wasli-statement-use-signing-avatars/

Cited By

View all
  • (2024)What Factors Motivate Culturally Deaf People to Want Assistive Technologies?Extended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650871(1-7)Online publication date: 11-May-2024
  • (2024)AI empowered Auslan learning for parents of deaf children and children of deaf adultsAI and Ethics10.1007/s43681-024-00457-y4:4(877-887)Online publication date: 18-Mar-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SA '23: SIGGRAPH Asia 2023 Emerging Technologies
November 2023
53 pages
ISBN:9798400703126
DOI:10.1145/3610541
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2023

Check for updates

Author Tags

  1. Auslan
  2. Deaf
  3. accessibility
  4. gesture recognition
  5. sign language
  6. signing avatar
  7. smart home
  8. virtual assistant

Qualifiers

  • Abstract
  • Research
  • Refereed limited

Funding Sources

Conference

SA '23
Sponsor:
SA '23: SIGGRAPH Asia 2023
December 12 - 15, 2023
NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)64
  • Downloads (Last 6 weeks)1
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)What Factors Motivate Culturally Deaf People to Want Assistive Technologies?Extended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650871(1-7)Online publication date: 11-May-2024
  • (2024)AI empowered Auslan learning for parents of deaf children and children of deaf adultsAI and Ethics10.1007/s43681-024-00457-y4:4(877-887)Online publication date: 18-Mar-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media