skip to main content
research-article

Effect of Sign-recognition Performance on the Usability of Sign-language Dictionary Search

Published: 15 October 2021 Publication History

Abstract

Advances in sign-language recognition technology have enabled researchers to investigate various methods that can assist users in searching for an unfamiliar sign in ASL using sign-recognition technology. Users can generate a query by submitting a video of themselves performing the sign they believe they encountered somewhere and obtain a list of possible matches. However, there is disagreement among developers of such technology on how to report the performance of their systems, and prior research has not examined the relationship between the performance of search technology and users’ subjective judgements for this task. We conducted three studies using a Wizard-of-Oz prototype of a webcam-based ASL dictionary search system to investigate the relationship between the performance of such a system and user judgements. We found that, in addition to the position of the desired word in a list of results, the placement of the desired word above or below the fold and the similarity of the other words in the results list affected users’ judgements of the system. We also found that metrics that incorporate the precision of the overall list correlated better with users’ judgements than did metrics currently reported in prior ASL dictionary research.

References

[1]
Azzah Al-Maskari, Mark Sanderson, and Paul Clough. 2007. The relationship between IR effectiveness measures and user satisfaction. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’07). Association for Computing Machinery, New York, NY, 773–774. https://doi.org/10.1145/1277741.1277902
[2]
Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. 2019. Effect of automatic sign recognition performance on the usability of video-based search interfaces for sign language dictionaries. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS’19). Association for Computing Machinery, New York, NY, 56–67. https://doi.org/10.1145/3308561.3353791
[3]
Vassilis Athitsos, Carol Neidle, Stan Sclaroff, Joan Nash, Alexandra Stefan, Ashwin Thangali, Haijing Wang, and Quan Yuan. 2010. Large lexicon project: American sign language video corpus and sign language indexing/retrieval algorithms. In Proceedings of the Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies (CSLT’10), Vol. 2. European Language Resources Association (ELRA), Valletta, Malta, 11–14.
[4]
Debra L. Blackwell, Jacqueline W. Lucas, and Tainya C. Clarke. 2014. Summary health statistics for U.S. adults: National health interview survey, 2012. Vital and Health Statistics. Series 10, Data from the National Health Survey 1, 260 (Feb. 2014), 171. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/24819891.
[5]
Danielle Bragg, Kyle Rector, and Richard E. Ladner. 2015. A user-powered american sign language dictionary. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work; Social Computing (CSCW’15). Association for Computing Machinery, New York, NY, 1837–1848. https://doi.org/10.1145/2675133.2675226
[6]
Andrei Broder. 2002. A taxonomy of web search. SIGIR Forum 36, 2 (Sept. 2002), 3–10. https://doi.org/10.1145/792550.792552
[7]
Naomi K. Caselli, Zed Sevcikova Sehyr, Ariel M. Cohen-Goldberg, and Karen Emmorey. 2017. ASL-LEX: A lexical database of american sign language. Behav. Res. Methods 49, 2 (2017), 784–801.
[8]
Dorothy Casterline, Carl Croneberg, et al. 1965. A dictionary of American Sign Language on linguistic principles. Gallaudet College.
[9]
Matias D. Cattaneo, Nicolas Idrobo, and Rocio Titiunik. 2019. A Practical Introduction to Regression Discontinuity Designs: Foundations. Retrieved from https://ideas.repec.org/p/arx/papers/1911.09511.html.
[10]
Christopher Conly, Zhong Zhang, and Vassilis Athitsos. 2015. An integrated RGB-D system for looking up the meaning of signs. In Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA’15). Association for Computing Machinery, New York, NY, Article 24, 8 pages. https://doi.org/10.1145/2769493.2769534
[11]
H. Cooper, N. Pugeault, and R. Bowden. 2011. Reading the signs: A video based sign dictionary. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV’11). IEEE Computer Society, Los Alamitos, CA, 914–919. https://doi.org/10.1109/ICCVW.2011.6130349
[12]
Natalia Lusin and Dennis Looney. 2018. Enrollments in languages other than english in united states institutions of higher education, summer 2016 and fall 2016: Preliminary report. In Proceedings of the Modern Language Association of America. 92. Retrieved from https://www.mla.org/content/download/110154/2406932/2016-Enrollments-Final-Report.pdf.
[13]
Ralph Elliott, Helen Cooper, John Glauert, Richard Bowden, and François Lefebvre-Albaret. 2011. Search-by-example in multilingual sign language databases. In Proceedings of the 2nd International Workshop on Sign Language Translation and Avatar Technology (SLTAT’11). SLTAT, Dundee, Scotland, 8. Retrieved from http://personal.ee.surrey.ac.uk/Personal/H.Cooper/research/papers/SBE_SLTAT.pdf.
[14]
Scott B. Huffman and Michael Hochster. 2007. How well does result relevance predict session satisfaction? In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’07). Association for Computing Machinery, New York, NY, 567–574. https://doi.org/10.1145/1277741.1277839
[15]
Guido Imbens and Karthik Kalyanaraman. 2012. Optimal bandwidth choice for the regression discontinuity estimator. Rev. Econ. Studies 79, 3 (2012), 933–959.
[16]
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Info. Syst. 20, 4 (Oct. 2002), 422–446.https://doi.org/10.1145/582415.582418
[17]
Jiepu Jiang and James Allan. 2016. Correlation between system and user metrics in a session. In Proceedings of the ACM on Conference on Human Information Interaction and Retrieval (CHIIR’16). Association for Computing Machinery, New York, NY, 285–288. https://doi.org/10.1145/2854946.2855005
[18]
Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’05). Association for Computing Machinery, New York, NY, 154–161. https://doi.org/10.1145/1076034.1076063
[19]
Matt Jones, Gary Marsden, Norliza Mohd-Nasir, Kevin Boone, and George Buchanan. 1999. Improving web interaction on small displays. Comput. Netw. 31, 11–16 (1999), 1129–1137.
[20]
Diane Kelly and Leif Azzopardi. 2015. How many results per page? A study of SERP size, search behavior and user experience. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15). Association for Computing Machinery, New York, NY, 183–192. https://doi.org/10.1145/2766462.2767732
[21]
Jaewon Kim, Paul Thomas, Ramesh Sankaranarayana, Tom Gedeon, and Hwan-Jin Yoon. 2015. Eye-tracking analysis of user behavior and performance in web search on large and small screens. J. Assoc. Info. Sci. Technol. 66, 3 (2015), 526–544.
[22]
J. Lapiak. 2021. Handspeak. Retrieved from https://www.handspeak.com/.
[23]
Scott Liddell. 2003. Grammar, gesture, and meaning in american sign language. Grammar, Gesture, and Meaning in American Sign Language 1 (Mar. 2003), 404. https://doi.org/10.1017/CBO9780511615054
[24]
Dimitris Metaxas, Mark Dilsizian, and Carol Neidle. 2018. Linguistically-driven framework for computationally efficient and scalable sign recognition. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC’18). European Language Resources Association (ELRA), Miyazaki, Japan, 8. Retrieved from https://www.aclweb.org/anthology/L18-1271.
[25]
Ross Mitchell and Michael Karchmer. 2004. Chasing the mythical ten percent: Parental hearing status of deaf and hard of hearing students in the united states. Sign Lang. Studies 4 (12 2004), 138–163. https://doi.org/10.1353/sls.2004.0005
[26]
Ross Mitchell, Travas Young, Bellamie Bachleda, and Michael Karchmer. 2006. How many people use ASL in the united states? Why estimates need updating. Sign Lang. Studies 6 (03 2006). https://doi.org/10.1353/sls.2006.0019
[27]
Carol Neidle and Joan Cottle Poole Nash. 2015. American sign language. Jullie Bakken Jepsen, Goedele De Clerck, Sam Lutalo Kiingi, and William B. McGregor (eds.), Sign Languages of the World 1, 1 (2015), 31–70.
[28]
Carol Neidle and Christian Vogler. 2012. A new web interface to facilitate access to corpora: Development of the ASLLRP data access interface (DAI). In Proceedings of the 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon (LREC’12). Citeseer, OpenBU, Istanbul, Turkey, 8. https://open.bu.edu/handle/2144/31886.
[29]
Antti Oulasvirta, Janne P. Hukkinen, and Barry Schwartz. 2009. When more is less: The paradox of choice in search engine use. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’09). Association for Computing Machinery, New York, NY, 516–523. https://doi.org/10.1145/1571941.1572030
[30]
John Sören Pettersson and Malin Wik. 2015. The longevity of general purpose wizard-of-oz tools. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI’15). Association for Computing Machinery, New York, NY, 422–426. https://doi.org/10.1145/2838739.2838825
[31]
Kishore K. Reddy and Mubarak Shah. 2013. Recognizing 50 human action categories of web videos. Mach. Vision Appl. 24, 5 (2013), 971–981.
[32]
Naomi Robbins and Richard Heiberger. 2011. Plotting likert and other rating scales. Proceedings of the Joint Statistical Meeting. 9.
[33]
Daniel E. Rose and Danny Levinson. 2004. Understanding user goals in web search. In Proceedings of the 13th International Conference on World Wide Web (WWW’04). Association for Computing Machinery, New York, NY, 13–19. https://doi.org/10.1145/988672.988675
[34]
ShuR. 2021. SLintoDictionary. Retrieved from http://slinto.com/us.
[35]
Jenny L. Singleton and Elissa L. Newport. 2004. When learners surpass their models: The acquisition of american sign language from inconsistent input. Cogn. Psychol. 49, 4 (2004), 370–407. https://doi.org/10.1016/j.cogpsych.2004.05.001
[36]
Richard A. Tennant, Marianne Gluszak, and Marianne Gluszak Brown. 1998. The American Sign Language Handshape Dictionary. Gallaudet University Press, Washington, D.C.
[37]
U. von Agris, C. Blomer, and K. Kraiss. 2008. Rapid signer adaptation for continuous sign language recognition using a combined approach of eigenvoices, MLLR, and MAP. In Proceedings of the 19th International Conference on Pattern Recognition (ICPR’08). IEEE Computer Society, Los Alamitos, CA. https://doi.org/10.1109/ICPR.2008.4761363
[38]
Haijing Wang, Alexandra Stefan, Sajjad Moradi, Vassilis Athitsos, Carol Neidle, and Farhad Kamangar. 2012. A system for large vocabulary sign search. In Trends and Topics in Computer Vision, Kiriakos N. Kutulakos (Ed.). Springer, Berlin, 342–353.
[39]
Kimberly A. Weaver and Thad Starner. 2011. We need to communicate! helping hearing parents of deaf children learn american sign language. In The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS’11). Association for Computing Machinery, New York, NY, 91–98. https://doi.org/10.1145/2049536.2049554
[40]
Polina Yanovich, Carol Neidle, and Dimitris Metaxas. 2016. Detection of major ASL sign types in continuous signing for ASL recognition. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16). European Language Resources Association (ELRA), Portorož, Slovenia, 3067–3073. Retrieved from https://www.aclweb.org/anthology/L16-1490.

Cited By

View all
  • (2024)Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL LearnersACM Transactions on Accessible Computing10.1145/369064717:3(1-35)Online publication date: 4-Oct-2024
  • (2024)Designing and Evaluating an Advanced Dance Video Comprehension Tool with In-situ Move Identification CapabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642710(1-19)Online publication date: 11-May-2024
  • (2023)Sign Spotter: Design and Initial Evaluation of an Automatic Video-Based American Sign Language Dictionary SystemProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614497(1-5)Online publication date: 22-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Accessible Computing
ACM Transactions on Accessible Computing  Volume 14, Issue 4
December 2021
171 pages
ISSN:1936-7228
EISSN:1936-7236
DOI:10.1145/3485142
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2021
Accepted: 01 June 2021
Revised: 01 March 2021
Received: 01 July 2020
Published in TACCESS Volume 14, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. American sign language (ASL)
  2. dictionary
  3. search
  4. information retrieval

Qualifiers

  • Research-article
  • Refereed

Funding Sources

  • National Science Foundation

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)98
  • Downloads (Last 6 weeks)5
Reflects downloads up to 30 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Exploring the Benefits and Applications of Video-Span Selection and Search for Real-Time Support in Sign Language Video Comprehension among ASL LearnersACM Transactions on Accessible Computing10.1145/369064717:3(1-35)Online publication date: 4-Oct-2024
  • (2024)Designing and Evaluating an Advanced Dance Video Comprehension Tool with In-situ Move Identification CapabilitiesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642710(1-19)Online publication date: 11-May-2024
  • (2023)Sign Spotter: Design and Initial Evaluation of an Automatic Video-Based American Sign Language Dictionary SystemProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614497(1-5)Online publication date: 22-Oct-2023
  • (2023)Querying A Sign Language Dictionary with Videos Using Dense Vector Search2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)10.1109/ICASSPW59220.2023.10193531(1-5)Online publication date: 4-Jun-2023
  • (2022)Design and Evaluation of Hybrid Search for American Sign Language to English Dictionaries: Making the Most of Imperfect Sign RecognitionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501986(1-13)Online publication date: 29-Apr-2022
  • (2022)Performance Analysis of Sign Language Recognition System Using Hybrid Feature DescriptorCyber Technologies and Emerging Sciences10.1007/978-981-19-2538-2_38(381-388)Online publication date: 30-Aug-2022

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media