skip to main content
research-article

Access to multimodal articles for individuals with sight impairments

Published: 01 January 2013 Publication History

Abstract

Although intelligent interactive systems have been the focus of many research efforts, very few have addressed systems for individuals with disabilities. This article presents our methodology for an intelligent interactive system that provides individuals with sight impairments with access to the content of information graphics (such as bar charts and line graphs) in popular media. The article describes the methodology underlying the system's intelligent behavior, its interface for interacting with users, examples processed by the implemented system, and evaluation studies both of the methodology and the effectiveness of the overall system. This research advances universal access to electronic documents.

References

[1]
Abu Doush, I., Pontelli, E., Simon, D., and Ma, O. 2009. Making microsoft excel accessible: Multimodal presentation of charts. In Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility. 147--154.
[2]
Alm, N., Newell, A., and Arnott, J. 1987. A communication aid which models conversational patterns. In Proceedings of the 10th Annual Conference on Rehabilitation Technology. 127--129.
[3]
Alty, J. L. and Rigas, D. 2005. Exploring the use of structured musical stimuli to communicate simple diagrams: The role of context. Int. J. Hum. Comput. Stud. 62, 1, 21--40.
[4]
Ang, J., Dhillon, R., Krupski, A., Shriberg, E., and Stolcke, A. 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In Proceedings of the 7th International Conference on Spoken Language Processing. 2037--2040.
[5]
Black, R., Reddington, J., Reiter, E., Tintarev, N., and Waller, A. 2010. Using NLG and sensors to support personal narrative for children with complex communication needs. In Proceedings of the NAACL Workshop on Speech and Language Processing for Assistive Technologies. 1--9.
[6]
Bradley, D. C., Steil, G. M., and Bergman, R. N. 1995. OOPSEG: a data smoothing program for quantitation and isolation of random measurement error. Comput. Meth. Program. Biomed. 46, 1, 67--77.
[7]
Brennan, R. and Prediger, D. 1981. Coefficient kappa: Some uses, misuses, and alternatives. Educ. Psychol. Meas. 41, 3, 687--699.
[8]
Brin, S. and Page, L. 1998. The anatomy of a large-scale hypertextual web search engine. Comput. Netwo. ISDN Syst. 30, 1--7, 107--117.
[9]
Brown, L. M. and Brewster, S. A. 2003. Drawing by ear: Interpreting sonified line graphs. In Proceedings of the 9th International Conference on Auditory Display. 152--156.
[10]
Burns, R., Carberry, S., and Elzer, S. 2010. Visual and spatial factors in a bayesian reasoning framework for the recognition of intended messages in grouped bar charts. In Proceedings of the AAAI Workshop on Visual Representations and Reasoning. 6--13.
[11]
Burns, R., Carberry, S., Elzer, S., and Chester, D. 2012. Automatically recognizing intended messages in grouped bar charts. In Proceedings of the 7th International Conference on the Theory and Application of Diagrams. 8--22.
[12]
Carberry, S. and Elzer, S. 2007. Exploiting evidence analysis in plan recognition. In Proceedings of the International Conference on User Modeling. 7--16.
[13]
Carberry, S., Elzer, S., and Demir, S. 2006. Information graphics: An untapped resource for digital libraries. In Proceedings of the 29th Annual International ACM Conference on Research and Development in Information Retrieval. 581--588.
[14]
Chen, C. 2004. Information Visualization: Beyond the Horizon. Springer.
[15]
Clark, H. H. 1996. Using Language. Cambridge University Press, Cambridge, UK.
[16]
Cockburn, A., Karlson, A., and Bederson, B. B. 2008. A review of overview+detail, zooming, and focus+context interfaces. ACM Comput. Surv. 41, 1--31.
[17]
Cohen, J. A. 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 1, 37--46.
[18]
Cohen, R. F., Yu, R., Meacham, A., and Skaff, J. 2005. Plumb: Displaying graphs to the blind using an active auditory interface. In Proceedings of the 7th International ACM Conference on Computers and Accessibility. 182--183.
[19]
Covington, M. A., He, C., Brown, C., Naçi, L., and Brown, J. 2006. How complex is that sentence? A proposed revision of the Rosenberg and Abbeduto D-level scale. CASPR Res. rep. 2006-01, Artificial Intelligence Center, University of Georgia, Athens, Georgia.
[20]
Coyne, P. and Nielsen, J. 2001. Beyond ALT Text: Making the Web Easy to Use for Users with Disabilities. Nielsen-Norman Group, Fremont, CA.
[21]
Daniel, W. 1989. Applied Nonparametric Statistics, 2nd ed. PWS-Kent Publishing.
[22]
Demir, S. 2010. SIGHT for visually impaired users: Summarizing information graphics textually. Ph.D. thesis, University of Delaware.
[23]
Demir, S., Carberry, S., and McCoy, K. F. 2008. Generating textual summaries of bar charts. In Proceedings of the 5th International Natural Language Generation Conference. 7--15.
[24]
Demir, S., Carberry, S., and McCoy, K. F. 2010. A discourse-aware graph-based content-selection framework. In Proceedings of the 6th International Natural Language Generation Conference. 17--25.
[25]
Elhadad, M. and Robin, J. 1999. SURGE: A comprehensive plug-in syntactic realization component for text generation. Tech. rep. Department of Computer Science, Ben-Gurion University of the Negev, Beer Sheva, Israel.
[26]
Elzer, S. 2005. A probabilistic framework for the recognition of intention in information graphics. Ph.D. thesis, University of Delaware.
[27]
Elzer, S., Carberry, S., Chester, D., Demir, S., Green, N., Zukerman, I., and Trnka, K. 2005a. Exploring and exploiting the limited utility of captions in recognizing intention in information graphics. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics. 223--230.
[28]
Elzer, S., Carberry, S., Zukerman, I., Chester, D., Green, N., and Demir, S. 2005b. A probabilistic framework for recognizing intention in information graphics. In Proceedings of the International Joint Conference on Artificial Intelligence. 1042--1047.
[29]
Elzer, S., Green, N., Carberry, S., and Hoffman, J. 2006. A model of perceptual task effort for bar charts and its role in recognizing intention. User Model. User-Adapt. Interact. 16, 1--30.
[30]
Ferres, L., Lindgaard, G., and Sumegi, L. 2010. Evaluating a tool for improving accessibility to charts and graphs. In Proceedings of the 12th International ACM Conference on Computers and Accessibility. 83--90.
[31]
Ferres, L., Verkhogliad, P., Lindgaard, G., Boucher, L., Chretien, A., and Lachance, M. 2007. Improving accessibility to statistical graphs: The iGraph-Lite system. In Proceedings of the 9th International ACM Conference on Computers and Accessibility. 67--74.
[32]
Fleiss, J. L. 1971. Measuring nominal scale agreement among many raters. Psychol. Bull. 76, 5, 378--382.
[33]
Flowers, J. H. and Hauer, T. A. 1995. Musical versus visual graphs: Cross-modal equivalence in perception of time series data. Human Factors: The J. Hum. Factors Ergonom. Soc. 37, 553--569.
[34]
Fritz, J. P. and Barner, K. E. 1999. Design of a haptic data visualization system for people with visual impairments. IEEE Trans. Rehab. Engin. 7, 372--384.
[35]
Gerber, E. 2002. Surfing by ear: Usability concerns of computer users who are blind or visually impaired. AccessWorld 3, 1, 38--43.
[36]
Gips, J. 1998. On building intelligence into eagleeyes. In Assistive Technology and Artificial Intelligence, Applications in Robotics, User Interfaces and Natural Language Processing, V. O. Mittal, H. A. Yanco, J. Aronis, and R. Simpson, Eds., Springer, 50--58.
[37]
Goncu, C. and Marriott, K. 2008. Tactile chart generation tool. In Proceedings of the 10th International ACM Conference on Computers and Accessibility. 255--256.
[38]
Goncu, C., Marriott, K., and Hurst, J. 2010. Usability of accessible bar charts. In Proceedings of the 6th International Conference on the Theory and Application of Diagrams. 167--181.
[39]
Green, N. L., Carenini, G., Kerpedjiev, S., Mattis, J., Moore, J. D., and Roth, S. F. 2004. Autobrief: An experimental system for the automatic generation of briefings in integrated text and information graphics. Int. J. Hum. Comput. Stud. 61, 1, 32--70.
[40]
Greenbacker, C. F., McCoy, K. F., Carberry, S., and McDonald, D. D. 2011. Semantic modeling of multimodal documents for abstractive summarization. In Proceedings of the Canadian Workshop on Automatic Text Summarization. 29--40.
[41]
Grice, H. P. 1975. Logic and conversation. In Syntax and Semantics III: Speech Acts, P. Cole and J. L. Morgan, Eds. Academic Press, New York, 41--58.
[42]
Grosz, B. and Sidner, C. 1986. Attention, intentions, and the structure of discourse. Comput. Linguist. 12, 3, 175--204.
[43]
Ina, S. 1996. Computer graphics for the blind. SIGCAPH Comput. Physically Handicapp. 55, 16--23.
[44]
Jayant, C., Renzelmann, M., Wen, D., Krisnandi, S., Ladner, R., and Comden, D. 2007. Automated tactile graphics translation: in the field. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility. 75--82.
[45]
John, B. E. and Newell, A. 1990. Toward an engineering model of stimulus response compatibility. In Stimulus-Response Compatibility: An Integrated Approach, R. W. Gilmore and T. G. Reeve, Eds., North-Holland, New York, 107--115.
[46]
Joshi, A., Webber, B., and Weischedel, R. M. 1986. Living up to expectations: computing expert responses. In Proceedings of the Strategic Computing - Natural Language Workshop. 179--189.
[47]
Kendall, M. and Babington-Smith, B. 1939. The problem of m rankings. The Ann. Math. Statist. 10, 3, 275--287.
[48]
Kennel, A. R. 1996. Audiograf: A diagram-reader for the blind. In Proceedings of the Second Annual ACM Conference on Assistive Technologies. 51--56.
[49]
Krufka, S. E. and Barner, K. E. 2006. A user study on tactile graphic generation methods. Behav. Inf. Technol. 25, 4, 297--311.
[50]
Kurze, M. 1995. Giving blind people access to graphics (example: Business graphics). In Proceedings of the Software-Ergonomie Workshop on Nicht-Visuelle Graphische Benutzungsoberflächen (Nonvisual Graphical User Interfaces).
[51]
Larkin, J. and Simon, H. 1987. Why a diagram is (sometimes) worth ten thousand words. Cogn. Sci. 11, 65--99.
[52]
Lewandowsky, S. and Spence, I. 1989. The perception of statistical graphs. Sociol. Meth. Res. 18, 2-3, 200--242.
[53]
Liu, X. and Croft, W. B. 2002. Passage retrieval based on language models. In Proceedings of the 11th International Conference on Information and Knowledge Management. 375--382.
[54]
Lohse, G. L. 1993. A cognitive model for understanding graphical perception. Hum. Comput. Interac. 8, 353--388.
[55]
MacAulay, F., Judson, A., Etchels, M., Ashraf, S., Ricketts, I. W., Waller, A., Brodie, J. K., Alm, N., Warden, A., Shearer, A. J., and Gordon, B. 2002. ICU-Talk, a communication aid for intubated intensive care patients. In Proceedings of the 5th International ACM Conference on Assistive Technologies. 226--230.
[56]
McCoy, K. F., Carberry, M. S., Roper, T., and Green, N. 2001. Towards generating textual summaries of graphs. In Proceedings of the 1st International Conference on Universal Access in Human-Computer Interaction. 695--699.
[57]
McGookin, D. K. and Brewster, S. A. 2006. Soundbar: Exploiting multiple views in multimodal graph browsing. In Proceedings of the 4th Nordic Conference on Human-Computer Interaction. 145--154.
[58]
Meijer, P. B. 1992. An experimental system for auditory image representations. IEEE Trans. Biomed. Engin. 39, 2, 112--121.
[59]
Michaud, L. and McCoy, K. 2006. Capturing the evolution of gramatical knowledge in a CALL system for deaf learners of english. Int. J. Artif. Intell. Educ. 16, 1, 65--97.
[60]
Nayak, A. and Barner, K. 2004. Optimal halftoning for tactile imaging. IEEE Trans. Neural Syst. Rehabi. Engin. 12, 216--227.
[61]
Newell, A., Arnott, J., Booth, L., Beattie, W., Brophy, B., and Ricketts, I. 1992. Effect of the “PAL” word prediction system on the quality and quantity of text generation. Augment. Alternat. Comm. 8, 4, 304--311.
[62]
NSF Committee on Equal Opportunities in Science and Engineering. 2000. Enhancing the diversity of the science and engineering workforce to sustain america's leadership in the 21st century: Executive summary of the CEOSE report to congress, section 6. http://www.nsf.gov/pubs/2001/ceose2000rpt/start.htm.
[63]
O'Donnell, M., Mellish, C., Oberlander, J., and Knott, A. 2001. ILEX: an architecture for a dynamic hypertext generation system. Natural Lang. Engin. 7, 3, 225--250.
[64]
Pinker, S. 1990. A theory of graph comprehension. In Artificial Intelligence and the Future of Testing, R. Freedle, Ed. Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, 73--126.
[65]
Porter. 2006. Porter stemmer. http://tartarus.org/~martin/PorterStemmer/index.html.
[66]
Ramloll, R., Yu, W., Brewster, S., Riedel, B., Burton, M., and Dimigen, G. 2000. Constructing sonified haptic line graphs for the blind student: First steps. In Proceedings of the 4th International ACM Conference on Assistive Technologies. 17--25.
[67]
Rodgers, J. L. and Nicewander, W. A. 1988. Thirteen ways to look at the correlation coefficient. The Amer. Stat. 42, 1, 59--66.
[68]
Roth, P., Kamel, H., Petrucci, L., and Pun, T. 2002. A comparison of three nonvisual methods for presenting scientific graphs. J. Vis. Impair. Blind. 96, 6, 420--428.
[69]
Schwartz, E. 2007. A browser extension for providing visually impaired users access to the content of bar charts on the web. Honors Thesis - Millersville University.
[70]
Sinha, R. and Mihalcea, R. 2007. Unsupervised graph-based word sense disambiguation using measures of word semantic similarity. In Proceedings of the 1st IEEE International Conference on Semantic Computing. 363--369.
[71]
Tan, P.-N., Steinbach, M., and Kumar, V. 2005. Introduction to Data Mining. Addison Wesley.
[72]
Todman, J. and Alm, N. 2003. Modelling conversational pragmatics in communication aids. J. Pragmat. 35, 4, 523--538.
[73]
TreTagger 1995. Treetagger. http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger.
[74]
Trnka, K., McCaw, J., Yarrington, D., McCoy, K. F., and Pennington, C. 2009. User interaction with word prediction: The effects of prediction quality. ACM Transa. Access. Comput. 1, 17:1--17:34.
[75]
Tufte, E. 1983. The Visual Display of Quantitative Information. Graphics Press, Cheshire, CT.
[76]
Walker, B. N. and Nees, M. A. 2005. An agenda for research and development of multimodal graphs. In Proceedings of the 11th International Conference on Auditory Display. 428--432.
[77]
Walker, M., Rambow, O., and Rogati, M. 2002. Training a sentence planner for spoken dialogue using boosting. Comput. Speech Lang. 16, 3, 409--434.
[78]
Waller, A., Broumley, L., and Newell, A. 1992. Incorporating conversational narratives in an AAC device. Presented at International Symposium on Algorithms and Computation (ISAAC-92). Abstract appears in Augment. Alternat. Comm. 8.
[79]
Waller, A., O'Mara, D., Tait, L., Booth, L., Brophy-Arnott, B., and Hood, H. 2001. Using written stories to support the use of narrative in conversational interactions: Case study. Augment. Alternat. Comm. 17, 4, 221--232.
[80]
Way, T. P. and Barner, K. E. 1997a. Automatic visual to tactile translation--part I: Human factors, access methods and image manipulation. IEEE Trans. Rehab. Engin. 5, 1, 81--94.
[81]
Way, T. P. and Barner, K. E. 1997b. Automatic visual to tactile translation--part II: Evaluation of the TACTile image creation system. IEEE Trans. Rehab. Engin. 5, 1, 95--105.
[82]
Wickens, C. D. and Carswell, C. M. 1995. The proximity compatibility principle: Its psychological foundation and relevance to display design. Hum. Factors 37, 3, 473--494.
[83]
Witten, I. and Eibe Frank, M. A. H. 2011. Practical Machine Learning: Tools and Techniques, 3rd ed. Morgan Kaufmann.
[84]
Wu, P. 2012. Recognizing the intended message of line graphs: Methodology and applications. Ph.D. thesis, University of Delaware.
[85]
Wu, P., Carberry, S., and Elzer, S. 2010a. Segmenting line graphs into trends. In Proceedings of the 12th International Conference on Artificial Intelligence. 697--703.
[86]
Wu, P., Carberry, S., Elzer, S., and Chester, D. 2010b. Recognizing the intended message of line graphs. In Proceedings of the 6th International Conference on the Theory and Application of Diagrams. 220--234.
[87]
Yu, W. and Brewster, S. 2002. Comparing two haptic interfaces for multimodal graph rendering. In Proceedings of the 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 3--9.
[88]
Yu, W. and Brewster, S. 2003. Evaluation of multimodal graphs for blind people. Univer. Access Inf. Soc. 2, 2, 105--124.
[89]
Yu, W., Kangas, K., and Brewster, S. 2003. Web-based haptic applications for blind people to create virtual graphs. In Proceedings of the 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 318--325.
[90]
Zacks, J. and Tversky, B. 1999. Bars and lines: A study of graphic communication. Memory Cogn. 27, 6, 1073--1079.
[91]
Zhai, C. 2008. Statistical Language Models for Information Retrieval. Synthesis Lectures on Human Language Technologies Series, vol. 1. Morgan & Claypool, San Rafael, CA.

Cited By

View all
  • (2024)How Do Low-Vision Individuals Experience Information Visualization?Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642188(1-15)Online publication date: 11-May-2024
  • (2024)Beyond Vision Impairments: Redefining the Scope of Accessible Data RepresentationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.335656630:12(7619-7636)Online publication date: 1-Dec-2024
  • (2023)Comparing Natural Language and Vibro-Audio Modalities for Inclusive STEM Learning with Blind and Low Vision UsersProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608429(1-17)Online publication date: 22-Oct-2023
  • Show More Cited By

Index Terms

  1. Access to multimodal articles for individuals with sight impairments

                          Recommendations

                          Comments

                          Information & Contributors

                          Information

                          Published In

                          cover image ACM Transactions on Interactive Intelligent Systems
                          ACM Transactions on Interactive Intelligent Systems  Volume 2, Issue 4
                          Special issue on highlights of the decade in interactive intelligent systems
                          December 2012
                          205 pages
                          ISSN:2160-6455
                          EISSN:2160-6463
                          DOI:10.1145/2395123
                          Issue’s Table of Contents
                          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

                          Publisher

                          Association for Computing Machinery

                          New York, NY, United States

                          Publication History

                          Published: 01 January 2013
                          Accepted: 01 June 2012
                          Revised: 01 April 2012
                          Received: 01 June 2011
                          Published in TIIS Volume 2, Issue 4

                          Permissions

                          Request permissions for this article.

                          Check for updates

                          Author Tags

                          1. Human-computer interaction
                          2. accessibility
                          3. blind individuals
                          4. intelligent systems
                          5. multimodal

                          Qualifiers

                          • Research-article
                          • Research
                          • Refereed

                          Funding Sources

                          Contributors

                          Other Metrics

                          Bibliometrics & Citations

                          Bibliometrics

                          Article Metrics

                          • Downloads (Last 12 months)9
                          • Downloads (Last 6 weeks)1
                          Reflects downloads up to 17 Feb 2025

                          Other Metrics

                          Citations

                          Cited By

                          View all
                          • (2024)How Do Low-Vision Individuals Experience Information Visualization?Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642188(1-15)Online publication date: 11-May-2024
                          • (2024)Beyond Vision Impairments: Redefining the Scope of Accessible Data RepresentationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.335656630:12(7619-7636)Online publication date: 1-Dec-2024
                          • (2023)Comparing Natural Language and Vibro-Audio Modalities for Inclusive STEM Learning with Blind and Low Vision UsersProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608429(1-17)Online publication date: 22-Oct-2023
                          • (2023)SeeChart: Enabling Accessible Visualizations Through Interactive Natural Language Interface For People with Visual ImpairmentsProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584099(46-64)Online publication date: 27-Mar-2023
                          • (2022)Distributional Semantics of Line Charts for Trend ClassificationAdvances in Visual Computing10.1007/978-3-031-20716-7_20(259-269)Online publication date: 10-Dec-2022
                          • (2021)Information Graphic Summarization using a Collection of Multimodal Deep Neural Networks2020 25th International Conference on Pattern Recognition (ICPR)10.1109/ICPR48806.2021.9412146(10188-10195)Online publication date: 10-Jan-2021
                          • (2020)Generating Graph from 2D Flowchart using Region-Based Segmentation2020 IEEE International Students' Conference on Electrical,Electronics and Computer Science (SCEECS)10.1109/SCEECS48394.2020.165(1-5)Online publication date: Feb-2020
                          • (2019)An automated approach for the recognition of intended messages in grouped bar chartsComputational Intelligence10.1111/coin.1222735:4(955-1002)Online publication date: 23-Jun-2019
                          • (2019)Visualizing for the Non‐Visual: Enabling the Visually Impaired to Use VisualizationComputer Graphics Forum10.1111/cgf.1368638:3(249-260)Online publication date: 10-Jul-2019
                          • (2018)Survey and empirical comparison of different approaches for text extraction from scholarly figuresMultimedia Tools and Applications10.5555/3288251.328832577:22(29475-29505)Online publication date: 1-Nov-2018
                          • Show More Cited By

                          View Options

                          Login options

                          Full Access

                          View options

                          PDF

                          View or Download as a PDF file.

                          PDF

                          eReader

                          View online with eReader.

                          eReader

                          Figures

                          Tables

                          Media

                          Share

                          Share

                          Share this Publication link

                          Share on social media