Skip to main content
SpringerLink
Log in

Web accessibility automatic evaluation tools: to what extent can they be automated?

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

Web accessibility automatic evaluation tools (WAET) are used to evaluate the conformance of the web content to the web content accessibility guidelines (WCAG) success criteria (SC). This paper aims to identify performance criteria that can be used to compare between automatic web accessibility evaluation tools (WAET), determine which SC can be automatically tested based on current technologies, which one requires more advanced technologies, and how can WAET reduce the number of mistakenly reported errors. WCAG 2.1 SC level-A, AA, and AAA are analyzed. The obtained results help in exploring new directions that can lead to more efficient and reliable automatic Web contents assessment as well as development tools. The outcome can help the developers of WAET to increase the number of SC checked in their tools by utilizing cutting-edge technologies. In addition, the presented performance indicators can help to identify how to measure the performance of WAET.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Notes

  1. https://www.w3.org/WAI/WCAG21/Techniques/general/G130.

  2. https://www.deque.com/.

References

  • Abascal, J., Arrue, M., Valencia, X.: Tools for web accessibility evaluation. In: Web Accessibility, pp. 479–503. Springer, London (2019). https://doi.org/10.1007/978-1-4471-7440-0_26

  • Abduganiev, S.G.: Towards automated web accessibility evaluation: a comparative study. Int. J. Inf. Technol. Comput. Sci. (IJITCS) 9(9), 18–44 (2017)

    Google Scholar 

  • Abou-Zahra, S.: A data model to facilitate the automation of web accessibility evaluations. Electron. Notes Theor. Comput. Sci. 157(2), 3–9 (2006)

    Article  Google Scholar 

  • Abu Doush, I., Pontelli, E., Son, T.C., Simon, D., Ma, O.: Multimodal presentation of two-dimensional charts: an investigation using open office xml and Microsoft Excel. ACM Trans. Access. Comput. (TACCESS) 3(2), 1–50 (2010)

    Article  Google Scholar 

  • Acosta, T., Acosta-Vargas, P., Lujan-Mora, S.: Accessibility of egovernment services in Latin America. In: 2018 International Conference on eDemocracy and eGovernment (ICEDEG), pp. 67–74. IEEE (2018)

  • Acosta-Vargas, P., Acosta, T., Lujan-Mora, S.: Framework for accessibility evaluation of hospital websites. In: 2018 International Conference on eDemocracy and eGovernment (ICEDEG), pp. 9–15. IEEE (2018)

  • Acosta-Vargas, P., Lujan-Mora, S., Salvador-Ullauri, L.: Evaluation of the web accessibility of higher-education websites. In: 2016 15th International Conference on Information Technology Based Higher Education and Training (ITHET), pp. 1–6. IEEE (2016)

  • Acosta-Vargas, P., Lujan-Mora, S., Salvador-Ullauri, L.: Quality evaluation of government websites. In: 2017 Fourth International Conference on eDemocracy and eGovernment (ICEDEG), pp. 8–14. IEEE (2017)

  • Acosta-Vargas, P., Rybarczyk, Y., Perez, J., Gonzalez, M., Jimenes, K., Leconte, L., Esparza, D.: Towards web accessibility in telerehabilitation platforms. In: 2018 IEEE Third Ecuador Technical Chapters Meeting (ETCM), pp. 1–6. IEEE (2018)

  • Acosta-Vargas, P., Acosta, T., Lujan-Mora, S.: Challenges to assess accessibility in higher education websites: a comparative study of Latin America universities. IEEE Access 6, 36500–36508 (2018)

    Article  Google Scholar 

  • Adepoju, S.A., Shehu, I.S., Bake, P.: Accessibility evaluation and performance analysis of e-government websites in Nigeria. J. Adv. Inf. Technol. 7(1), 49–53 (2016)

    Google Scholar 

  • Aftab, A., Ali, N.: Accessibility analysis of library websites of select central universities of India: an exploratory study. J. Knowl. Commun. Manag. 7(2), 193–200 (2017)

    Article  Google Scholar 

  • Ahmi, A., Mohamad, R.: Evaluating accessibility of Malaysian ministries websites using WCAG 2.0 and section 508 guideline. J. Telecommun. Electron. Comput. Eng. 8(8), 177–183 (2016)

    Google Scholar 

  • Aizpurua, A., Arrue, M., Vigo, M.: Prejudices, memories, expectations and confidence influence experienced accessibility on the web. Comput. Hum. Behav. 51, 152–160 (2015)

    Article  Google Scholar 

  • Aizpurua, A., Harper, S., Vigo, M.: Exploring the relationship between web accessibility and user experience. Int. J. Hum Comput. Stud. 91, 13–23 (2016)

    Article  Google Scholar 

  • Akgul, Y., Vatansever, K.: Web accessibility evaluation of government websites for people with disabilities in Turkey. J. Adv. Manag. Sci. 4(3), 201–210 (2016)

    Article  Google Scholar 

  • Akgul, Y.: Web content accessibility of municipal web sites in Turkey. In: European Conference on Digital Government, p. 1. Academic Conferences International Limited (2015)

  • Akgul, Y., et al.: Accessibility evaluation of moocs’ websites of Turkey. J. Life Econ. 5(4), 23–36 (2018)

    Article  Google Scholar 

  • Akgul, Y., et al.: Web accessibility of moocs for elderly students: the case of Turkey. J. Life Econ. 5(4), 141–150 (2018)

    Article  Google Scholar 

  • Al-Ahmad, A., Ahmaro, I., Mustafa, M.: Comparison between web accessibility evaluation tools. Al-Madinah Tech. Stud. 1(4), 1–12 (2015)

    Google Scholar 

  • Alahmadi, T., Drew, S.: An evaluation of the accessibility of top-ranking university websites: accessibility rates from 2005 to 2015. In: DEANZ Biennial Conference, pp. 224–233 (2016)

  • Alahmadi, T., Drew, S., et al.: Accessibility evaluation of top-ranking university websites in world, Oceania, and Arab categories for home, admission, and course description webpages. J. Open Flex. Distance Learn. 21(1), 7 (2017)

    Google Scholar 

  • Al-Khalifa, H.S., Baazeem, I., Alamer, R.: Revisiting the accessibility of Saudi Arabia government websites. Univ. Access Inf. Soc. 16(4), 1027–1039 (2017)

    Article  Google Scholar 

  • AlMeraj, Z., Boujarwah, F., Alhuwail, D., Qadri, R.: Evaluating the Accessibility of Higher Education Institution Websites in the State of Kuwait: Empirical Evidence, pp. 1–18. Universal Access in the Information Society, London (2020)

    Google Scholar 

  • Almourad, M.B., Hussein, M., Kamoun, F., Wattar, Z.: Analysis of WCAG 2.0 data accessibility success criterion of e-government websites. Period. Eng. Nat. Sci. 7(1), 496–503 (2019)

    Google Scholar 

  • Alsaeedi, A.: Comparing web accessibility evaluation tools and evaluating the accessibility of webpages: proposed frameworks. Information 11(1), 40 (2020)

    Article  Google Scholar 

  • Alshamari, M.: Accessibility evaluation of Arabic e-commerce web sites using automated tools. J. Softw. Eng. Appl. 9(9), 439–451 (2016)

    Article  Google Scholar 

  • Antonelli, H.L., Rodrigues, S.S., Watanabe, W.M., de Mattos Fortes, R.P.: A survey on accessibility awareness of Brazilian web developers. In: Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion, pp. 71–79 (2018)

  • Bai, A., Stray, V., Mork, H.: What methods software teams prefer when testing web accessibility. Adv. Hum. Comput. Interact. 2019, 1–14 (2019)

    Article  Google Scholar 

  • Ballantyne, M., Jha, A., Jacobsen, A., Hawker, J.S., El-Glaly, Y.N.: Study of accessibility guidelines of mobile applications. In: Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia, pp. 305–315 (2018)

  • Battle, L., Duan, P., Miranda, Z., Mukusheva, D., Chang, R., Stonebraker, M.: Beagle: Automated extraction and interpretation of visualizations from the web. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–8 (2018)

  • Baule, S.M.: Evaluating the accessibility of special education cooperative websites for individuals with disabilities. TechTrends 64(1), 50–56 (2020)

    Article  Google Scholar 

  • BBC: Accessibility standards and guidelines. http://www.bbc.co.uk/guidelines/futuremedia/accessibility/ (2018). Accessed 7 Apr 2022

  • Bokhove, C., Downey, C.: Automated generation of ‘good enough’ transcripts as a first step to transcription of audio-recorded data. Methodol. Innov. 11(2), 2059799118790743 (2018)

    Google Scholar 

  • Brajnik, G., Mulas, A., Pitton, C.: Effects of sampling methods on web accessibility evaluations. In: Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 59–66 (2007)

  • Brajnik, G.: A comparative test of web accessibility evaluation methods. In: Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 113–120 (2008)

  • Brajnik, G.: Barrier walkthrough. https://users.dimi.uniud.it/~giorgio.brajnik/projects/bw/bw.html (2009). Accessed 7 Mar 2022

  • Brajnik, G.: Beyond conformance: the role of accessibility evaluation methods. In: International Conference on Web Information Systems Engineering, pp. 63–80. Springer (2008)

  • Brajnik, G.: Comparing accessibility evaluation tools: a method for tool effectiveness. Univ. Access Inf. Soc. 3(3–4), 252–263 (2004)

    Article  Google Scholar 

  • Brajnik, G., Yesilada, Y., Harper, S.: The expertise effect on web accessibility evaluation methods. Hum. Comput. Interact. 26(3), 246–283 (2011)

    Google Scholar 

  • Butler, J., Trager, B., Behm, B.: Exploration of automatic speech recognition for deaf and hard of hearing students in higher education classes. In: The 21st International ACM SIGACCESS Conference on Computers and Accessibility. ASSETS ’19, pp. 32–42. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3308561.3353772. https://doi.org/10.1145/3308561.3353772

  • Calle-Jimenez, T., Sanchez-Gordon, S., Rybarczyk, Y., Jadan, J., Villarreal, S., Esparza, W., Acosta-Vargas, P., Guevara, C., Nunes, I.L.: Analysis and improvement of the web accessibility of a tele-rehabilitation platform for hip arthroplasty patients. In: International Conference on Applied Human Factors and Ergonomics, pp. 233–245. Springer (2018)

  • Campoverde-Molina, M., Valverde, L.: Accessibility analysis of the web portals of the educational institutions in Cuenca, Ecuador. CATEDRA. Revista especializada en estudios culturales y humanisticos, 2019 2(2), 53–72 (2020)

    Google Scholar 

  • Carvajal, C.M., Piqueras, R.F., Merida, J.F.C.: Evaluation of web accessibility of higher education institutions in Chile. Int. Educ. Stud. 11(12), 140–148 (2018)

    Article  Google Scholar 

  • Collins-Thompson, K.: Computational assessment of text readability: a survey of current and future research. ITL Int. J. Appl. Linguist. 165(2), 97–135 (2014)

    Article  Google Scholar 

  • Commission, D.R.: The Web: Access and Inclusion for Disabled People; A Formal Investigation. The Stationery Office, London (2004)

    Google Scholar 

  • Costa, D., Carrico, L., Duarte, C.: The differences in accessibility of tv and desktop web applications from the perspective of automated evaluation. Procedia Comput. Sci. 67, 388–396 (2015)

    Article  Google Scholar 

  • Csontos, B., Heckl, I.: Accessibility, Usability, and Security Evaluation of Hungarian Government Websites. Universal Access in the Information Society, pp. 1–18 (2020)

  • CTIC, F.: TAW. https://www.tawdis.net (2021). Accessed 4 Apr 2022

  • De Araujo, T.M.U., Ferreira, F.L., Silva, D.A., Oliveira, L.D., Falcao, E.L., Domingues, L.A., Martins, V.F., Portela, I.A., Nobrega, Y.S., Lima, H.R., et al.: An approach to generate and embed sign language video tracks into multimedia contents. Inf. Sci. 281, 762–780 (2014)

    Article  Google Scholar 

  • Deedam, F.B., Thomas, E., Taylor, O.E.: Accessibility and usability evaluation of state-owned universities website in Nigeria. Int. J. Eng. Trends Technol. 56(1), 31–36 (2018)

    Article  Google Scholar 

  • Dettenborn, R., de Brito, G.L.R.: Automated validation of accessibility of Brazilian judiciary agencies websites. J. Adv. Inf. Technol. 10(2), 54–59 (2019)

    Google Scholar 

  • Dominguez Vila, T., Alen Gonzalez, E., Darcy, S.: Website accessibility in the tourism industry: an analysis of official national tourism organization websites around the world. Disabil. Rehabil. 40(24), 2895–2906 (2018)

    Article  Google Scholar 

  • Doush, I.A., Al-Jarrah, A., Alajarmeh, N., Alnfiai, M.: Learning Features and Accessibility Limitations Of Video Conferencing Applications: Are People with Visual Impairment Left Behind. Universal Access in the Information Society, pp. 1–16 (2022)

  • Doush, I.A., AlMeraj, Z.: Evaluating the accessibility of Kuwaiti e-government websites. Jordan. J. Comput. Inf. Technol. (JJCIT). 5(03), 152–169 (2019)

  • Doush, I.A., Damaj, I., Al-Betar, M.A., Awadallah, M.A., Ra’ed, M., Alchalabi, A.E., Bolaji, A.L.: A survey on accessible context-aware systems. In: Technological Trends in Improved Mobility of the Visually Impaired, pp. 29–63. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-16450-8_2

  • Doush, I.A., Alhami, I.: Evaluating the accessibility of computer laboratories, libraries, and websites in Jordanian universities and colleges. Int. J. Inf. Syst. Soc. Change (IJISSC) 9(2), 44–60 (2018)

    Article  Google Scholar 

  • Doush, I.A., Alkhateeb, F., Al Maghayreh, E., Al-Betar, M.A.: The design of ria accessibility evaluation tool. Adv. Eng. Softw. 57, 1–7 (2013)

    Article  Google Scholar 

  • Doush, I.A., Bany-Mohammed, A., Ali, E., Al-Betar, M.A.: Towards a more accessible e-government in Jordan: an evaluation study of visually impaired users and web developers. Behav. Inf. Technol. 32(3), 273–293 (2013)

    Article  Google Scholar 

  • Dror, A.A., Layous, E., Mizrachi, M., Daoud, A., Eisenbach, N., Morozov, N.G., Srouji, S., Avraham, K.B., Sela, E.: A lesson from covid-19 on inaccessibility of web-based information for disabled populations worldwide. bioRxiv (2020)

  • Duarte, C., Salvado, A., Akpinar, M.E., Yesilada, Y., Carrico, L.: Automatic role detection of visual elements of web pages for automatic accessibility evaluation. In: Proceedings of the Internet of Accessible Things, pp. 1–4. ACM, France (2018). https://doi.org/10.1145/3192714.3196827

  • Egorov, M., Funkner, A.: Automatic extraction and decryption of abbreviations from domain-specific texts. Stud. Health Technol. Inform. 285, 281–284 (2021)

    Google Scholar 

  • Elkabani, I., Hamandi, L., Zantout, R., Mansi, S.: Toward better web accessibility. In: 2015 5th International Conference on Information and Communication Technology and Accessibility (ICTA), pp. 1–6. IEEE (2015)

  • ElKabani, I., Zantout, R., Hamandi, L., Mansi, S.: Improving web accessibility. Int. J. Appl. Eng. Res. 11(12), 77836–7845 (2016)

    Google Scholar 

  • ElRahman, S.A.: Evaluation of Saudi educational websites. Int. J. ELearn. Educ. Technol. Digit. Media (IJEETDM) 2(4), 141–147 (2016)

    Google Scholar 

  • Emam, S.S., Miller, J.: Automated testing of motion-based events in mobile applications. In: HICSS (2019)

  • EvalAccess: EvalAccess2.0. http://sipt07.si.ehu.es/evalaccess2/howto.html (2021). Accessed 4 Apr 2022

  • Fernandes, N., Costa, D., Duarte, C., Carriço, L.: Evaluating the accessibility of web applications. Procedia Comput. Sci. 14, 28–35 (2012)

    Article  Google Scholar 

  • Fu, J., Rui, Y.: Advances in deep learning approaches for image tagging. APSIPA Trans. Signal Inf. Process. 6, 1–13 (2017)

    Article  Google Scholar 

  • Garcia-Santiago, L., Olvera-Lobo, M.-D.: How Accessibility Guidelines are Used in Spanish World Heritage Websites: An Exploratory Study. Library Hi Tech, Delhi (2020)

    Google Scholar 

  • Gary Kacmarcik, D.S. Travis Leithead: W3C working draft. https://www.w3.org/TR/uievents/ (2013). Accessed 7 Apr 2022

  • Gay, G., Li, C.Q.: Achecker: open, interactive, customizable, web accessibility checking. In: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A), pp. 1–2 (2010)

  • Géry, M., Chevallet, J.-P.: Toward a structured information retrieval system on the web: automatic structure extraction of web pages. In: International Workshop on Web Dynamics (2001)

  • Gil, A., Postal, J., Ferreira, A., Gonçalves, D., Bianco, B.H., Gadelha, M.R.: Automatic contrast evaluation for android themes. In: International Conference on Human-Computer Interaction, pp. 267–278. Springer (2020)

  • Gilbertson, T.D., Machin, C.H.: Guidelines, icons and marketable skills: an accessibility evaluation of 100 web development company homepages. In: Proceedings of the International Cross-disciplinary Conference on Web Accessibility, pp. 1–4 (2012)

  • Graham, Y., Awad, G., Smeaton, A.: Evaluation of automatic video captioning using direct assessment. PLoS One 13(9), 0202789 (2018)

    Article  Google Scholar 

  • Grigera, J., Garrido, A., Rivero, J.M., Rossi, G.: Automatic detection of usability smells in web applications. Int. J. Hum Comput Stud. 97, 129–148 (2017)

    Article  Google Scholar 

  • Gutierrez, M.M., Caceres, J.R.R.: Evaluating the effectiveness of accessible web sites for deaf users. In: 2019 International Conference on Inclusive Technologies and Education (CONTIE), pp. 129–1295. IEEE (2019)

  • Halbach, T., Wideroe, T., Bai, A.: Raising awareness for universal design by integrating accessibility testing into a continuous-deployment process: a large-company case study. In: Norsk Konferanse for Organisasjoners Bruk at IT (2019)

  • Hammad, M., Alnabhan, M., Doush, I.A., Alsalem, G.M., Al-Alem, F.A., Al-Awadi, M.M.: Evaluating usability and content accessibility for e-learning websites in the middle east. Int. J. Technol. Hum. Interact. (IJTHI) 16(1), 54–62 (2020)

    Article  Google Scholar 

  • IBM: IBM accessibility list. https://www.ibm.com/able/guidelines/ci162/accessibility_checklist.html (2018). Accessed 7 Apr 2022

  • In Mind, W.W.A.: Accessibility evaluation tools. https://webaim.org/articles/tools/#classification (2022). Accessed 29 Nov 2022

  • Inal, Y., Rizvanoglu, K., Yesilada, Y.: Web accessibility in turkey: awareness, understanding and practices of user experience professionals. Univ. Access Inf. Soc. 18(2), 387–398 (2019)

    Article  Google Scholar 

  • Iseri, E.I., Uyar, K., Ilhan, U.: The accessibility of Cyprus Islands’ higher education institution websites. Procedia Comput. Sci. 120, 967–974 (2017)

    Article  Google Scholar 

  • Ismail, A., Kuppusamy, K.: Accessibility analysis of north eastern India region websites for persons with disabilities. In: 2016 International Conference on Accessibility to Digital World (ICADW), pp. 145–148. IEEE (2016)

  • Ismail, A., Kuppusamy, K.: Web accessibility investigation and identification of major issues of higher education websites with statistical measures: a case study of college websites. J. King Saud Univ. Comput. Inf. Sci. 34, 901–911 (2019)

    Google Scholar 

  • Ismail, A., Kuppusamy, K.: Accessibility of Indian universities’ homepages: an exploratory study. J. King Saud Univ. Comput. Inf. Sci. 30(2), 268–278 (2018)

    Google Scholar 

  • Ismail, A., Kuppusamy, K., Nengroo, A.S.: Multi-tool accessibility assessment of government department websites: a case-study with JKGAD. Disabil. Rehabil. Assist. Technol. 13(6), 504–516 (2018)

    Article  Google Scholar 

  • Ismail, A., Kuppusamy, K., Kumar, A., Ojha, P.K.: Connect the dots: accessibility, readability and site ranking-an investigation with reference to top ranked websites of government of India. J. King Saud Univ. Comput. Inf. Sci. 31(4), 528–540 (2019)

    Google Scholar 

  • Kafle, S., Huenerfauth, M.: Usability evaluation of captions for people who are deaf or hard of hearing. ACM SIGACCESS Access. Comput. 1(122), 1–1 (2020)

    Article  Google Scholar 

  • Katiyar, A., Srividya, V., Tripathy, B.: Tagit: A system for image auto-tagging and clustering. In: Data Engineering and Intelligent Computing, pp. 259–268. Springer, Singapore (2021). https://doi.org/10.1007/978-981-16-0171-2_25

  • Kaur, A., Dani, D.: Comparing and evaluating the effectiveness of mobile web adequacy evaluation tools. Univ. Access Inf. Soc. 16(2), 411–424 (2017)

    Article  Google Scholar 

  • Khandkar, S.H., Sohan, S., Sillito, J., Maurer, F.: Tool support for testing complex multi-touch gestures. In: ACM International Conference on Interactive Tabletops and Surfaces, pp. 59–68 (2010)

  • Konkol, P.: A comparison of web accessibility evaluation tools. Inf. Technol. Pract. 2016, 211 (2016)

    Google Scholar 

  • Kontokostas, D., Westphal, P., Auer, S., Hellmann, S., Lehmann, J., Cornelissen, R., Zaveri, A.: Test-driven evaluation of linked data quality. In: Proceedings of the 23rd International Conference on World Wide Web, pp. 747–758 (2014)

  • Krol, K., Zdonek, D.: Local government website accessibility-evidence from Poland. Adm. Sci. 10(2), 22 (2020)

    Article  Google Scholar 

  • Kuckartz, U., Radiker, S.: Transcribing audio and video recordings. In: Analyzing Qualitative Data with MAXQDA, pp. 41–49. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15671-8_4

  • Kulkarni, G., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A., Berg, T.: Baby talk: understanding and generating image descriptions. In: CVPR 2011 (2011)

  • Kurt, S.: Moving toward a universally accessible web: web accessibility and education. Assist. Technol. 4, 199–208 (2018)

    Google Scholar 

  • Kurt, S.: Accessibility of Turkish university web sites. Univ. Access Inf. Soc. 16(2), 505–515 (2017)

    Article  Google Scholar 

  • Kuzma, J., Law, J., Bell, V., Williams, N.: A study of global hospital websites for accessibility compliance. Eur. J. Bus. Soc. Sci. 6(6), 15–28 (2017)

    Google Scholar 

  • Lee, S.C., Li, B., Starner, T.: Airtouch: synchronizing in-air hand gesture and on-body tactile feedback to augment mobile gesture interaction. In: 2011 15th Annual International Symposium on Wearable Computers, pp. 3–10. IEEE (2011)

  • Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  • Lopez, C., Prince, V., Roche, M.: Automatic titling of electronic documents with noun phrase extraction. In: 2010 International Conference of Soft Computing and Pattern Recognition, pp. 168–171 (2010). IEEE

  • Lui, M., Lau, J.H., Baldwin, T.: Automatic detection and language identification of multilingual documents. Trans. Assoc. Comput. Linguist. 2, 27–40 (2014)

    Article  Google Scholar 

  • Maez-Carvajal, C., Cervera-Merida, J.F., Fernandez-Piqueras, R.: Web Accessibility Evaluation of Top-ranking University Web Sites in Spain, Chile and Mexico, pp. 1–6. Universal Access in the Information Society, London (2019)

    Google Scholar 

  • Manez-Carvajal, C., Fernandez-Piqueras, R., Cervera-Merida, J.F., et al.: Evaluating web accessibility of Spanish universities. J. Eng. Appl. Sci. 14(14), 4876–4881 (2019)

    Article  Google Scholar 

  • Martins, J., Goncalves, R., Branco, F.: A full scope web accessibility evaluation procedure proposal based on Iberian ehealth accessibility compliance. Comput. Hum. Behav. 73, 676–684 (2017)

    Article  Google Scholar 

  • Mishra, P., Diwan, C., Srinivasa, S., Srinivasaraghavan, G.: Automatic title generation for text with pre-trained transformer language model. In: 2021 IEEE 15th International Conference on Semantic Computing (ICSC), pp. 17–24. IEEE (2021)

  • Morillo, P., Chicaiza-Herrera, D., Vallejo-Huanga, D.: System of recommendation and automatic correction of web accessibility using artificial intelligence. In: International Conference on Applied Human Factors and Ergonomics, pp. 479–489. Springer (2019)

  • Mozilla: Managing screen orientation. https://developer.mozilla.org/en-US/docs/Web/API/CSS_Object_Model/Managing_screen_orientation (2022). Accessed 27 Mar 2022

  • Nakatumba-Nabende, J., Kanagwa, B., Kivunike, F.N., Tuape, M.: Evaluation of accessibility standards on Ugandan e-government websites. Electron. Gov. Int. J. 15(4), 355–371 (2019)

    Google Scholar 

  • Navarrete, R., Lujan-Mora, S.: Bridging the accessibility gap in open educational resources. Univ. Access Inf. Soc. 17(4), 755–774 (2018)

    Article  Google Scholar 

  • Nazar, H., Sarfraz, M.S., Shoaib, U.: Web accessibility evaluation of banking website in Pakistan. Int. J. Comput. Sci. Inf. Secur. 15(1), 642 (2017)

    Google Scholar 

  • Ochoa, R.L., Crovi, D.M.: Evaluation of accessibility in Mexican cybermedia. Univ. Access Inf. Soc. 18(2), 413–422 (2019)

    Article  Google Scholar 

  • Petersen, S.E., Ostendorf, M.: A machine learning approach to reading level assessment. Comput. Speech Lang. 23(1), 89–106 (2009)

    Article  Google Scholar 

  • Pirsa, A., Stanic, B., Stracak, L., Todorovic, Z., Vdovic, H., Zilak, M., Vukovic, M., Car, Z.: Front-end solution for enhancing web sites accessibility. In: 2015 13th International Conference on Telecommunications (ConTEL), pp. 1–8. IEEE (2015)

  • Pribeanu, C., Gheorghe-Moisii, M., Fogarassy-Neszly, P.: Accessibility of Romanian municipal websites-conformance with wcag2. In: RoCHI, pp. 25–28 (2015)

  • Pribeanu, C.: Large-scale accessibility evaluation of Romanian municipal websites. Int. J. User-Syst. Interact. 12(2), 83–98 (2019)

    Google Scholar 

  • Putra, J.W.G., Khodra, M.L.: Automatic title generation in scientific articles for authorship assistance: a summarization approach. J. ICT Res. Appl. 11(3), 253–267 (2017)

    Article  Google Scholar 

  • Rahmatizadeh, S., Valizadeh-Haghi, S.V.-H.: Monitoring for accessibility in medical university websites: meeting the needs of people with disabilities. J. Access. Des. All 8(2), 102–124 (2018)

    Google Scholar 

  • Reis, A., Martins, P., Borges, J., Sousa, A., Rocha, T., Barroso, J.: Supporting accessibility in higher education information systems: a 2016 update. In: International Conference on Universal Access in Human–Computer Interaction, pp. 227–237. Springer (2017)

  • Rodriguez Vazquez, S.: Measuring the impact of automated evaluation tools on alternative text quality: a web translation study. In: Proceedings of the 13th Web for All Conference, pp. 1–10 (2016)

  • Rodriguez Vazquez, S.: Unlocking the potential of web localizers as contributors to image accessibility: what do evaluation tools have to offer? In: Proceedings of the 12th Web for All (W4A) Conference. ACM (2015)

  • Rodriguez, G., Perez, J., Cueva, S., Torres, R.: A framework for improving web accessibility and usability of open course ware sites. Comput. Educ. 109, 197–215 (2017)

    Article  Google Scholar 

  • Ruan, S., Wobbrock, J.O., Liou, K., Ng, A., Landay, J.A.: Comparing speech and keyboard text entry for short messages in two languages on touchscreen phones. Proc. ACM Interact. Mobile Wearable Ubiquitous Technol. 1(4), 1–23 (2018)

    Article  Google Scholar 

  • Sánchez, D., Isern, D., Millan, M.: Content annotation for the semantic web: an automatic web-based approach. Knowl. Inf. Syst. 27(3), 393–418 (2011)

    Article  Google Scholar 

  • Sanchez-Gordon, S., Lujan-Mora, S., Sanchez-Gordon, M.: E-government accessibility in Ecuador: a preliminary evaluation. In: 2020 Seventh International Conference on eDemocracy and eGovernment (ICEDEG), pp. 50–57. IEEE (2020)

  • Sefara, T.J., Manamela, M.J., Modipa, T.I.: Web-based automatic pronunciation assistant. In: Southern Africa Telecommunication Networks and Applications Conference (SATNAC), pp. 112–117 (2017)

  • Silva, C., Eler, M.M., Fraser, G.: A survey on the tool support for the automatic evaluation of mobile accessibility. In: Proceedings of the 8th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-Exclusion, pp. 286–293 (2018)

  • Sohaib, O., Kang, K.: E-commerce web accessibility for people with disabilities. In: Complexity in Information Systems Development, pp. 87–100. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52593-8_6

  • Tashtoush, Y.M., Ala’F, D., Al-Sarhan, H.N.: The Arabian e-government websites accessibility: a case study. In: 2016 7th International Conference on Information and Communication Systems (ICICS), pp. 276–281. IEEE (2016)

  • Taylor, Z., Burnett, C.A.: Hispanic-serving institutions and web accessibility: Digital equity for Hispanic students with disabilities in the 21st century. J. Hisp. High. Educ. 20, 402–421 (2019)

    Article  Google Scholar 

  • Teixeira, L., Eusebio, C., Silveiro, A.: Website accessibility of Portuguese travel agents. In: 2019 14th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6. IEEE (2019)

  • Temko, A., Macho, D., Nadeu, C.: Fuzzy integral based information fusion for classification of highly confusable non-speech sounds. Pattern Recogn. 41(5), 1814–1823 (2008)

    Article  MATH  Google Scholar 

  • Timbi-Sisalima, C., Amor, C.I.M., Oton, S., Hilera, J.R., Aguado-Delgado, J.: Comparative analysis of online web accessibility evaluation tools. In: Information Systems Development: Complexity in Information Systems Development (ISD2016 Proceedings) (2016)

  • Uyar, K., Delele, G.M., Iseri, E.I.: Web accessibility of the federal democratic republic of Ethiopia governmental websites. IJCSNS 20(6), 144 (2020)

    Google Scholar 

  • Vajjala, S., Lučić, I.: Onestopenglish corpus: a new corpus for automatic readability assessment and text simplification. In: Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 297–304 (2018)

  • Vazquez, S.R., Lehmann, S.: Acrolinx: a controlled-language checker turned into an accessibility evaluation tool for image text alternatives. In: Proceedings of the 12th Web for All Conference, pp. 1–2 (2015)

  • Vazquez, S.R.: A controlled language-based evaluation approach to ensure image accessibility during web localisation. Transl. Spaces 4(2), 187–215 (2015)

    Article  Google Scholar 

  • Velleman, E., Abou-Zahra, S.: Website accessibility conformance evaluation methodology (WCAG-EM) 1.0. https://www.w3.org/TR/WCAG-EM/#step3 (2014). Accessed 17 Apr 2022

  • Velleman, E., Velasco, C., Snaprud, M., Burger, D.: D-wab4 unified web evaluation methodology (uwem 1.0). Technical report, WAB Cluster (2006)

  • Verkijika, S.F., De Wet, L.: Accessibility of South African university websites. Univ. Access Inf. Soc. 19(1), 201–210 (2020)

    Article  Google Scholar 

  • Vigo, M., Brown, J., Conway, V.: Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. In: Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, pp. 1–10 (2013)

  • Vigo, M., Brajnik, G.: Automatic web accessibility metrics: where we are and where we can go. Interact. Comput. 23(2), 137–155 (2011)

    Article  Google Scholar 

  • Vila, T.D., Gonzalez, E.A., Darcy, S.: Accessibility of tourism websites: the level of countries’ commitment. Univ. Access Inf. Soc. 19(2), 331–346 (2020)

    Article  Google Scholar 

  • W3C/WAI: WCAG-EM overview: website accessibility conformance evaluation methodology. https://www.w3.org/WAI/eval/conformance.html#scope (2005). Accessed 7 Apr 2022

  • W3C: Controlling foreground and background colors. https://www.w3.org/WAI/WCAG21/Techniques/failures/F24 (2022). Accessed 25 Apr 2022

  • W3C: Forms: using aria-invalid to identify failed fields. https://www.w3.org/WAI/WCAG21/working-examples/aria-invalid-data-format/ (2022). Accessed 14 Apr 2022

  • W3C: providing a style switcher to switch to high contrast. https://www.w3.org/WAI/WCAG21/Techniques/silverlight/SL13 (2022). Accessed 27 Mar 2022

  • W3C: Providing submit buttons. https://www.w3.org/WAI/WCAG21/Techniques/html/H32 (2022). Accessed 14 Apr 2022

  • W3C: Resizing text. https://www.w3.org/WAI/WCAG21/Techniques/silverlight/SL23 (2022). Accessed 25 Mar 2022

  • W3C: Sampling of pages. https://www.w3.org/WAI/ER/conformance/ED-methodology-20120222 (2012). Accessed 21 Apr 2022

  • W3C: Understanding success criterion 1.1.1: non-text content. https://www.w3.org/WAI/WCAG21/Understanding/non-text-content.html#examples (2022). Accessed 25 Mar 2022

  • W3C: Understanding success criterion 1.4.11: non-text contrast. https://www.w3.org/WAI/WCAG21/Understanding/non-text-contrast.html (2021). Accessed 4 Apr 2022

  • W3C: Using a button with a select element to perform an action. https://www.w3.org/WAI/WCAG21/Techniques/html/H84 (2022). Accessed 14 Apr 2022

  • W3C: Using HTML 5.2 autocomplete attributes. https://www.w3.org/WAI/WCAG21/Techniques/html/H98 (2022). Accessed 27 Mar 2022

  • W3C: Using the language attribute on the HTML element. https://www.w3.org/WAI/WCAG21/Techniques/html/H57 (2022). Accessed 14 Apr 2022

  • W3C: Validate multiple controls. https://www.w3.org/WAI/WCAG21/working-examples/script-check-multiple-controls/ (2022). Accessed 14 Apr 2022

  • W3C: W3C: accessibility fundemntals- video captions. https://www.w3.org/WAI/perspective-videos/captions/ (2022). Accessed 1 Mar 2022

  • W3C: Web accessibility evaluation tools list. https://www.w3.org/WAI/ER/tools/ (2022). Accessed 1 Mar 2022

  • WC3: Research report on web accessibility metrics. https://www.w3.org/WAI/RD/2011/metrics/note/ED-metrics (2014). Accessed 7 Apr 2021

  • WebAIM: WAVE. http://wave.webaim.org/ (2021). Accessed 4 Apr 2022

  • WebsiteBuilder: The ultimate list of Internet statistics. https://websitebuilder.org/blog/internet-statistics/ (2021). Accessed 7 Apr 2022

  • WHO: The world health organization (WHO). http://www.who.int/disabilities/world_report/2011/en/ (2022). Accessed 1 Mar 2022

  • Yang, G., Huang, J.: Automated generation of event-oriented exploits in android hybrid apps. In: Proceedings of the Network and Distributed System Security Symposium (NDSS’18) (2018)

  • Yesilada, Y., Brajnik, G., Vigo, M., Harper, S.: Exploring perceptions of web accessibility: a survey approach. Behav. Inf. Technol. 34(2), 119–134 (2015)

    Article  Google Scholar 

  • Yoon, K., Dols, R., Hulscher, L., Newberry, T.: An exploratory study of library website accessibility for visually impaired users. Libr. Inf. Sci. Res. 38(3), 250–258 (2016)

    Article  Google Scholar 

  • Zeleke, Y.: Usability and accessibility model for e-government websites in Ethiopia. In: THE 6TH ANNUAL African Conference on Information Systems and Technology (2020)

  • Zhang, M., Wang, C., Bu, J., Yu, Z., Lu, Y., Zhang, R., Chen, C.: An optimal sampling method for web accessibility quantitative metric. In: Proceedings of the 12th Web for All Conference, pp. 1–4 (2015)

  • Ziman, K., Heusser, A.C., Fitzpatrick, P.C., Field, C.E., Manning, J.R.: Is automatic speech-to-text transcription ready for use in psychological experiments? Behav. Res. Methods 50(6), 2597–2605 (2018)

    Article  Google Scholar 

  • Zitkus, E., Brigatto, A.C., Ferrari, A.L.M., Bonfim, G.H., Carvalho Filho, I.F., Reis, T.D., Medola, F.O., Paschoarelli, L.C.: Accessibility and usability of websites intended for people with disabilities: a preliminary study. In: International Conference of Design, User Experience, and Usability, pp. 678–688 (2016). Springer

Download references

Author information

Authors and Affiliations

  1. College of Engineering and Applied Sciences, American University of Kuwait, Salmiya, Kuwait

    Iyad Abu Doush & Khalid Sultan

  2. Computer Science Department, Yarmouk University, Irbid, Jordan

    Iyad Abu Doush

  3. Department of Information Technology, Ajman University, Ajman, UAE

    Mohammed Azmi Al-Betar

  4. Information Science Department, Kuwait University, Adailiya, Kuwait

    Zainab Almeraj

  5. ECE Department, University of Kufa, Najaf, Iraq

    Zaid Abdi Alkareem Alyasseri

  6. Department of Computer Science, Al-Aqsa University, Gaza, Palestine

    Mohammed A. Awadallah

Authors
  1. Iyad Abu Doush
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khalid Sultan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammed Azmi Al-Betar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zainab Almeraj
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zaid Abdi Alkareem Alyasseri
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mohammed A. Awadallah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Iyad Abu Doush.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this research.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abu Doush, I., Sultan, K., Al-Betar, M.A. et al. Web accessibility automatic evaluation tools: to what extent can they be automated?. CCF Trans. Pervasive Comp. Interact. 5, 288–320 (2023). https://doi.org/10.1007/s42486-023-00127-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42486-023-00127-8

Keywords

Search

Navigation