Skip to main content
SpringerLink
Log in

Acceptance tests for validating ARIA requirements in widgets

  • Long paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

Accessibility refers to a quality requirement for web applications. However, current accessibility automatic evaluation tools cannot evaluate dynamic generated content that characterizes Ajax applications and RIAs. In this context, this paper describes an approach for evaluating Accessible Rich Internet Applications (ARIA) requirements, by using acceptance tests. The authors implemented a set of disabled user interaction scenarios as acceptance tests in order to verify keyboard accessibility in RIA and automatically evaluate ARIA conformance in widgets. The inclusion of disabled user interaction scenarios in the evaluation process is necessary to analyse ARIA requirements since dynamic changes are accommodated in the DOM structure. Two evaluation tool prototypes were developed and validated in separate case studies. The results show evidence that the proposed evaluation approach is capable of evaluating ARIA conformance in RIA widgets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. http://www.sidar.org/hera/index.php.en.

  2. http://www.dasilva.org.br/.

  3. http://bluefish.openoffice.nl/.

  4. http://www.adobe.com/products/dreamweaver/.

  5. https://github.com/heynemann/pyccuracy/wiki.

  6. http://web.archive.org/web/20130313020931/http://news.yahoo.com/.

  7. http://www.welie.com/patterns/showPattern.php?patternID=fly-out-menu.

  8. http://www.welie.com/patterns.

  9. https://github.com/watinha/Pyccuracy-Accessibility-Actions/tree/master/tests/pyccuracy_tests/fixtures/templates.

  10. http://www.w3.org/WAI/ER/tools/complete.

  11. http://github.com/watinha/aria-check.

  12. http://www.w3.org/TR/wai-aria/roles#tab.

  13. http://www.w3.org/TR/wai-aria/roles#tooltip.

  14. http://www.w3.org/TR/wai-aria/roles#alert.

  15. https://addons.mozilla.org/en-US/developers/docs/sdk/latest/.

  16. http://pivotal.github.com/jasmine/.

  17. http://test.cita.illinois.edu/aria/tabpanel/tabpanel1.php.

  18. http://www.welie.com/patterns/showPattern.php?patternID=tabbing.

  19. http://test.cita.illinois.edu/aria/.

  20. http://test.cita.illinois.edu/aria/tabpanel/tabpanel1.php.

  21. http://www.w3.org/TR/wai-aria/roles#tab.

  22. http://www.w3.org/TR/wai-aria/roles#tablist.

  23. http://www.w3.org/TR/wai-aria/roles#tabpanel.

  24. http://www.w3.org/TR/wai-aria/states_and_properties#aria-controls.

  25. http://www.w3.org/TR/wai-aria/states_and_properties#aria-hidden.

  26. http://www.w3.org/TR/wai-aria/states_and_properties.

  27. http://www.w3.org/TR/wai-aria/states_and_properties.

  28. http://test.cita.illinois.edu/aria/tabpanel/tabpanel1.php.

  29. http://jqueryui.com.

  30. http://jquery.com.

  31. http://yuilibrary.com.

  32. http://dojotoolkit.org.

  33. http://www.kendoui.com.

  34. http://developers.google.com/web-toolkit/.

  35. http://prototypejs.org.

  36. http://livepipe.net/.

  37. http://www.sencha.com/products/extjs.

  38. http://dhtmlx.com.

  39. http://jxlib.org.

  40. http://mootools.net.

References

  1. Araújo, B.C., Rocha, A.C., Xavier, A., Muniz, A.I., Garcia, F.P.: Web-based tool for automatic acceptance test execution and scripting for programmers and customers. In: Proceedings of the 2007 Euro American Conference on Telematics and Information Systems, EATIS ’07, vol. 1, pp. 56:1–56:4. ACM, New York, NY (2007). doi:https://doi.org/10.1145/1352694.1352752

  2. Borodin, Y., Bigham, J.P., Dausch, G., Ramakrishnan, I.V.: More than meets the eye: a survey of screen-reader browsing strategies. In: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A), W4A ’10, vol. 1, pp. 13:1–13:10. ACM, New York, NY, USA (2010). doi:https://doi.org/10.1145/1805986.1806005

  3. Brajnik, G.: Beyond conformance: the role of accessibility evaluation methods. In: Web Information Systems Engineering—WISE 2008 Workshops, Lecture Notes in Computer Science, vol. 5176, pp. 63–80. Springer Berlin/Heidelberg (2008). doi:https://doi.org/10.1007/978-3-540-85200-1_9. http://www.springerlink.com/content/p3l7030k62j65029/

  4. Brajnik, G., Yesilada, Y., Harper, S.: Testability and validity of WCAG 2.0: the expertise effect. In: Proceedings of the 12th international ACM SIGACCESS Conference on Computers and Accessibility, ASSETS ’10, vol. 1, pp. 43–50. ACM, New York, NY (2010). doi:https://doi.org/10.1145/1878803.1878813

  5. Cannizzo, F., Clutton, R., Ramesh, R.: Pushing the boundaries of testing and continuous integration. In: Proceedings of the Agile 2008, pp. 501–505. IEEE Computer Society, Washington, DC, USA (2008). doi:https://doi.org/10.1109/Agile.2008.31. http://dl.acm.org/citation.cfm?id=1443221.1443547

  6. Chen, A., Harper, S., Lunn, D., Brown, A.: Widget identification: A high-level approach to accessibility. World Wide Web 16(1), 73–89 (2013). doi:https://doi.org/10.1007/s11280-012-0156-6

    Article  Google Scholar 

  7. Cooper, M.: Accessibility of emerging rich web technologies: web 2.0 and the semantic web. In: Proceedings of the 2007 International Cross-Disciplinary Conference on Web Accessibility (W4A), W4A ’07, vol. 1, pp. 93–98. ACM, New York, NY (2007). doi:https://doi.org/10.1145/1243441.1243463

  8. Doush, I.A., Alkhateeb, F., Maghayreh, E.A., Al-Betar, M.A.: The design of RIA accessibility evaluation tool. Adv. Eng. Softw. 57, 1–7 (2013). doi:https://doi.org/10.1016/j.advengsoft.2012.11.004. http://www.sciencedirect.com/science/article/pii/S0965997812001512

    Article  Google Scholar 

  9. Fernandes, N., Batista, A.S., Costa, D., Duarte, C., Carriço, L.: Three web accessibility evaluation perspectives for RIA. In: Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, W4A ’13, vol. 1, pp. 12:1–12:9. ACM, New York, NY, USA (2013). doi:https://doi.org/10.1145/2461121.2461122

  10. Fernandes, N., Costa, D., Neves, S., Duarte, C., Carriço, L.: Evaluating the accessibility of rich internet applications. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A ’12, vol. 1, pp. 13:1–13:4. ACM, New York, NY, USA (2012). doi:https://doi.org/10.1145/2207016.2207019

  11. Fernandes, N., Lopes, R., Carriço, L.: On web accessibility evaluation environments. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A ’11, vol. 1, pp. 4:1–4:10. ACM, New York, NY, USA (2011). doi:https://doi.org/10.1145/1969289.1969295

  12. Fowler, M., Foemmel, M.: Continuous integration, http://www.martinfowler.com/articles/continuousIntegration.html (2005)

  13. Freire, A.P.: Disabled people and the web: user-based measurement of accessibility. Ph.D. thesis, University of York, Department of Computer Science, Submitted for the degree of Doctor of Philosophy, York, England (2012)

  14. Freire, A.P., Fortes, R.P.M., Turine, M.A.S., Paiva, D.M.B.: An evaluation of web accessibility metrics based on their attributes. In: SIGDOC ’08: Proceedings of the 26th Annual ACM International Conference on Design of Communication, vol. 1, pp. 73–80. ACM, New York, NY, USA (2008). doi:https://doi.org/10.1145/1456536.1456551

  15. Freire, A.P., Goularte, R., Fortes, R.P.M.: Techniques for developing more accessible web applications: a survey towards a process classification. In: SIGDOC ’07: Proceedings of the 25th Annual ACM International Conference on Design of Communication, vol. 1, pp. 162–169. ACM, New York, NY, USA (2007). doi:https://doi.org/10.1145/1297144.1297177

  16. Freire, A.P., Russo, C.M., Fortes, R.P.M.: A survey on the accessibility awareness of people involved in web development projects in Brazil. In: W4A ’08: Proceedings of the 2008 International Cross-Disciplinary Conference on Web Accessibility (W4A), vol. 1, pp. 87–96. ACM, New York, NY, USA (2008). doi:https://doi.org/10.1145/1368044.1368064

  17. Gehtland, J., Almaer, D., Galbraith, B.: Pragmatic Ajax: A Web 2.0 Primer. Pragmatic Bookshelf (2006)

  18. Gibson, B.: Enabling an accessible web 2.0. In: Proceedings of the 2007 International Cross-Disciplinary Conference on Web Accessibility (W4A), W4A ’07, vol. 1, pp. 1–6. ACM, New York, NY, USA (2007). doi:https://doi.org/10.1145/1243441.1243442

  19. Henry, S.L., Grossnickle, M.: Accessibility in the User–Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA (2004). http://www.uaiaccess.com/accessucd(On-line book)

  20. Hsia, P., Gao, J., Samuel, J., Kung, D., Toyoshima, Y., Chen, C.: Behavior-based acceptance testing of software systems: a formal scenario approach. In: Computer Software and Applications Conference, 1994. COMPSAC 94. Proceedings, Eighteenth Annual International, pp. 293 –298 (1994). doi:https://doi.org/10.1109/CMPSAC.1994.342789

  21. IEEE, I.o.E.a.E.E.: IEEE standard for software verification and validation plans. IEEE Std 1012-1986, pp. I (1986). doi:https://doi.org/10.1109/IEEESTD.1986.79647

  22. Kelly, B., Sloan, D., Brown, S., Seale, J., Petrie, H., Lauke, P., Ball, S.: Accessibility 2.0: people, policies and processes. In: W4A ’07: Proceedings of the 2007 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 138–147. ACM, New York, NY, USA (2007). doi:https://doi.org/10.1145/1243441.1243471

  23. Kelly, B., Sloan, D., Phipps, L., Petrie, H., Hamilton, F.: Forcing standardization or accommodating diversity? A framework for applying the wcag in the real world. In: W4A ’05: Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A), pp. 46–54. ACM, New York, NY, USA (2005). doi:https://doi.org/10.1145/1061811.1061820

  24. Lazar, J., Feng, J.H., Hochheiser, H.: Research Methods in Human–Computer Interaction. Wiley, New York (2010)

    Google Scholar 

  25. Leung, H.K.N., Wong, P.W.L.: A study of user acceptance tests. Softw Qual Control 6, 137–149 (1997). doi:https://doi.org/10.1023/A:1018503800709. http://dl.acm.org/citation.cfm?id=599113.599137

    Article  Google Scholar 

  26. Nielsen, J., Mack, R.L. (eds.): Usability Inspection Methods. Wiley, New York (1994)

    Google Scholar 

  27. North, D.: Introducing BDD (2006). http://dannorth.net/introducing-bdd/

  28. Reid, L.G., Snow-Weaver, A.: WCAG 2.0: a web accessibility standard for the evolving web. In: W4A ’08: Proceedings of the 2008 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 109–115. ACM, New York, NY, USA (2008). doi:https://doi.org/10.1145/1368044.1368069

  29. Stephanidis, C.: The Universal Access Handbook. CRC Press Inc, Boca Raton, FL, USA (2009)

    Book  Google Scholar 

  30. Tateishi, T., Miyashita, H., Naoshi, T., Saito, S., Ono, K.: DHTML accessibility checking based on static javascript analysis. In: Stephanidis, C. (ed.) Universal Access in Human–Computer Interaction. Applications and Services, Lecture Notes in Computer Science, vol. 4556, pp. 167–176. Springer, Berlin(2007). doi:https://doi.org/10.1007/978-3-540-73283-9_20. http://dx.doi.org/10.1007/978-3-540-73283-9_20

  31. Thiessen, P., Hockema, S.: WAI-ARIA live regions: eBuddy IM as a case example. In: Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A), W4A ’10, vol. 1, pp. 33:1–33:9. ACM, New York, NY (2010). doi:https://doi.org/10.1145/1805986.1806030

  32. Velasco, C.A., Denev, D., Stegemann, D., Mohamad, Y.: A web compliance engineering framework to support the development of accessible rich internet applications. In: W4A ’08: Proceedings of the 2008 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 45–49. ACM, New York, NY (2008). doi:https://doi.org/10.1145/1368044.1368054

  33. Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., Abascal, J.: Quantitative metrics for measuring web accessibility. In: W4A ’07: Proceedings of the 2007 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 99–107. ACM, New York, NY, USA (2007). doi:https://doi.org/10.1145/1243441.1243465

  34. W3C: Web content accessibility guidelines 1.0. W3C Recommendation (1999). http://www.w3.org/TR/WCAG10/

  35. W3C: Web content accessibility guidelines 2.0. W3C Recommendation (2008). http://www.w3.org/TR/WCAG20/

  36. W3C: WAI-ARIA 1.0 authoring practices—an author’s guide to understanding and implementing accessible rich internet applications. W3C Working Draft (2013). http://www.w3.org/TR/wai-aria-practices/

  37. W3C: Accessible rich internet applications—version 1.0. W3C Recommendation (2014). http://www.w3.org/TR/wai-aria/

  38. Watanabe, T.: Experimental evaluation of usability and accessibility of heading elements. In: W4A ’07: Proceedings of the 2007 International Cross-Disciplinary Conference on Web Accessibility (W4A), pp. 157–164. ACM, New York, NY (2007). doi:https://doi.org/10.1145/1243441.1243473

  39. Watanabe, W.M., Neto, D.F., Bittar, T.J., Fortes, R.P.M.: WCAG conformance approach based on model-driven development and WebML. In: Proceedings of the 28th ACM International Conference on Design of Communication, SIGDOC ’10, vol. 1, pp. 167–174. ACM, New York, NY (2010). doi:10.1145/1878450.1878479

Download references

Acknowledgments

The authors would like to thank the ICMC-USP and UTFPR for the assistance while developing this research and FAPESP for the financial support.

Author information

Authors and Affiliations

  1. Technological Federal University of the Paraná, Campus Cornélio Procópio, 1640, Alberto Carazzai Avenue, Cornélio Procópio, PR, 86300-000, Brazil

    Willian Massami Watanabe

  2. Institute of Mathematical and Computer Sciences, University of São Paulo, 400, Trabalhador São-carlense Avenue - Centro, P.O. Box 668, São Carlos, SP, 13560-970, Brazil

    Renata P. M. Fortes & Ana Luiza Dias

Authors
  1. Willian Massami Watanabe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Renata P. M. Fortes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ana Luiza Dias
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Willian Massami Watanabe.

Appendix: Tab panel acceptance test cases in aria-check

Appendix: Tab panel acceptance test cases in aria-check

  • Checks if the focus is changed to the next tab element as the down arrow key is pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches down arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the focus is moved to the next tab element.

  • Checks if the focus is changed to the next tab element as the right arrow key is pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches right arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the focus is moved to the next tab element.

  • Checks if the focus is changed to the previous tab element as the up arrow key is pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches up arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the focus is moved to the previous tab element.

  • Checks if the focus is changed to the previous tab element as the left arrow key is pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches left arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the focus is moved to the previous tab element.

  • Checks if the focus is changed to the first tab element as the control and home keys are pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches control and home key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the focus is moved to the first tab element.

    4. 4.

      Asserts if the tab panel element associated to the active tab element is visible, while all other tab panels are not visible (checking the HTML properties aria-hidden and style properties display and visibility).

  • Checks if the focus is changed to the last tab element as the control and end keys are pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches control and end key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the focus is moved to the last tab element.

    4. 4.

      Asserts if the tab panel element that is associated to the active tab element is visible, while all other tab panels are not visible (checking the HTML properties aria-hidden and style properties display and visibility).

  • Checks if the respective tab panel element is visible as the down arrow key is pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches down arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the tab panel element that is associated to the active tab element is visible, while all other tab panels are not visible (checking the HTML properties aria-hidden and style properties display and visibility).

  • Checks if the respective tab panel element is visible as the right arrow key is pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches right arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the tab panel element that is associated to the active tab element is visible, while all other tab panels are not visible (checking the HTML properties aria-hidden and style properties display and visibility).

  • Checks if the respective tab panel element is visible as the up arrow key is pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches up arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the tab panel element that is associated to the active tab element is visible, while all other tab panels are not visible (checking the HTML properties aria-hidden and style properties display and visibility).

  • Checks if the respective tab panel element is visible as the left arrow key is pressed in all tab elements:

    1. 1.

      Sets the focus to one tab element.

    2. 2.

      Dispatches left arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the tab panel element that is associated to the active tab element is visible, while all other tab panels are not visible (checking the HTML properties aria-hidden and style properties display and visibility).

  • Checks if the focus is set back to the tab element when the user is interacting with the tab panel element and when the user presses the control and up arrow keys:

    1. 1.

      Sets the focus to one tab panel element.

    2. 2.

      Dispatches the control and up arrow key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the focus is set back to the current active tab element.

  • Checks if the focus is set to the previous tab element when the user is interacting with the tab panel element and when the user presses the control and page up keys:

    1. 1.

      Sets the focus to one tab panel element.

    2. 2.

      Dispatches the control and page up key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the active tab is changed to the previous tab element and the focus is set to it.

  • Checks if the focus is set to the next tab element when the user is interacting with the tab panel element and when the user presses the control and page down keys:

    1. 1.

      Sets the focus to one tab panel element.

    2. 2.

      Dispatches the control and page down key events (keydown, keyup, keypress).

    3. 3.

      Asserts if the active tab is changed to the next tab element and the focus is set to it.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Watanabe, W.M., Fortes, R.P.M. & Dias, A.L. Acceptance tests for validating ARIA requirements in widgets. Univ Access Inf Soc 16, 3–27 (2017). https://doi.org/10.1007/s10209-015-0437-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-015-0437-9

Keywords

Search

Navigation