Skip to main content
SpringerLink
Log in

Comparing and evaluating the effectiveness of mobile Web adequacy evaluation tools

  • Long paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

The number of Web sites is growing exponentially and so are the people who are accessing them on mobile devices including people with special abilities. Mobile-friendly Web sites further increase traffic to the Web site, customer participation, and boost e-Commerce. In such a scenario, it is important to ensure that the Web sites are accessible on mobile devices for all users. Automatic tools provide an easy and a less error-prone mechanism for evaluating the mobile adequacy of these Web sites. The present study focuses primarily on the comparison between four tools of mobile Web adequacy evaluation, namely mobileOK checker, taw, EvalAcess Mobile, and mobiReady. The comparison is made based on three parameters, i.e., correctness, completeness, coverage with respect to the conformance of Mobile Web Best Practices (MWBP), and four groups of mobile Web best practice guidelines, i.e., Navigability and Links, Page Layout and Content, Page Definition, and User Input. Intrareliability of the tools and inter-reliability between the tools are also investigated. The improvement required in MWBP and the role of various mobile platforms in mobile adequacy today are also discussed. The study was conducted using automatic tools, manual evaluation, and statistical tools. The results of the study indicate that the coverage of the automated tools was found to lie between 45 and 68 %. The completeness ranged from 14 % (by MobileOK) to 59 % (by EvalAccess mobile). The correctness of the tools ranged from 42 to 51 %. Statistical analysis highlighted that the tools showed no significant differences in reporting false positives, false negatives and true positives, except taw and EvalAccess which reported some difference in reporting true positives. The intrareliability evaluation showed that the tools were highly stable. Intrareliability evaluation done by Krippendorff’s alpha revealed that taw and mobileOK show the similar results while evaluating the MWBP guidelines in most cases. However, for more accurate evaluations manual evaluation along with automatic evaluation must be performed. The practical significance of such a study can lead to improving the guidelines, modification of the tools, proactive use of the tools, and an overall awareness about building the Web sites accessible for all.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Kemp, S.: Digital, social and mobile worldwide in 2015. http://wearesocial.net/blog/2015/01/digital-social-mobile-worldwide-2015/ (2015)

  2. Mobile and Desktop Collide: What Are You Doing About It? http://www.mequoda.com/articles/digital-magazine-publishing/web-usage-prediction-when-mobile-and-desktop-collide/ (2015)

  3. Infographic: Mobile growth statistics. http://www.digitalbuzzblog.com/infographic-2013-mobile-growth-statistics/ (2013)

  4. Importance of mobile web in higher education recruitment: A student’s perspective. http://www.bluetrainmobile.com/blog/the-importance-of-mobile-web-in-higher-education-recruitment-a-students-perspective/ (2013)

  5. The importance of mobile optimization. http://www.getelastic.com/the-importance-of-mobile-optimization-infographic (2013)

  6. Top 6 reasons your shopping cart abandonment rate is high on mobile. http://www.mobify.com/insights/high-shopping-cart-abandonment-rate-on-mobile/ (2014)

  7. Ivory, M.Y.: Automated web site evaluation: researchers’ and practitioners’ perspectives. Springer Science and Business Media, 31 Oct 2003-Computers (2003)

  8. Vigo, M., Brown, J., Conway, V.: Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests. In: Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility W4A (2013)

  9. Mankoff, J., Fait, H., Tran, T.: Is your page accessible? A comparative study of methods for assessing web page accessibility for the blind. In: Proceedings of the SIGCHI conference on Human factors in computing system, CHI’05, pp 41–50 (2005)

  10. From WCAG 2.0 to MWBP: Making content that meets Web Content Accessibility Guidelines 2.0 also meet Mobile Web Best Practices. http://www.w3.org/TR/mwbp-wcag/wcag20-mwbp.html (2009)

  11. Yesilada, Y., Brajnik, G., Harper, S.: Barriers common to mobile and disabled web users. Inter. Comput. 23(5), 525–542 (2011)

    Article  Google Scholar 

  12. Vigo, M., Aizpurua, A., Arrue, M., Abascal, J.: Evaluating the web accessibility for specific mobile devices. In: 5th International Cross-Disciplinary Conference on Web Accessibility, W4A 2008, pp 65–72 (2008)

  13. Bandeira, R., Lopes, R., Carriço, L.: Towards mobile Web accessibility evaluation. In: ETAPS, FOSS-AMA Workshop, Paphos, Cyprus (2010)

  14. Carriço, L., Lopes, L., Bandeira, R. Crosschecking the mobile web for people with visual impairments. In: Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A 2011, Hyderabad, Andhra Pradesh, India, March 28–29 (2011)

  15. Carriço, L., Lopes, L., Bandeira, R., Isacker, K.V.: Towards mobile web accessibility: vision and challenges. In: Proceedings of the 1st International AEGIS Conference, Seville, Spain, October (2010)

  16. Vigo, M., Aizpurua, A., Arrue, M., Abascal, J.: Quantitative assessment of mobile web guidelines conformance. Univers. Access Inf. Soc. 10(1), 33–49 (2011)

    Article  Google Scholar 

  17. Rabin, J., McCathieNevile, C.: Mobile Web Best Practices 1.0. W3C Mobile Web Initiative. http://www.w3.org/TR/mobile-bp/ (2008)

  18. Brajnik, G.: Comparing accessibility evaluation tools: a method for tool effectiveness. Univers. Access Inf. Soc. 3, 252–263 (2004)

    Article  Google Scholar 

  19. W3C: Web Content Accessibility Guidelines [online]. Recommendation, W3C. http://www.w3.org/TR/WCAG10/ (1999)

  20. W3C: Overview of WCAG 2.0 Documents, [online] Web Accessibility Initiative. http://www.w3.org/TR/WCAG20/ (2008)

  21. Mobile Web Best Practices. http://www.w3.org/TR/mobile-bp/

  22. Extended Guidelines for Mobile Web Best Practices 1.0. http://www.w3.org/TR/2009/NOTE-mwbp-guidelines-20091020/

  23. Owen, S., Rabin, J.: W3C mobileOK Basic Tests 1.0. W3C Mobile Web Initiative. http://www.w3.org/TR/mobileOK-basic10-tests/ (2008)

  24. MobileOk pro test. Addendum to Mobile Web Best Practices. http://www.w3.org/2005/MWI/BPWG/Group/TaskForces/mobileOKPro/drafts/ED-mobileOK-pro10-tests-20090923

  25. Hackett, S., Parmanto, B.: Homepage not enough when evaluating web site accessibility. Internet Res. 19(1), 78–87 (2009)

    Article  Google Scholar 

  26. Random Sampling. https://explorable.com/simple-random-sampling

  27. W3C mobileOk checker. http://validator.w3.org/mobile/

  28. EvalAccess. http://sipt07.si.ehu.es/evalaccess2/mobile/index.html

  29. Taw. http://www.tawdis.net/

  30. dotMobi, mobiReady. http://ready.mobi/launch.jsp?locale=en_EN

  31. Keyboard accessibility. http://webaim.org/techniques/keyboard/tabindex

  32. WebAIM. http://webaim.org/

  33. Accesskeys. http://alistapart.com/article/accesskeys

  34. Everts, T.: Want to Deliver the best possible user experience in 2015?Consider these 8 web performance resolutions. http://blog.radware.com/applicationdelivery/applicationaccelerationoptimization/2015/01/8-web-performance-resolutions-for-2015/ (2015)

  35. Akamai report on Internet. http://www.akamai.com/dl/soti/q1-2015-soti-fullreport-a4.pdf

  36. Diaper, D., Worman, L.: Two falls out of three in the automated accessibility assessment of world wide web sites: A-Prompt v. Bobby. In: Johnson, P., Palanque, P., O’Neill, E. (eds.) People and Computers XVII, Proceeding of 17th Annual Human Computer Interaction Conference, pp. 349–363. Springer, Berlin (2004)

  37. Krippendorff, K.: Computing Krippendorff’s Alpha-Reliability. Retrieved from http://repository.upenn.edu/asc_papers/43 (2011)

  38. Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Meas. 1, 77–89 (2007)

    Article  Google Scholar 

  39. Hayes Macro for SPSS. http://www.afhayes.com/spss-sas-and-mplus-macros-and-code.html

  40. From MWBP to WCAG 2.0: Making content that meets Mobile Web Best Practices also meet Web Content Accessibility Guidelines 2.0. http://www.w3.org/TR/mwbp-wcag/mwbp-wcag20.html

  41. Clegg-Vinell, R., Bailey, C., Gkatzidou, V.: Investigating the appropriateness and relevance of mobile Web guidelines. In: Proceedings of the 11th Web for All Conference W4A’14 (2014)

Download references

Author information

Authors and Affiliations

  1. Guru Gobind Singh Inderprastha University, Sector 16C, Dwarka, New Delhi, Delhi, 110078, India

    Arvinder Kaur

  2. Inderprastha Engineering College, 63- site IV, Ghaziabad, India

    Diksha Dani

Authors
  1. Arvinder Kaur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diksha Dani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diksha Dani.

Appendix

Appendix

See Table 11.

Table 11 List of webpages considered in the study
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kaur, A., Dani, D. Comparing and evaluating the effectiveness of mobile Web adequacy evaluation tools. Univ Access Inf Soc 16, 411–424 (2017). https://doi.org/10.1007/s10209-016-0466-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-016-0466-z

Keywords

Search

Navigation