Skip to main content
Springer Nature Link
Log in

Quantitative assessment of mobile web guidelines conformance

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

Conformance metrics for the mobile web can play a crucial role as far as engineering mobile websites are concerned, especially if they are automatically obtained. In this way, developers can have an idea in numeric terms of how suitable their developments are for mobile devices. However, there are a plethora of devices with their own particular features (screen size, formats support, etc.) that restrict a unified automatic assessment process. This paper proposes a tool-supported method for device-tailored assessment in terms of conformance with Mobile Web Best Practices 1.0, including the definition of five quantitative metrics for automatically measuring mobile web conformance: Navigability, Page layout, Page definition, User input and Overall score. The behaviour of these metrics was analysed for different devices and different web paradigms, both mobile web pages and their equivalent desktop pages. As expected, the results show that mobile web pages on more capable devices score higher. In addition, 20 users took part in an experiment aimed at discovering how conformance-based scores relate to usability dimensions. The results demonstrate that automatic scoring approaches strongly correlate with usability scores obtained by direct observation, such as task completion time and user satisfaction. This correlation is even stronger for the device-tailored assessment than the one that assumes a general profile for all devices. For instance, results show a strong negative correlation between Overall score and task completion time: ρ (9) = −0.81, (p < 0.05) for the generalist approach and ρ (9) = −0.88 for the device-tailored one, entailing that mobile web guidelines and the metrics based on their conformance capture usability aspects. This result challenges the widely accepted belief that conformance to guidelines does not imply more usable web pages, at least for web accessibility conformance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. http://validadores.tawdis.net/mobileOK/

  2. http://ready.mobi/

  3. http://validator.w3.org/mobile/

  4. http://dev.w3.org/cvsweb/2007/mobileok-ref/

  5. http://www.w3.org/TR/mobile-bp/#ddc

  6. http://w3development.de/rdf/uaprof_repository/

  7. http://wurfl.sourceforge.net/

  8. http://deviceatlas.com/

  9. These websites can be found at http://cantoni.mobi

  10. Further information on the experimental settings can be found at http://sipt07.si.ehu.es/UAIS09/index.html

References

  1. Abou-Zahra, S., Squillace, M.: Evaluation and report language (EARL) 1.0 schema. W3C Evaluation and Repair Tools Working Group. http://www.w3.org/TR/EARL10-Schema/ (2009). Accessed Sep 2009

  2. Bailey, J., Burd, E.: Tree-map visualisation for web accessibility. Computer Software and Applications Conference, COMPSAC’05, pp. 275–280 (2005)

  3. Bickmore, T.W., Schilit, B.N.: Digestor: device-independent access to the World Wide Web. Computer Networks and ISDN Systems 29, 1075–1082 (1997)

    Article  Google Scholar 

  4. Buchanan, G., Farrant, S., Jones, M., Thimbleby, H.: Improving mobile internet usability. International Conference on World Wide Web, WWW’01, pp. 673–680 (2001)

  5. Buyukkokten, O., Garcia-Molina, H., Paepcke, A., Winograd, T.: Power browser: efficient web browsing for PDAs. SIGCHI Conference on Human Factors in Computing Systems, CHI’00, pp. 430–437 (2000)

  6. Chisholm, W., Vanderheiden, G., Jacobs, I.: Web content accessibility guidelines 1.0. W3C Web accessibility initiative. http://www.w3.org/TR/WAI-WEBCONTENT/ (1999). Accessed Sep 2009

  7. Church, K., Smyth, B., Cotter, P., Bradley, K.: Mobile information access: a study of emerging search behavior on the mobile internet. ACM Trans. Web 1(1), article 4 (2007)

  8. Coursaris, C.K., Kim, D.J.: A Research agenda for mobile usability. 26th Conference on Human Factors in Computing Systems, CHI’07, pp. 2345–2350 (2007)

  9. Cui, Y., Roto, V.: How people use the web on mobile devices. International Conference on World Wide Web, WWW’08, pp. 905–914 (2008)

  10. DeMarco, T.: Controlling Software Projects: Management, Measurement and Estimates. Prentice Hall, Upper Saddle River, NJ (1986)

    Google Scholar 

  11. Dujmovic, J.J.: Neural networks—concepts, applications, and implementations. In: Antognetti, P., Milutinovic, V. (eds.) Preferential Neural Networks, pp. 155–206. Prentice Hall, Upper Saddle River, NJ (1991)

    Google Scholar 

  12. Dujmovic, J.J.: A Method for evaluation and selection of complex hardware and software systems. 22nd International Computer Measurement Group Conference, pp. 368–378 (1996)

  13. Garofalakis, J., Stefanis, V.: MokE: a tool for Mobile-ok evaluation of web content. International Cross-Disciplinary Conference on Web Accessibility, W4A’08, pp. 57–64 (2008)

  14. Harper, S.: Mobile web: reinventing the wheel? ACM SIGACCESS Access. Comput. 90, 16–18 (2008)

    Article  Google Scholar 

  15. Hunter, J., Lagoze, C.: Combining RDF and XML schemas to enhance interoperability between metadata application profiles. International World Wide Web Conference, WWW’01, pp. 457–466 (2001)

  16. ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs)—Part 11: Guidance on usability. International Organization of Standardization (1998)

  17. International Telecommunication Union, ITU: Worldwide mobile cellular subscribers to reach 4 billion mark late 2008. http://www.itu.int/newsroom/press_releases/2008/29.html (2008). Accessed Sep 2009

  18. Jain, R.: The mobile web in developing countries. W3C Workshop on the Mobile Web in the Developing Countries (2006)

  19. Kaikkonen, A., Roto, V.: Navigating in a mobile XHTML application. SIGCHI Conference on Human Factors in Computer Systems, CHI 2003, pp. 329–336 (2003)

  20. Kelly, B., Sloan, D., Phipps, L., Petrie, H., Hamilton, F.: Forcing standardization or accommodating diversity?: a framework for applying the WCAG in the real world. International Cross-Disciplinary Workshop on Web Accessibility, W4A’05, pp. 46–54 (2005)

  21. Klyne, G., Reynolds, F., Woodrow, F., Ohto, H., Hjelm, J., Butler, M., Tran, L.: Composite capability/preference profiles (CC/PP): structure and vocabularies 1.0. W3C Device Independence Working Group. http://www.w3.org/TR/2004/REC-CCPP-struct-vocab-20040115/ (2004). Accessed Sep 2009

  22. Leporini, B., Paternò, F.: Applying web usability criteria for vision-impaired users: does it really improve task performance? Int. J. Human-Comput. Interact. 24(1), 17–47 (2008)

    Google Scholar 

  23. Lewis, J.R.: IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int. J. Human-Comput. Interact. 7(1), 57–78 (1995)

    Article  Google Scholar 

  24. Mankoff, J., Dey, A., Batra, U., Moore, M.: Web accessibility for low bandwidth input. ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’02, pp. 17–24 (2002)

  25. Mich, L., Franch, M., Gaio, L.: Evaluating and designing web site quality. IEEE Multimed 10(1), 34–43 (2003)

    Article  Google Scholar 

  26. Nielsen, J., Tahir, M.: Homepage Usability: 50 Websites Deconstructed. New Riders Publishing, Indianapolis (2000)

    Google Scholar 

  27. Nielsen, J.: Mobile Web 2009 = Desktop Web 1998. Jakob Nielsen’s Alertbox. http://www.useit.com/alertbox/mobile-usability.html (2009). Accessed Sep 2009

  28. Olsina, L., Rossi, G.: Measuring web application quality with WebQEM. IEEE Multimed. 9(4), 20–29. IEEE Computer Society Press (2002)

    Google Scholar 

  29. Owen, S., Rabin, J. W3C mobileOK Basic Tests 1.0. W3C Mobile Web Initiative. http://www.w3.org/TR/mobileOK-basic10-tests/ (2008). Accessed Sep 2009

  30. Rabin, J., McCathieNevile, C.: Mobile web best practices 1.0. W3C Mobile Web Initiative. http://www.w3.org/TR/mobile-bp/ (2008). Accessed Sep 2009

  31. Rohra, P.: Re: People with disabilities using mobile devices to interact with the Web. WAI-IG mailing list. http://lists.w3.org/Archives/Public/w3c-wai-ig/2008AprJun/0099.html (2008). Accessed Sep 2009

  32. Roto, V.: Web browsing on mobile phones—characteristics of user experience. Dissertation, Helsinki University of Technology (2006)

  33. Scheppe, K.: mobileOK Pro Tests Version 1. W3C Mobile Web Initiative http://www.w3.org/2005/MWI/BPWG/Group/TaskForces/mobileOKPro/drafts/ED-mobileOK-pro10-tests-20080610 (2008). Accessed Sep 2009

  34. Shrestha, S.: Mobile web browsing: usability study. International Conference on Mobile Technology, Applications, and Systems, MC’07, pp. 187–194 (2007)

  35. Sloan, D., Heath, A., Hamilton, F., Kelly, B., Petrie, H., Phipps, L.: Contextual web accessibility—maximizing the benefit of accessibility guidelines. International Cross-Disciplinary Workshop on Web Accessibility, W4A’06, pp. 121–131 (2006)

  36. Sullivan, T., Matson, R.: Barriers to use: usability and content accessibility on the Web’s most popular sites. ACM Conference on Universal Usability, CUU’00, pp. 139–144 (2000)

  37. Thatcher, J., Burks, M.R., Heilmann, C., Henry, S.L., Kirkpatrick, A., Lauke, P.H., Lawson, B., Regan, B., Rutter, R., Urban, M., Waddell, C.D.: Web Accessibility: Web Standards and Regulatory Compliance. Springer, New York (2006)

    Google Scholar 

  38. Trewin, S.: Physical usability and the mobile web. International Cross-Disciplinary Workshop on Web Accessibility, W4A’06, pp. 109-112 (2006)

  39. Vigo, M., Arrue, M., Brajnik, G., Lomuscio, R., Abascal, J.: Quantitative metrics for measuring web accessibility. International Cross-Disciplinary Workshop on Web Accessibility, W4A’07, pp. 99–107 (2007)

  40. Vigo, M., Kobsa, A., Arrue, M., Abascal, J.: User-tailored web accessibility evaluations. ACM Conference on Hypertext and Hypermedia, HYPERTEXT’07, pp. 95–104 (2007)

  41. Vigo, M., Aizpurua, A., Arrue, M., Abascal, J.: Evaluating web accessibility for specific mobile devices. International Cross-Disciplinary Conference on Web Accessibility, W4A’08, pp. 65–72 (2008)

  42. Vigo, M., Brajnik, G., Arrue, M., Abascal, J.: Tool independence for the web accessibility quantitative metric. Disabil. Rehabil. Assist. Technol. 4(4), 248–263 (2009)

    Article  Google Scholar 

  43. Vigo, M., Leporini, B., Paternò, F.: Enriching web information scent for blind users. ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’09, pp. 123–130 (2009)

  44. Yesilada, Y., Chuter, A., Henry, S.L.: Shared web experiences: barriers common to mobile device users and people with disabilities. W3C Web accessibility initiative. http://www.w3.org/WAI/mobile/experiences (2008). Accessed Sep 2009

Download references

Author information

Authors and Affiliations

  1. Computer Science School, Department of Computer Architecture and Technology, University of the Basque Country, Manuel Lardizabal 1, 20018, Donostia, Spain

    Markel Vigo, Amaia Aizpurua, Myriam Arrue & Julio Abascal

Authors
  1. Markel Vigo
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Amaia Aizpurua
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Myriam Arrue
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Julio Abascal
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Markel Vigo.

Appendix

Appendix

See Table 7.

Table 7 URLs used in the user test
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Vigo, M., Aizpurua, A., Arrue, M. et al. Quantitative assessment of mobile web guidelines conformance. Univ Access Inf Soc 10, 33–49 (2011). https://doi.org/10.1007/s10209-010-0186-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-010-0186-8

Keywords