Skip to main content
SpringerLink
Log in

A unified methodology for the evaluation of accessibility and usability of mobile applications

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

This article reports a unified methodology developed to evaluate the accessibility and usability of mobile computing applications, which is intended to guarantee universal access as far as possible. As a basis for the methodology, this paper presents an analysis of the accessibility guidelines, conducted to take into account the specificity of mobile systems, as well as a set of usability heuristics, specifically devised for mobile computing. Finally, it presents the results of the application of the proposed methodology to applications that have been semi-automatically developed by the MAIS Designer, a new design tool that provides applications suited to different mobile devices.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. The severity problem classification developed by Nielsen in his Heuristic Evaluation method [50] is used. Problems are evaluated according to a severity rating scale of 0–4. 0 means no usability problem at all; 1 = cosmetic problem only: need not be fixed unless extra time is available on project; 2 = minor usability problem: fixing this should be given low priority; 3 = major usability problem: important to fix, so should be given high priority; 4 = usability catastrophe: imperative to fix this before product can be released.

  2. It is worth noting that an individual problem is sometimes indicated by the participants to be associated to more than one heuristic and thus the total of the right column of Table 2 does not equal the actual number of problems.

References

  1. Bertini, E., Calì, A., Santucci, G.: Automatic interface generation through interaction, users, and devices modeling. In: Proceedings of 6th International Conference on Computer-Aided Design of User Interfaces CADUI2006, Bucharest, Romania, June 5–8, 2006

  2. Stephanidis, C., Savidis, A.: Universal Access in the Information Society: Methods, Tools and Interaction Technologies. Univers. Access Inf. Soci. J. 40–55, Springer-Verlag (2001)

  3. Emiliani, P.L., Stephanidis, C.: Universal access to ambient intelligence environments: Opportunities and challenges for people with disabilities. IBM Sys. J. 44(3) (2005)

  4. Gray, W.D., Salzman, M.C.: Damaged merchandise? A review of experiments that compare usability evaluation methods. Hum. Comput. Interact. 13(3), 203–261 (1998)

    Article  Google Scholar 

  5. Lang, T.: Comparing website accessibility evaluation methods and learnings from usability evaluation methods http://www.peakusability.com.au/about-us/pdf/website_accessibility.pdf, December 2003

  6. ISO/IEC Guide 71Guidelines for standards developers to address the needs of older persons and persons with disabilities. International Organization for Standardization (ISO) (2001)

  7. ISO 9241-171Ergonomics of human-system interaction—Part 171: Guidance on software accessibility (2006)

  8. ISO 9241-20 Ergonomics of human-system interaction—Part 20: Accessibility guidelines for information/communication technology (ICT) equipment and services. (2006)

  9. Wegge, K.P., Zimmermann, D.: Accessibility, usability, safety, ergonomics: concepts, models, and differences. Universal Access in HCI, Part I, HCII 2007, LNCS 4554, pp. 294–301, Springer-Verlag, (2007)

  10. Henry, S.L.: Introduction to web Accessibility. http://www.w3.org/wai/intro/accessibility.php (2005)

  11. Pernici, B. (ed.): Mobile Information Systems: Infrastructure and Design for Adaptivity and Flexibility, p. 237. Springer, Berlin (2006)

    Google Scholar 

  12. Petrie, H., Kheir, O.: The relationship between accessibility and usability of websites. CHI ‘07: In: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 397–406, ACM Press, USA (2007)

  13. Thatcher, J., Waddell, C.D., Henry, S.L., Swierenga, S., Urban, M.D., Burks, M., Regan, B., Bohman, P.: Constructing Accessible Web Sites. Glasshaus, San Francisco (2003)

    Google Scholar 

  14. Shneiderman, B.: Universal usability. Commun. ACM 43(5), 85–91 (2000)

  15. Shneiderman, B.: Promoting universal usability with multi-layer interface design. In: Proceedings of the 2003 Conference on Universal Usability (CUU 2003)

  16. Antona, M., Mourouzis, A., Stephanidis, C.: Towards a Walk- through Method for Universal Access Evaluation. Universal Access in HCI, Part I, HCII 2007, LNCS 4554, pp. 325–334, Springer-Verlag (2007)

  17. Theofanos, M.F., Redish, J.: Bridging the gap: between accessibility and usability. ACM Interact. 10(6), 36–51 (2003). ACM Press

    Article  Google Scholar 

  18. Chisholm, W., Vanderheiden, G., Jacobs, I.: Web Content Accessibility Guidelines 1.0 W3C Recommendation, (http://www.w3.org/TR/1999/WAI-WEBCONTENT-19990505). May 5, 1999

  19. Henry, S., Grossnicklem, M.: Just Ask: Accessibility in the User-Centered Design Process. Georgia Tech Research Corporation, Atlanta, Georgia, USA, 2004. On-line book: http://www.uiaccess.com/accessucd/index.html

  20. Brajnik, G.: A comparative test of web accessibility evaluation methods. In: Proceedings of ASSETS’08, ACM Press, Halifax, 13–15 Oct 2008

  21. Souza, R.: Design Accessible Sites Now, Forrester Report, December 2001 http://www.forrester.com/ER/Research/Report/Sum-mary/0,1338,11431,00.html

  22. Zimmermann, G., Vanderheiden, G.: Accessible design and testing in the application development process: considerations for an integrated approach. Univer. Access Infor. Soci. Int. J. 7, 117–128 (2007). Springer-Verlag

    Article  Google Scholar 

  23. Manko, J., Dey, A.K., Hsieh, G., Kientz, J., Lederer, S., Ames, M.: Heuristic evaluation of ambient displays. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: CHI’03, ACM Press, pp. 169–176 (2003)

  24. Baker, K., Greenberg, S., Gutwin, C.: Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration. In: Proceedings of the 8th IFIP Working Conference on Engineering for Human-Computer Interaction (EHCI’01). (May 11–13, Toronto, Canada), 2001

  25. Abowd, G.D., Mynatt, E.D.: Charting past, present, and future research in ubiquitous computing. ACM Trans. Comput. Hum. Interact. (2000)

  26. Dix, A., Finlay, J., Abowd, G., Beale, R.: Human-Computer Interaction, 3rd edn. Prentice Hall, Upper Saddle River (2004)

    Google Scholar 

  27. Abowd, G.D., Mynatt, E.D., Rodden, T.: The human experience. IEEE Perva. Comput. pp. 48–57, (2002)

  28. Gabrielli, S., Mirabella, V., Kimani, S., Catarci, T.: Supporting Cognitive Walkthrough with Video Data: A Mobile Learning Evaluation Study. MobileHCI 05 Conf., Salzburg (Austria), September 2005

  29. Duh, H.B.-L., Tan, G.C.B. et Chen, V.H.-H.: Usability evaluation for mobile device: a comparison of laboratory and field tests. In: Proceedings of the 8th conference on human-computer interaction with mobile devices and services (Helsinki, Finland), pp. 181–186, ACM Press (2006)

  30. Kaikkonen, A., Keklinen, A., Cankar, M., Kallio, T., Kankainen, A.: Usability testing of mobile applications: a comparison between laboratory and field testing. J. Usabil. Stud. 1(1), 4–17 (2005)

    Google Scholar 

  31. Kjeldskov, J., Stage, J.: New techniques for usability evaluation of mobile systems. IJHCS 60(5–6), 599–620 (2004)

    Google Scholar 

  32. Roto, V., Oulasvirta, A., Haikarainen, T., Lehmuskallio, H. et Nyyssnen, T.: Examining mobile phone use in the wild with quasi-experimentation. Helsinky Institute for Information Technology (HIIT), August 2004, Technical Report 2004-1

  33. Waterson, S., Landay, J.A. et Matthews, T.: In the lab and out in the wild: remote web usability testing for mobile devices. CHI’02 conference on Human Factors in Computing Systems extended abstracts (Minneapolis, Minnesota, USA), pp. 796–797, ACM Press (2002)

  34. Po, S.: Mobile usability testing and evaluation. Master’s Thesis, University of Melbourne, Australia (2003)

  35. Vetere, F., Howard, S., Pedell, S., Balbo, S.: Walking through mobile use: Novel heuristics and their application. In: Proceedings of Conference of the Computer-Human Interaction Special Interest Group of the Human Factors and Ergonomics Society of Australia OZCHI2003 (2003)

  36. Bertini, E., Gabrielli, S., Kimani, S., Catarci, T., Santucci, G.: Appropriating and Assessing Heuristics for Mobile Computing. In: Proceedings of the International Conference in Advanced Visual Interfaces (AVI) (2006)

  37. Bertini, E., Catarci, T., Dix, A., Gabrielli, S., Kimani, S., Santucci, G.: Appropriating heuristic evaluation for mobile computing. Int. J. Mobile Hum. Comput. Interact. 1(1), 20–41 (2009)

    Google Scholar 

  38. Thorp, J., Henry, S.L.: Web Content Accessibility and Mobile Web: Making a Web Site Accessible Both for People with Disabilities and for Mobile Devices. http://www.w3.org/WAI/mobile/

  39. Rabin, J., McCathieNevile, C.: Mobile Web Best Practices 1.0. W3C Recommendation 29 July 2008 http://www.w3.org/TR/mobile-bp/

  40. Smillie, D.: Web accessibility: is it just a merry-go-round?. In: Proceedings of the 2006 International Cross-Disciplinary Work- Shop on Web Accessibility (W4A). pp. 90–91. ACM Press, Edinburgh, 2006

  41. Harper, S., Yesilada, Y.: Building the mobile web: rediscovering acessibility? Univers. Access Infor. Soci. Int. J. 6, 219–220 (2007). Springer-Verlag

    Article  Google Scholar 

  42. Trewin, S. Physical usability and the mobile web. In: Proceedings of the 2006 international cross-disciplinary workshop on Web accessibility (W4A). pp. 109–112. ACM Press, Edinburgh, 2006

  43. Owen, S., Rabin, J.: “W3C mobileOK Basic Tests 1.0—Can didate Recommendation” http://www.w3.org/TR/2007/CR-mobileOK-basic10-tests-20071130/30 November, 2007

  44. Garofalakis, J., Stefanis, V.: MokE: a tool for Mobile-ok evaluation of Web Content. In: Proceedings of the 2008 international cross-disciplinary conference on Web accessibility (W4A), Beijing, China, pp. 57–64. ACM press, 2008

  45. Vigo, M., Aizpurua, A., Arrue, M., Abascal, J.: Evaluating Web Accessibility for Specific Mobile Devices.In: Proceedings of the 2008 international cross-disciplinary conference on Web accessibility (W4A), Beijing, China, pp. 65–72. ACM press, 2008

  46. TAW mobileOK Basic checker, http://www.validadores.tawdis.net/mobileok/en/

  47. W3C mobileOK Checker http://www.validator.w3.org/mobile/

  48. Chandler, E., Dixon, E., Moniz Pereira, L., Espadinha, C.: A Comparison Study between a Heuristic Evaluation Technique and End User Trial for Mobile Phone Accessibility—Published by COST (2006)

  49. Cockton, G., Lavery, D., Woolrych, A.: Inspection-based evaluations. In Jacko, J.J., Sears, A. (eds.). The Human-Computer Interaction Handbook (2003)

  50. Nielsen, J.: Usability Engineering. Morgan Kaufmann Publishers, San Francisco (1994)

    Google Scholar 

  51. Caldwell, B., Chisholm, W., Vanderheiden, G., White, J.: Web Content Accessibility Guidelines 2.0 W3C Working Draft, (http://www.w3.org/TR/2005/WD-WCAG20-20051123/). November 23, 2005

  52. Jacobs, I., Gunderson, J., Hansen, E.: User Agent Accessibility Guidelines 1.0. W3C Recomendation, (http://www.w3.org/TR/UAAG10/). 17 December 2002

  53. Guidelines containing the technical requirements, the definitions of the different accessibility levels and the technical methodologies for the testing of Web site accessibility. Law 4 of 9/1/2004, Article 11, Paragraph 1, Letters a) and b) December 2004 (Version 3) http://www.pubbliaccesso.gov.it/biblioteca/documentazione/guidelines_study/

  54. Chuter, A.: Relationship Between Mobile Web Best Practices 1.0 and Web Content Accessibility Guidelines, (http://www.w3.org/TR/mwbp-wcag/)

  55. Po, S., Howard, S., Vetere, F., Skov, M.B. In: Brewster, S., Dunlop, M. (eds.). Heuristic Evaluation and Mobile Usability: Bridging the Realism Gap. In: Proceedings of Mobile HCI 2004, pp. 49–60, 2004

  56. Kjeldskov, J., Graham, C., Pedell, S., Vetere, F., Howard, S., Balbo, S., Davies, J.: Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behav. Inf. Technol. 24(1), 51–65 (2005)

    Article  Google Scholar 

  57. Bertini, E., Santucci, G.: Modeling user-system data exchange to design adaptive interfaces. In: Proceedings del International Workshop on Plastic Services for Mobile Devices 18 PSMD05, Rome, Italy, September 12, 2005, during the10th International IFIP Conference on Human-Computer Interaction Interact 2005

  58. MAIS Deliverable R.7.3.4: Specifiche del Prototipo per la Generazione di Interfacce Utente, 2004

  59. MAIS Deliverable P.7.3.8: Prototipo per la generazione di interfacce utente, 2005

  60. MAIS Deliverable R.7.3.8: Specifications of the Second Level Prototype of the MaisDesigner, 2005

  61. Kjeldskov, J., Graham, C.: A Review of Mobile HCI Research Methods. In L. Chittaro (ed.). Proceedings of Mobile HCI 2003, pp. 317–335, Springer-Verlag, 2003

  62. Battestini, Flanagan, J.A.: “Modelling and Simulating Context Data in a Mobile Environment”, Workshop on Context Awareness for Proactive Systems (CAPS), 2005, pp. 127–136, Helsinki, Finland

  63. De Waard, D., Brookhuis, K.A., Toffetti, K.A. (eds.): Developments in Human Factors in Transportation, Design, and Evaluation. Field Versus Laboratory Usability Testing: A First Comparison, pp. 205–212. Shaker Publishing, Maastricht (2006)

    Google Scholar 

  64. Nielsen, C.: Testing in the Field. In: Proceedings of APCHI 98, pp. 285–290, (1998)

  65. Kjeldskov, J., Skov, M.B., Als, B.S., Høegh, R.B.: “Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field”. In: Proceedings of 6th International Conference on Human Computer Interaction with Mobile Devices and Services, September 2004, University of Strathclyde, Glasgow, Scotland

Download references

Acknowledgments

Research partially supported by the MIUR/FIRB Project “MAIS” (http://www.black.elet.polimi.it/mais) and the EC NoE G038-507618, “DELOS” (http://www.delos.info/). We would like to thank Alan Dix for many fruitful discussions and comments on earlier drafts of this paper and the anonymous Reviewers for their useful suggestions.

Author information

Authors and Affiliations

  1. Consiglio Nazionale delle Ricerche, Istituto di Fisica Applicata “Nello Carrara”, Via Madonna del Piano, 10, 50019, Sesto Fiorentino (FI), Italy

    Marco Billi, Laura Burzagli, Francesco Gabbanini & Enrico Palchetti

  2. Universià di Roma “La Sapienza”, Dipartimento di Informatica e Sistemistica, Via Salaria, 113, 00198, Rome, Italy

    Tiziana Catarci, Giuseppe Santucci & Enrico Bertini

Authors
  1. Marco Billi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Laura Burzagli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tiziana Catarci
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Giuseppe Santucci
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Enrico Bertini
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Francesco Gabbanini
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Enrico Palchetti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marco Billi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Billi, M., Burzagli, L., Catarci, T. et al. A unified methodology for the evaluation of accessibility and usability of mobile applications. Univ Access Inf Soc 9, 337–356 (2010). https://doi.org/10.1007/s10209-009-0180-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-009-0180-1

Keywords

Search

Navigation