skip to main content
10.1145/3535782.3535814acmotherconferencesArticle/Chapter ViewAbstractPublication PagesmsieConference Proceedingsconference-collections
research-article

User Experience Evaluation Using the Cognitive Walkthrough Method

Published: 18 July 2022 Publication History

Abstract

The redesign of the website has been carried out without formal user experience testing regarding user experience in using the system so that similar complaints will arise on the latest website. The purpose of this research is to identify and analyze what problems users will face and also provide recommendations for maintaining the website interface to make it more user-friendly. In this study, the author tested the user experience on the PMB UAJY website using the cognitive walkthrough method. A cognitive walkthrough is a method devoted to new users by testing usability in which the examiner performs a series of task scenarios to achieve his goals. The usability test components include effectiveness, efficiency, and user satisfaction. The results of this test indicate that the lowest usability level of 55.10% is still classified as good. The margin of error in this task scenario is found in the Undergraduate Student Registration menu. Problems faced by users when using websites that display a graphic design that is monotonous and tends to be text-based, navigation and buttons that are too small in size, lack of consistency of buttons on the Undergraduate Registration menu, and users are required to fill out Guardian Data Forms. Improvements needed by this website are providing complete information content, attractive interface appearance, enlarged and consistent layout and navigation, and reconfirmation notifications for every action on the registration menu.

References

[1]
Cascio, Wayne & Montealegre, Ramiro. 2013. “How Technology Is Changing Work and Organizations. Annual Review of Organizational Psychology and Organizational Behavior,” 10.1146/annurev-orgpsych041015-062352. Sam Anzaroot and Andrew McCallum. 3. 349-375. UMass Citation Field Extraction Dataset. Retrieved May 27, 2019, from http://www.iesl.cs.umass.edu/data/data-umasscitationfield
[2]
Mifsud, Justin.,” Usability Metrics – A Guide o Quantify The Usability Of Any System,” https://usabilitygeek.com/usability-metrics-a-guide-to-quantifysystem-usability/
[3]
Lewis C., Polson P., Wharton C., and Rieman J. 1990. Testing a walkthrough methodology for theory based design of walk-up-and-use interface. In Proceeding of ACM CHI’90 Conference, New York: 235-242.
[4]
Wang X. 2008. Design and evaluation of intelligent menu interface through cognitive walkthrough procedure and automated logging for management information system. 11th International Conference, CSCWD 2007, Melbourne, Australia, April 26-28, 2007. Revised Selected Papers.
[5]
Brinck T, Gergle D, and Wood SD. 2002. Designing website that work usability for the web. California, USA: Academic Press.Vermeeren Arnold, Kaisa Väänänen-Vainio-Mattila, Effie Law,. 2011. “User experience Evaluation – Which Method to Choose?,” 13th International Conference on Human-Computer Interaction (INTERACT), Lisbon, Portugal.
[6]
Fadel, Luciane Maria, 2014. “Experience-Centered Web Design Model,” Federal University of Santa Catarina Florianópolis Brazil.
[7]
Garrett, Jesse James,. 2011. “The Elements of User experience: User Centered Design for The Web and Beyond,” Barkeley, CA: New Riders.
[8]
R. Rubinoff. 2004. “How to Quantify The User experience,” www.sitepoint.com. www.sitepoint.com/quantify-user-experience/ (accessed Apr. 08, 2021)
[9]
Blackmon, M. H. Polson, P.G. Muneo, K & Lewis, C.,. 2002. “Cognitive walkthrough for the Web CHI,” vol.4 No.1 pp463–470.
[10]
Wharton, C., Rieman, J., Lewis, C. & Polson, P. 1994. “The Cognitive walkthrough: A practitioner's guide. In J. Nielsen & R. L. Mack (Eds.),” Usability inspections methods (pp. 105-140). New York: Wiley.
[11]
Jacobsen, N & John, B. 2000. “Two case studies in using Cognitive walkthrough for interface evaluation,“Pittsburgh (PA): Human Computer Interaction Institute, School of Computer Science, Carnegie Mellon University
[12]
W. Hwang and G. Salvendy, 2010 “Number of people required for usability evaluation: The 10±2 rule,” Commun. ACM, vol. 53, no. 5, pp. 130–133.
[13]
Wilson C. 2014 “User interface inspection methods: a user-centered design methods,” Waltham, MA: Morgan Kaufmann.
[14]
Nielsen, Jakob. 2001. “Success Rate: The Simplest Usability Metric,” Available: https://www.nngroup.com/articles/success-rate-the-simplest-usabilitymetric/.
[15]
Rauterberg M. 2006. Usability engineering evaluation method. http://www.idemployee.id.tue.nl/g.w.m.rauterberg/lecturenotes/JFS-lecture/JFS-USI-primer-8.pdf (Access date 1 December 2021)
[16]
Nielsen, Jakob. 2000. "Why You Only Need to Test with 5 Users," Nielsen Norman Group, Available: http://www.nngroup.com/articles/why-you-only-need-totest-with-5-users/.
[17]
H. B. Santoso, M. Schrepp, R. Y. K. Isal, A. Y. Utomo, and B. Priyogi. 2016. “Measuring User Experience of the Student-Centered e-Learning Environment,” J. Educ. Online, vol. 13, no. 1, pp. 58–79.
[18]
Schrepp, M. 2019. “User Experience Questionnaire Handbook”. Germany.
[19]
Therdoost, H. 2016. “Sampling Methods in Research Methology: How to Choose a Sampling Technique for Research.” International Journal of Academic Research in Management.
[20]
Schrepp, M. 2019. “User Experience Questionnaire Handbook – All You Need to Know How to Apply Th UEQ Succesfully in Your Projects.”
[21]
Sauro, J., Lewis, J. R. 2016. “Quatifying th e User Experience: Pratical Statics for User Research.” Boton: Morgan Kauffmann.
[22]
Ptten, M. L., Newhart, M. 2013. Understanding Research Methods: an Overview of The Essentials. New York: Routledge.
[23]
Lee, S. 2020. A Study on User Experience of Unmanned Payment Kiosk System in ast Food Restaurants. International Journal of Smart Business and Technology.
[24]
Abdilah, L. A. 2020. Fintech E-Commerce Payment Appliaction User Experience Analysis during COVID-19 Pandemic. Scientific Journal of Informatics.
[25]
Deng, L., Turner, D. E., Gehling, R., Prince, B. 2010. User Experience, Satisfaction, and Continual usage intention of IT. European Journal of Information Systems.

Cited By

View all
  • (2025)User Model for Virtual Learning Based on Adaptive GamificationIEEE Access10.1109/ACCESS.2025.353759913(24028-24040)Online publication date: 2025
  • (2023)Adopting Five Planes Framework for Developing User Interface based on User Experience Aspects2023 11th International Conference on Cyber and IT Service Management (CITSM)10.1109/CITSM60085.2023.10455146(1-6)Online publication date: 10-Nov-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
MSIE '22: Proceedings of the 4th International Conference on Management Science and Industrial Engineering
April 2022
497 pages
ISBN:9781450395816
DOI:10.1145/3535782
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Cognitive walkthrough
  2. User experience
  3. Website

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

MSIE 2022

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)129
  • Downloads (Last 6 weeks)10
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)User Model for Virtual Learning Based on Adaptive GamificationIEEE Access10.1109/ACCESS.2025.353759913(24028-24040)Online publication date: 2025
  • (2023)Adopting Five Planes Framework for Developing User Interface based on User Experience Aspects2023 11th International Conference on Cyber and IT Service Management (CITSM)10.1109/CITSM60085.2023.10455146(1-6)Online publication date: 10-Nov-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media