Maria Ntouvaleti
Christos Katsanos
ABSTRACT
When it comes to creating website information structures, open card sorting is the main approach used. However, scientific research for the method's validity is lacking. This paper explores the validity of open card sorting for website structural design. To this end, participants first performed an open card sort for the redesign of a website with usability issues in its information structure. Next, a within-subjects user testing study was performed to compare two functional prototypes that differed only in their structure: one replicated the existing website structure, whereas the other implemented the structure produced by card sort data analysis. Results showed that participants using the redesigned structure had significantly better usability metrics (first click success rate, task time, SEQ score and SUS score) compared to when interacting with the existing structure. These findings provide support for the validity of the open card sorting method for the design of website information structures.
Supplemental Material
- George Adamides, Georgios Christou, Christos Katsanos, Michalis Xenos, and Thanasis Hadzilacos. 2015. Usability guidelines for the design of robot teleoperation: A taxonomy. IEEE Transactions on Human-Machine Systems 45 (2), 256–262.Google Scholar
Cross Ref
- William Albert and Thomas S. Tullis. 2013. Measuring the user experience: Collecting, analyzing, and presenting usability metrics (2nd ed.). Morgan Kaufmann.Google Scholar
- Aaron Bangor, Philip Kortum, and James Miller. 2008. An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 24 (6), 574–594.Google ScholarCross Ref
- Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS Scores mean: adding an adjective rating scale. J. Usability Stud. 4 (3), 114–123.Google ScholarDigital Library
- John Brooke. 1996. SUS: a “quick and dirty” usability scale. In Usability Evaluation in Industry, P. W. Jordan, B Thomas, B. A. Weerdmeester and A. L. McClelland (eds.). Taylor and Francis, London.Google Scholar
- Miranda G. Capra. 2005. Factor analysis of card sort data: an alternative to hierarchical cluster analysis. In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting, HFES, Santa Monica, CA, 691–695.Google ScholarCross Ref
- Brendan Cassidy, Dipti Saurabh Antani, and Janet C C. Read. 2013. Using an open card sort with children to categorize games in a mobile phone application store. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), ACM, New York, NY, USA, 2287–2290.Google Scholar
- Barbara S. Chaparro, Veronica D. Hinkle, and Shannon K. Riley. 2008. The usability of computerized card sorting: a comparison of three applications by researchers and end users. Journal of Usability Studies 4 (1), 31–48.Google ScholarDigital Library
- Jacob Cohen. 1992. A power primer. Psychological Bulletin 112 (1), 155–159.Google ScholarCross Ref
- Jianming Dong, Shirley Martin, and Paul Waldo. 2001. A user input and analysis tool for information architecture. In CHI ’01 extended abstracts on Human factors in computing systems, ACM, Seattle, Washington, 23–24.Google Scholar
- Brian Dorn and Mark Guzdial. 2010. Learning on the job: characterizing the programming knowledge and learning strategies of web designers. In Proceedings of the 28th international conference on Human factors in computing systems (CHI ’10), Atlanta, Georgia, USA, 703–712.Google ScholarDigital Library
- Andy P Field. 2013. Discovering statistics using IBM SPSS statistics (4th ed.). SAGE, London.Google ScholarDigital Library
- Michelle E. Harper, Florian Jentsch, Lori Rhodenizer Van Duyne, Kimberly Smith-Jentsch, and Alicia D. Sanchez. 2002. Computerized card sort training tool: Is it comparable to manual card sorting? In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, SAGE Publications Inc, 2049–2053.Google Scholar
- William Hudson. 2013. Card sorting. In The Encyclopedia of Human-Computer Interaction, 2nd Ed., Mads Soegaard and Rikke Friis Dam (eds.). Interaction Design Foundation. Retrieved from https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-edGoogle Scholar
- Rebecca Jeong and Sonia Chiasson. 2020. “Lime”, “open lock”, and “blocked”: children's perception of colors, symbols, and words in cybersecurity warnings. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), ACM, New York, NY, USA, 14 pages.Google Scholar
- Karin Kappel, Martin Tomitsch, Thomas Költringer, and Thomas Grechenig. 2006. Developing user interface guidelines for DVD menus. In CHI ’06 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’06), ACM, New York, NY, USA, 177–182.Google Scholar
- Christos Katsanos, Nikolaos Tselios, and Nikolaos Avouris. 2008. AutoCardSorter: designing the information architecture of a web site using latent semantic analysis. In Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, CHI 2008, ACM, Florence, Italy, 875–878.Google ScholarDigital Library
- Christos Katsanos, Nikolaos Tselios, and Nikolaos Avouris. 2008. Automated semantic elaboration of web site information architecture. Interacting with Computers 20 (6), 535–544.Google ScholarDigital Library
- Christos Katsanos, Nikolaos Tselios, Nikolaos Avouris, Stavros Demetriadis, Ioannis Stamelos, and Lefteris Angelis. 2019. Cross-study reliability of the open card sorting method. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19), ACM, New York, NY, USA, LBW2718:1-LBW2718:6.Google ScholarDigital Library
- Christos Katsanos, Nikolaos Tselios, Jorge Goncalves, Tomi Juntunen, and Vassilis Kostakos. 2014. Multipurpose public displays: Can automated grouping of applications and services enhance user experience? International Journal of Human-Computer Interaction 30 (3), 237–249.Google ScholarCross Ref
- Christos Katsanos, Nikolaos Tselios, and Michalis Xenos. 2012. Perceived usability evaluation of learning management systems: a first step towards standardization of the System Usability Scale in Greek. In 2012 16th Panhellenic Conference on Informatics, 302–307.Google ScholarDigital Library
- Christos Katsanos, Georgia Zafeiriou, and Alexandros Liapis. 2022. Effect of self-efficacy on open card sorts for websites. In Proceedings of the 24th International Conference on Human-Computer Interaction, HCI International 2022, Springer International Publishing, Gothenburg, Sweden, (in press).Google Scholar
- Christina Kelley, Bongshin Lee, and Lauren Wilcox. 2017. Self-tracking for mental wellness: understanding expert perspectives and student experiences. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), ACM, New York, NY, USA, 629–641.Google ScholarDigital Library
- Georgios Melissourgos and Christos Katsanos. 2020. CardSorter: Towards an open source tool for online card sorts. In Proceedings of the 24th Pan-Hellenic Conference on Informatics (PCI 2020), ACM, New York, NY, USA, 77–81.Google ScholarDigital Library
- Ather Nawaz. 2012. A Comparison of card-sorting analysis methods. In Proceedings of the 10th Asia Pacific Conference on Computer-Human Interaction, APCHI ’12, ACM Press, 583–592.Google Scholar
- Jacob Nielsen. 2004. Card Sorting: How many users to test. Alertbox. Retrieved from http://www.useit.com/alertbox/20040719.htmlGoogle Scholar
- Konstantina Orfanou, Nikolaos Tselios, and Christos Katsanos. 2015. Perceived usability evaluation of learning management systems: Empirical evaluation of the System Usability Scale. The International Review of Research in Open and Distributed Learning 16 (2), 227–246.Google ScholarCross Ref
- Sione Paea and Ross Baird. 2018. Information Architecture (IA): Using multidimensional scaling (MDS) and K-Means clustering algorithm for analysis of card sorting data. Journal of Usability Studies 13 (3), 138–157.Google ScholarDigital Library
- Styliani Pampoukidou and Christos Katsanos. 2021. Test-retest reliability of the open card sorting method. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (CHI EA ’21), Association for Computing Machinery, New York, NY, USA, Article330:1-Article330:7.Google ScholarDigital Library
- Celeste Paul. 2008. A modified Delphi approach to a new card sorting methodology. Journal of Usability Studies 4 (1), 7–30.Google ScholarDigital Library
- Helen Petrie, Christopher Power, Paul Cairns, and Cagla Seneler. 2011. Using card sorts for understanding website information architectures: technological, methodological and cultural Issues. In Human-Computer Interaction – INTERACT 2011 (Lecture Notes in Computer Science), Springer Berlin Heidelberg, 309–322.Google Scholar
- Carol Righi, Janice James, Michael Beasley, Donald L. Day, Jean E. Fox, Jennifer Gieber, Chris Howe, and Laconya Ruby. 2013. Card sort analysis best practices. Journal of Usability Studies 8 (3), 69–89.Google ScholarDigital Library
- Louis Rosenfeld, Peter Morville, and Jorge Arango. 2015. Information architecture: For the Web and beyond (4 edition ed.). O'Reilly Media, Sebastopol, CA.Google Scholar
- Jeff Sauro. 2012. 10 things to know about the Single Ease Question (SEQ). MeasuringU. Retrieved from http://www.measuringu.com/blog/seq10.phpGoogle Scholar
- Jeff Sauro and Joseph S. Dumas. 2009. Comparison of three one-question, post-task usability questionnaires. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), ACM, New York, NY, USA, 1599–1608.Google Scholar
- Hasti Seifi, Michael Oppermann, Julia Bullard, Karon E. MacLean, and Katherine J. Kuchenbecker. 2020. Capturing experts’ mental models to organize a collection of haptic devices: affordances outweigh attributes. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), ACM, New York, NY, USA, 13 pages.Google Scholar
- Donna Spencer. 2009. Card sorting: designing usable categories. Rosenfeld Media, Brooklyn, N.Y.Google Scholar
- Thomas Tullis and J Stetson. 2004. A comparison of questionnaires for assessing website usability. In Usability Professionals Association (UPA) 2004 Conference, 7–11.Google Scholar
- Thomas Tullis and Larry Wood. 2004. How many users are enough for a card-sorting study? In Usability Professionals Association (UPA) 2004 Conference, Minneapolis, MN.Google Scholar
- Thomas Tullis and Larry Wood. 2004. How can you do a card-sorting study with LOTS of cards? In Usability Professionals Association (UPA) 2004 Conference, Minneapolis, MN.Google Scholar
- Jed Wood and Larry Wood. 2008. Card sorting: current practices and beyond. Journal of Usability Studies 4 (1), 1–6.Google ScholarDigital Library
- Georgia Zafeiriou, Christos Katsanos, and Alexandros Liapis. 2021. Effect of sense of direction on open card sorts for websites. In CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter (CHI Greece 2021), Association for Computing Machinery, New York, NY, USA, 1–8.Google Scholar
- Panayiotis Zaphiris, Mariya Ghiawadwala, and Shabana Mughal. 2005. Age-centered research-based web design guidelines. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’05), ACM, New York, NY, USA, 1897–1900.Google Scholar
Recommendations
Test-Retest Reliability of the Open Card Sorting Method
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing SystemsOpen card sorting is a popular method for designing information architectures. However, there is a lack of empirical evidence on the validity and reliability of the method. This paper explores the test-retest reliability of open card sorting. To this end,...
CardSorter: Towards an Open Source Tool for Online Card Sorts
PCI '20: Proceedings of the 24th Pan-Hellenic Conference on InformaticsStructural design of shared information environments is rather challenging. Card sorting constitutes a valuable method towards this end. However, there is a lack of free software tools for conducting card sorts, especially from a distance, and analysing ...
Effect of Sense of Direction on Open Card Sorts for Websites
CHI Greece 2021: CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI ChapterCard sorting is one widely used method to design website information architectures based on an understanding of their users’ mental models. This paper studies the effect of participants’ sense of direction on card sorting results. To this end, a two-...
Comments