Skip to main content
SpringerLink
Log in

How Many Participants Do You Need for an Open Card Sort? A Case Study of E-commerce Websites

  • Conference paper
  • First Online:
Human-Computer Interaction – INTERACT 2023 (INTERACT 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14145))

Included in the following conference series:

  • 1056 Accesses

Abstract

Open card sorting is the most used method for developing user-centered information architectures. One important question for every HCI method is how many users to involve. Existing studies that address this question for open card sorts have involved trained professionals sorting content items of rather specialized domains. In addition, they employ data analysis approaches that might decrease the confidence one can place on the reported findings. This paper investigates the minimum number of participants for open card sorts performed on a general public website domain (e-commerce). In specific, it involves 203 and 210 participants sorting content items of two real-world e-commerce websites. Results from all the participants were compared with those of different-sized and randomly selected samples of the participants. It was found that 15 to 20 participants is a cost-effective way to obtain reliable open card sort data for general public websites.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Rosenfeld, L., Morville, P., Arango, J.: Information Architecture: For the Web And Beyond. O’Reilly Media, Sebastopol, CA (2015)

    Google Scholar 

  2. Spencer, D.: Card Sorting: Designing Usable Categories. Rosenfeld Media, Brooklyn, N.Y. (2009)

    Google Scholar 

  3. Adamides, G., Christou, G., Katsanos, C., Xenos, M., Hadzilacos, T.: Usability guidelines for the design of robot teleoperation: a taxonomy. IEEE Trans. Hum. Mach. Syst. 45, 256–262 (2015)

    Article  Google Scholar 

  4. Zaphiris, P., Ghiawadwala, M., Mughal, S.: Age-centered research-based web design guidelines. In: CHI 2005 Extended Abstracts on Human Factors in Computing Systems, pp. 1897–1900. ACM, New York, NY, USA (2005)

    Google Scholar 

  5. Cassidy, B., Antani, D.S., Read, J.C.C.: Using an open card sort with children to categorize games in a mobile phone application store. In: Proceedings of the 2013 CHI Conference on Human Factors in Computing Systems, pp. 2287–2290. ACM, New York, NY, USA (2013)

    Google Scholar 

  6. Jeong, R., Chiasson, S.: “Lime”, “open lock”, and “blocked”: children’s perception of colors, symbols, and words in cybersecurity warnings. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, p. 14 pages. ACM, New York, NY, USA (2020)

    Google Scholar 

  7. Kelley, C., Lee, B., Wilcox, L.: Self-tracking for mental wellness: understanding expert perspectives and student experiences. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 629–641. ACM, New York, NY, USA (2017)

    Google Scholar 

  8. Katsanos, C., Tselios, N., Goncalves, J., Juntunen, T., Kostakos, V.: Multipurpose public displays: can automated grouping of applications and services enhance user experience? Int. J. Hum. Comput. Interact. 30, 237–249 (2014)

    Article  Google Scholar 

  9. Katsanos, C., Tselios, N., Avouris, N.: Automated semantic elaboration of web site information architecture. Interact. Comput. 20, 535–544 (2008)

    Article  Google Scholar 

  10. Katsanos, C., Tselios, N., Avouris, N.: AutoCardSorter: designing the information architecture of a web site using latent semantic analysis. In: Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, CHI 2008, pp. 875–878. ACM, Florence, Italy (2008)

    Google Scholar 

  11. Harper, M.E., Jentsch, F., Van Duyne, L.R., Smith-Jentsch, K., Sanchez, A.D.: Computerized card sort training tool: is it comparable to manual card sorting? In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 2049–2053. SAGE (2002)

    Google Scholar 

  12. Petrie, H., Power, C., Cairns, P., Seneler, C.: Using card sorts for understanding website information architectures: technological, methodological and cultural issues. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6949, pp. 309–322. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23768-3_26

    Chapter  Google Scholar 

  13. Chaparro, B.S., Hinkle, V.D., Riley, S.K.: The usability of computerized card sorting: a comparison of three applications by researchers and end users. J. Usability Stud. 4, 31–48 (2008)

    Google Scholar 

  14. Melissourgos, G., Katsanos, C.: CardSorter: towards an open source tool for online card sorts. In: Proceedings of the 24th Pan-Hellenic Conference on Informatics, pp. 77–81. ACM, New York, NY, USA (2020)

    Google Scholar 

  15. Hudson, W.: Card sorting. In: Soegaard, M. and Dam, R.F. (eds.) The Encyclopedia of Human-Computer Interaction, 2nd edn. Interaction Design Foundation (2013)

    Google Scholar 

  16. Ntouvaleti, M., Katsanos, C.: Validity of the open card sorting method for producing website information structures. In: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, p. Article374:1-Article374:7. ACM, New York, NY, USA (2022)

    Google Scholar 

  17. Pampoukidou, S., Katsanos, C.: Test-retest reliability of the open card sorting method. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, p. Article330:1-Article330:7. ACM, New York, NY, USA (2021)

    Google Scholar 

  18. Katsanos, C., Tselios, N., Avouris, N., Demetriadis, S., Stamelos, I., Angelis, L.: Cross-study reliability of the open card sorting method. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, p. LBW2718:1-LBW2718:6. ACM, New York, NY, USA (2019)

    Google Scholar 

  19. Zafeiriou, G., Katsanos, C., Liapis, A.: Effect of sense of direction on open card sorts for websites. In: CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter, p. Article6:1-Article6:8. ACM, New York, NY, USA (2021)

    Google Scholar 

  20. Katsanos, C., Zafeiriou, G., Liapis, A.: Effect of self-efficacy on open card sorts for websites. In: Proceedings of the 24th International Conference on Human-Computer Interaction, HCI International 2022, pp. 75–87. Springer, Gothenburg, Sweden (2022)

    Google Scholar 

  21. Righi, C., et al.: Card sort analysis best practices. J. Usabil. Stud. 8, 69–89 (2013)

    Google Scholar 

  22. Paul, C.: Analyzing card-sorting data using graph visualization. J. Usabil. Stud. 9, 87–104 (2014)

    Google Scholar 

  23. Capra, M.G.: Factor analysis of card sort data: an alternative to hierarchical cluster analysis. In: Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting, pp. 691–695. HFES, Santa Monica, CA (2005)

    Google Scholar 

  24. Paea, S., Katsanos, C., Bulivou, G.: Information architecture: Using k-means clustering and the best merge method for open card sorting data analysis. Interact. Comput. 33, 670–689 (2021)

    Article  Google Scholar 

  25. Paea, S., Katsanos, C., Bulivou, G.: Information architecture: using best merge method, category validity, and multidimensional scaling for open card sort data analysis. Int. J. Hum. Comput. Interact. Forthcoming (2022)

    Google Scholar 

  26. Tullis, T., Wood, L.: How many users are enough for a card-sorting study? In: Usability Professionals Association (UPA) 2004 Conference (2004)

    Google Scholar 

  27. Nielsen, J.: Card Sorting: How many users to test. https://www.nngroup.com/articles/how-many-test-users/

  28. Lantz, E., Keeley, J.W., Roberts, M.C., Medina-Mora, M.E., Sharan, P., Reed, G.M.: Card sorting data collection methodology: how many participants is most efficient? J. Classif. 36, 649–658 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  29. Mantel, N.: The detection of disease clustering and a generalized regression approach. Can. Res. 27, 209–220 (1967)

    Google Scholar 

  30. Manly, B.F.J.: Randomization, Bootstrap and Monte Carlo Methods in Biology. Chapman & Hall, London (1997)

    MATH  Google Scholar 

  31. Wood, J., Wood, L.: Card sorting: current practices and beyond. J. Usabil. Stud. 4, 1–6 (2008)

    Google Scholar 

  32. Gates, A.J., Ahn, Y.-Y.: CluSim: a Python package for the comparison of clusterings and dendrograms. bioRxiv. 410084 (2018)

    Google Scholar 

  33. Gates, A.J., Wood, I.B., Hetrick, W.P., Ahn, Y.-Y.: Element-centric clustering comparison unifies overlaps and hierarchy. Sci. Rep. 9, 8574 (2019)

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank Optimal Workshop for kindly proving a free license to their OptimalSort software for conducting the open card sort studies reported in this paper.

Author information

Authors and Affiliations

  1. Department of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece

    Christos Pechlevanoudis, Grigorios Zilidis & Christos Katsanos

Authors
  1. Christos Pechlevanoudis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Grigorios Zilidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christos Katsanos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christos Katsanos .

Editor information

Editors and Affiliations

  1. University of West London, London, UK

    José Abdelnour Nocera

  2. Reykjavik University, Reykjavik, Iceland

    Marta Kristín Lárusdóttir

  3. University of York, York, UK

    Helen Petrie

  4. University of Bari Aldo Moro, Bari, Italy

    Antonio Piccinno

  5. Université Côte d’Azur, Sophia Antipolis Cedex, France

    Marco Winckler

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pechlevanoudis, C., Zilidis, G., Katsanos, C. (2023). How Many Participants Do You Need for an Open Card Sort? A Case Study of E-commerce Websites. In: Abdelnour Nocera, J., Kristín Lárusdóttir, M., Petrie, H., Piccinno, A., Winckler, M. (eds) Human-Computer Interaction – INTERACT 2023. INTERACT 2023. Lecture Notes in Computer Science, vol 14145. Springer, Cham. https://doi.org/10.1007/978-3-031-42293-5_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42293-5_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42292-8

  • Online ISBN: 978-3-031-42293-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation