skip to main content
10.1145/1753326.1753686acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Powerful and consistent analysis of likert-type rating scales

Published: 10 April 2010 Publication History

Abstract

Likert-type scales are used extensively during usability evaluations, and more generally evaluations of interactive experiences, to obtain quantified data regarding attitudes, behaviors, and judgments of participants. Very often this data is analyzed using parametric statistics like the Student t-test or ANOVAs. These methods are chosen to ensure higher statistical power of the test (which is necessary in this field of research and practice where sample sizes are often small), or because of the lack of software to handle multi-factorial designs nonparametrically. With this paper we present to the HCI audience new developments from the field of medical statistics that enable analyzing multiple factor designs nonparametrically. We demonstrate the necessity of this approach by showing the errors in the parametric treatment of nonparametric data in experiments of the size typically reported in HCI research. We also provide a practical resource for researchers and practitioners who wish to use these new methods.

References

[1]
C. A. Boneau. The effects of violations of assumptions underlying the t test. Psychological Bulletin, 57:49--64, 1960.
[2]
E. Brunner, S. Domhof, and F. Langer. Nonparametric Analysis of Longitudinal Data in Factorial Experiments, volume Total Quality Management. John Wiley & Sons, 2002.
[3]
E. Brunner and U. Munzel. Nichtparametrische Datenanalyse. Springer, 2002.
[4]
E. Brunner, U. Munzel, and M. L. Puri. Rank-score tests in factorial designs with repeated measures. Journal of Multivariate Analysis, 70:286--317, 1999.
[5]
E. Brunner and M. L. Puri. Nonparametric methods in factorial designs. Statistical Papers, 42:1--51, 2001.
[6]
J. Carifio and R. Perla. Resolving the 50-year debate around using and misusing Likert scales. Med Educ, 42:1150--52.
[7]
J. Carifio and R. Perla. Ten Common Misunderstandings, Misconceptions, Persistent Myths and Legends about Likert Scales and Likert Response Formats and their Antidotes. Journal of Social Sciences, 3:106--116.
[8]
R. A. Cribbie and H. J. Keselman. The effects of nonnormality on parametric, nonparametric, and model comparison approaches to pairwise comparisons. Educational and Psychological Measurement, 63:615--635, 2003.
[9]
G. V. Glass, P. D. Peckham, and J. R. Sanders. Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research, 42(3):237--288, 1972.
[10]
M. C. Hart. Improving the discrimination of SERVQUAL by using magnitude scaling, volume Total Quality Management. Chapman & Hall, 1996.
[11]
M. Hassenzahl. The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction, 19:319--349, 2004.
[12]
S. Hora and W. Conover. The f-statistic in the two-way layout with rank-score transformed data. Journal of the American Statistical Association, 79:668--673, 1984.
[13]
R. Likert. A technique for the measurement of attitudes. Journal of Social Psychology, 5:228--238, 1932.
[14]
U. Munzel and B. Bandelow. The use of parametric versus nonparametric tests in the statistical evaluation of ratings scales. Pharmacopsychiatry, 6:222--224, 1998.
[15]
J. Preece, Y. Rogers, and Sharp. Interaction Design. John Wiley & Sons, 2002.
[16]
S. S. Sawilowsky and R. C. Blair. A more realistic look at the robustness and type ii error probabilities of the t test to departures from population normality. Psychological Bulletin, 111:352--360, 1992.
[17]
D. A. Shah and L. V. Madden. Nonparametric analysis of ordinal data in designed factorial experiments. Pythopathology, 94:33--60, 2004.
[18]
T. Tullis and W. Albert. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufman, 2008.
[19]
R. R. Wilcos. Comparing the means of two independent groups. Biometrics Journal, Biometrics Journal, 32:771--780, 1990.

Cited By

View all
  • (2024)Görme Yetersizliği Olan Öğrenciler ile İlgili Tezlerdeki Genel EğilimlerBayburt Eğitim Fakültesi Dergisi10.35675/befdergi.138408719:43(2627-2655)Online publication date: 30-Sep-2024
  • (2024)Experience Report: Meet the Professor - A Large-Course Intervention for Increasing RapportProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630844(415-421)Online publication date: 7-Mar-2024
  • (2024)Beyond Aesthetics: Evaluating Response Widgets for Reliability & Construct Validity of Scale QuestionnairesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650751(1-7)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. Powerful and consistent analysis of likert-type rating scales

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    April 2010
    2690 pages
    ISBN:9781605589299
    DOI:10.1145/1753326
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 April 2010

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. nonparametric statistics
    2. researchmethods
    3. usability evaluation

    Qualifiers

    • Research-article

    Conference

    CHI '10
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)254
    • Downloads (Last 6 weeks)27
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Görme Yetersizliği Olan Öğrenciler ile İlgili Tezlerdeki Genel EğilimlerBayburt Eğitim Fakültesi Dergisi10.35675/befdergi.138408719:43(2627-2655)Online publication date: 30-Sep-2024
    • (2024)Experience Report: Meet the Professor - A Large-Course Intervention for Increasing RapportProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630844(415-421)Online publication date: 7-Mar-2024
    • (2024)Beyond Aesthetics: Evaluating Response Widgets for Reliability & Construct Validity of Scale QuestionnairesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650751(1-7)Online publication date: 11-May-2024
    • (2024)A tool to evaluate industrial cobot safety readiness from a system-wide perspective: An empirical validationSafety Science10.1016/j.ssci.2023.106380170(106380)Online publication date: Feb-2024
    • (2023)Generalized Estimating Equations in Longitudinal Studies: A Non-Parametric Alternative for Two-Way Repeated Measures Mixed ANOVAResearch Journal of Pharmacy and Technology10.52711/0974-360X.2023.00392(2381-2384)Online publication date: 31-May-2023
    • (2023)Using A/B Testing as a Pedagogical Tool for Iterative Design in HCI ClassroomsProceedings of the 5th Annual Symposium on HCI Education10.1145/3587399.3587412(43-48)Online publication date: 28-Apr-2023
    • (2022)Akran Eğitimi Yaklaşımının Fen Bilimleri Ders Planlama Sürecine Etkisi: Öğretmenlik Uygulaması ÖrneğiYuzunci Yil Universitesi Egitim Fakultesi Dergisi10.33711/yyuefd.1068094(155-181)Online publication date: 15-Jul-2022
    • (2022)Recruiting Participants With Programming Skills: A Comparison of Four Crowdsourcing Platforms and a CS Student Mailing ListProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501957(1-15)Online publication date: 29-Apr-2022
    • (2022)Effective Use of Likert Scales in Visualization Evaluations: A Systematic ReviewComputer Graphics Forum10.1111/cgf.1452141:3(43-55)Online publication date: 12-Aug-2022
    • (2022)Gestural interaction in the kitchen: Insights into designing an interactive display controlled by hand specific on-skin gesturesThe Design Journal10.1080/14606925.2022.205844425:3(353-373)Online publication date: 14-Apr-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media