skip to main content
10.1145/1518701.1518801acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Accounting for diversity in subjective judgments

Published: 04 April 2009 Publication History

Abstract

In this paper we argue against averaging as a common practice in the analysis of subjective attribute judgments, both across and within subjects. Previous work has raised awareness of the diversity between individuals' perceptions. In this paper it will furthermore become apparent that such diversity can also exist within a single individual, in the sense that different attribute judgments from a subject may reveal different, complementary, views. A Multi-Dimensional Scaling approach that accounts for the diverse views on a set of stimuli is proposed and its added value is illustrated using published data. We will illustrate that the averaging analysis provides insight to only 1/6th of the total number of attributes in the example dataset. The proposed approach accounts for more than double the information obtained from the average model, and provides richer and semantically diverse views on the set of stimuli.

References

[1]
Al-Azzawi, A., Frohlich, D., and Wilson, M., Beauty constructs for MP3 players. CoDesign, 2007. 3(1 supp 1): p. 59 -- 74.
[2]
Bech-Larsen, T. and Nielsen, N.A., A comparison of five elicitation techniques for elicitation of attributes of low involvement products. Journal of Economic Psychology, 1999. 20: p. 315--341.
[3]
Breivik, E. and Supphellen, M., Elicitation of product attributes in an evaluation context: A comparison of three elicitation techniques. Journal of Economic Psychology, 2003. 24: p. 77--98.
[4]
Burnham, K.P. and Anderson, D.R., Multimodel Inference: Understanding AIC and BIC in Model Selection. Sociological Methods&Research, 2004. 33(2): p. 261.
[5]
Collins, A.M. and Loftus, E.F., A spreading-activation theory of semantic processing. Psychological Review, 1975. 82(6): p. 407--428.
[6]
Davis, S.B. and Carini, C., Constructing a Player-Centred Definition of Fun for Video Games Design. HCI 2004 Conference, 2004: p. 117--132.
[7]
Draper, N.R. and Smith, H., Applied Regression Analysis. 1998, New Yoek: John Wiley and Sons Inc.
[8]
Egger, F.N., " Trust me, I'm an online vendor": towards a model of trust for e--commerce system design. Conference on Human Factors in Computing Systems, 2000: p. 101--102.
[9]
Fällman, D., In Romance with the Materials of Mobile Interaction: A Phenomenological Approach to the Design of Mobile Information Technology. 2003: Univ.
[10]
Fransella, F., Bell, R., and Bannister, D., A Manual for Repertory Grid Technique 2003: Wiley.
[11]
Frøkjær, E., Hertzum, M., and Hornbæk, K., Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?, in Proceedings of the SIGCHI conference on Human factors in computing systems. 2000, ACM Press: The Hague, The Netherlands.
[12]
Green, P.E., Carmone Jr., F.J., and Smith, S.M., Multi-dimensional Scaling, Concepts and Applications. 1989, Boston, MA: Allyn&Bacon.
[13]
Hassenzahl, M. and Wessler, R., Capturing design space from a user perspective: The Repertory Grid Technique revisited. International Journal of Human-Computer Interaction, 2000. 12(3-4): p. 441--459.
[14]
Hassenzahl, M. and Trautmann, T., Analysis of web sites with the repertory grid technique, in CHI '01 2001, ACM Press: Seattle, Washington.
[15]
Hassenzahl, M., Character Grid: a Simple Repertory Grid Technique for Web Site Analysis and Evaluation, in Human Factors and Web Development. Lawrence Erlbaum, J. Ratner, Editor. 2002: Mahwah, NJ. p. 183--206.
[16]
Hassenzahl, M., The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interaction, 2004. 19(4): p. 319--349.
[17]
Hassenzahl, M. and Sandweg, N., From mental effort to perceived usability: transforming experiences into summary assessments, in CHI '04 extended abstracts on Human factors in computing systems. 2004, ACM Press: Vienna, Austria.
[18]
Hassenzahl, M. and Tractinsky, N., User experience - a research agenda. Behaviour&Information Technology, 2006. 25(2): p. 91--97.
[19]
Hassenzahl, M., Aesthetics in interactive products: correlates and consequences of beauty, in Product Experience, Elsevier, Amsterdam, H.N.J. Schifferstein and P. Hekkert, Editors. 2007.
[20]
Hassenzahl, M. and Ullrich, D., To do or not to do: Differences in user experience and retrospective judgments depending on the presence or absence of instrumental goals. Interacting with Computers, 2007. 19: p. 429--437.
[21]
Heidecker, S. and Hassenzahl, M., Eine gruppenspezifische Repertory Grid Analyse der wahrgenommenen Attraktivität von Universitätswebsites, in Mensch&Computer 2007: Konferenz füür interative und kooperative Medien, T.Gross, Editor. 2007: Oldenbourg, Munich. p. 129--138.
[22]
Hinkin, T.R., A Review of Scale Development Practices in the Study of Organizations. Journal of Management, 1995. 21(5): p. 967.
[23]
Jordan, P.W., Designing Pleasurable Products: An Introduction to New Human Factors. 2000, London: Taylor&Francis.
[24]
Karapanos, E. and Martens, J.-B., Characterizing the Diversity in Users' Perceptions, in Human-Computer Interaction - INTERACT 2007. 2007, Springer. p. 515--518.
[25]
Karapanos, E., Hassenzahl, M., and Martens, J.-B., User experience over time, in CHI '08 extended abstracts on Human factors in computing systems. 2008, ACM: Florence, Italy.
[26]
Karapanos, E. and Martens, J.-B., The quantitative side of the Repertory Grid Technique: some concerns, in in the proceedings of the workshop Now Let's Do It in Practice: User Experience Evaluation Methods in Product Development, Human factors in computing systems, CHI'08. 2008: Florence.
[27]
Karapanos, E., Wensveen, S.A.G., Friederichs, B.H.W., and Martens, J.-B., Do knobs have character? Exploring diversity in users' inferences in CHI'08 extended abstracts on Human factors in computing systems. 2008, ACM Press: Florence.
[28]
Karapanos, E., Zimmerman, J., Forlizzi, J., and Martens, J.-B., User Experience Over Time: An initial framework, in Proceedings of the Twenty-Seventh Annual SIGCHI Conference on Human Factors in Computing Systems - CHI '09. 2009, ACM: Boston.
[29]
Markopoulos, P., Romero, N., van Baren, J., Ijsselsteijn, W., de Ruyter, B., and Farshchian, B. Keeping in touch with the family: home and away with the ASTRA awareness system. 2004: ACM Press New York, NY, USA.
[30]
Martens, J.-B., Image technology design: A perceptual approach. 2003, Boston: Kluwer Academic Publisher.
[31]
Osgood, C.E., Suci, G., and Tannenbaum, P., The measurement of meaning. 1957, Urbana, IL: University of Illinois Press.
[32]
Ramsay, J.O., MATFIT: A Fortran Subroutine for Comparing Two Matriced in a Subspace. Psychometrika, 1990. 55(3): p. 551--553.
[33]
Sarstedt, M., A review of recent approaches for capturing heterogeneity in partial least squares path modelling. Journal of Modelling in Management, 2008. 3(2): p. 140--161.
[34]
Schenkman, B.N. and Jönsson, F.U., Aesthetics and preferences of Web pages. Behaviour&Information Technology, 2000. 19: p. 367--377.
[35]
Steenkamp, J.-B.E.M. and Van Trijp, H.C.M., Attribute elicitation in Marketing Research: A comparison of Three Procedures. Marketing Letters, 1997. 8:2: p. 153--165.
[36]
van Kleef, E., van Trijp, H.C.M., and Luning, P., Consumer research in the early stages of new product development: a critical review of methods and techniques. Food Quality and Preference, 2005. 16(3): p. 181--201.

Cited By

View all
  • (2024)UX Design and Evaluation: Holism versus Reductionism Approaches - Which is Better?2024 IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI)10.1109/IATMSI60426.2024.10503367(1-6)Online publication date: 14-Mar-2024
  • (2020)Why did you pick that? A study on smartwatch design qualities and people’s preferencesBehaviour & Information Technology10.1080/0144929X.2020.183625941:4(827-844)Online publication date: 23-Oct-2020
  • (2019)Interpreting the Diversity in Subjective JudgmentsProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300449(1-11)Online publication date: 2-May-2019
  • Show More Cited By

Index Terms

  1. Accounting for diversity in subjective judgments

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    April 2009
    2426 pages
    ISBN:9781605582467
    DOI:10.1145/1518701
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 04 April 2009

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. multi-dimensional scaling
    2. quantitative methods
    3. repertory grid
    4. subjective judgments
    5. user experience

    Qualifiers

    • Research-article

    Conference

    CHI '09
    Sponsor:

    Acceptance Rates

    CHI '09 Paper Acceptance Rate 277 of 1,130 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)14
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 23 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)UX Design and Evaluation: Holism versus Reductionism Approaches - Which is Better?2024 IEEE International Conference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI)10.1109/IATMSI60426.2024.10503367(1-6)Online publication date: 14-Mar-2024
    • (2020)Why did you pick that? A study on smartwatch design qualities and people’s preferencesBehaviour & Information Technology10.1080/0144929X.2020.183625941:4(827-844)Online publication date: 23-Oct-2020
    • (2019)Interpreting the Diversity in Subjective JudgmentsProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300449(1-11)Online publication date: 2-May-2019
    • (2018)Exploring the Visualization Design Space with Repertory GridsComputer Graphics Forum10.1111/cgf.1340737:3(133-144)Online publication date: 10-Jul-2018
    • (2018)Towards Individual QoE for Multiparty VideoconferencingIEEE Transactions on Multimedia10.1109/TMM.2017.277746620:7(1781-1795)Online publication date: Jul-2018
    • (2018)Assessing the performance of short multi-item questionnaires in aesthetic evaluation of websitesBehaviour & Information Technology10.1080/0144929X.2018.153952138:5(469-485)Online publication date: 30-Oct-2018
    • (2017)Betas: Deriving quantiles from MOS-QoS relations of IQX models for QoE management2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM)10.23919/INM.2017.7987430(1011-1016)Online publication date: May-2017
    • (2017)Data-Driven Modules for Objective Visual Quality Assessment Focusing on Benchmarking and SLAsIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2016.263716411:1(196-205)Online publication date: Feb-2017
    • (2015)Journey through the crowd: Best practices and recommendations for crowdsourced QoE2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX.2015.7148150(1-2)Online publication date: May-2015
    • (2015)QoE beyond the MOS: Added value using quantiles and distributions2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX.2015.7148142(1-6)Online publication date: May-2015
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media