research-article

Utility of human-computer interactions: toward a science of preference measurement

Authors:

Michael Toomim,

Travis Kriplean,

Claus Pörtner,

James LandayAuthors Info & Claims

CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 2275 - 2284

https://doi.org/10.1145/1978942.1979277

Published: 07 May 2011 Publication History

Abstract

The success of a computer system depends upon a user choosing it, but the field of Human-Computer Interaction has little ability to predict this user choice. We present a new method that measures user choice, and quantifies it as a measure of utility. Our method has two core features. First, it introduces an economic definition of utility, one that we can operationalize through economic experiments. Second, we employ a novel method of crowdsourcing that enables the collection of thousands of economic judgments from real users.

References

[1]

Ariely, D. Predictably Irrational: The Hidden Forces That Shape Our Decisions. HarperCollins. 2008.

[2]

Ben-Bassat, T., Meyer, J., and Tractinsky, N. Economic and Subjective Measures of the Perceived Value of Aesthetics and Usability. in ACM Transactions of Computer-Human Interaction, 13 (2). 2006. 210--234.

Digital Library

[3]

Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J-H., Beenen, M., Leaver-Fay, A., Baker, D., Popovic, Z. &#38; Foldit players. Predicting Protein Structures with a Multiplayer Online Game. Nature 466, 756--760.

[4]

Greene, W.H. Econometric Analysis. Macmillan. 1990.

[5]

Grudin, J. Utility and Usability: Research Issues and Development Contexts. Interacting with Computers, 4, (1992) 209--217.

[6]

Grudin J. and Wang Laboratories. Adapting a Psychophysical Method to Measure Performance and Preference Tradeoffs in Human-Computer Interaction. in Proc. INTERACT (1984). 737--741.

[7]

Heer, J. and Bostock, M. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design. in Proc. CHI 2010. (2000).

Digital Library

[8]

Horton, J.J. and Chilton, L. The Labor Economics of Paid Crowdsourcing. in Proc. ACM Electronic Commerce 2010. (2010).

Digital Library

[9]

Ipeirotis, P.G. Analyzing the Amazon Mechanical Turk Marketplace. New York University Report no. CeDER-10-04. 2010.

Digital Library

[10]

Ipeirotis, P.G. The Explosion of Micro-Crowdsourcing Services. http://behind-the-enemy-lines.blogspot.com/2010/10/explosion-of-micro-crowdsourcing.html.

[11]

Ipeirotis, P.G. Provost, F. and Wang, J. Quality Management on Amazon Mechanical Turk. In Proceedings of KDD-HCOMP (2010).

Digital Library

[12]

Ipeirotis, P.G. Why People Participate on Mechanical Turk, Now Tabulated. http://behind-the-enemy-lines.blogspot.com/2008/09/why-people-participate-on-mechanical.htm.

[13]

Kittur, A.; Chi, E. H. ; Suh, B. Crowdsourcing user studies with Mechanical Turk. In Proc. CHI 2008, ACM Press (2008). 453--456.

Digital Library

[14]

Klein, J.P. Survival Analysis. Springer, 2003.

[15]

Kohavi, R. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. in Proc KDD 2007. (2007.

Digital Library

[16]

MacKenzie, S. Fitts' Law as a Research and Design Tool. in Human-Computer Interaction (HCI), 1992 7(1), 91--139.

Digital Library

[17]

Malone, T.W. Heuristics for Designing Enjoyable User Interfaces: Lessons from Computer Games. in CHI 1982. ACM Press (1982).

Digital Library

[18]

Nielsen, J. Usability Engineering. Morgan Kaufmann, 2001.

[19]

Ross, J., Irani, L., Silberman, M.S., Zaldivar, A. and Tomlinson, B. Who are the Crowdworkers? Shifting Demographics in Mechanical Turk. alt.chi '10, ACM Press.

Digital Library

[20]

Simon, H.A. "Designing Organizations for an Information-Rich World. in Martin Greenberger," Computers, Communication, and the Public Interest, Baltimore, MD: The Johns Hopkins Press (1971).

[21]

Smith, A. An Inquiry into the Nature and Causes of the Wealth of Nations. London: Methuen and Co., Ltd., ed. Edwin Cannan, 1904.

[22]

Van Ahn, L. and Dabbish, L. Labeling Images with a Computer Game. in Proc. CHI 2004. ACM Press (2004).

Digital Library

[23]

Varian, H.R. Microeconomic Analysis. Norton, 1984.

[24]

Wilson, E. Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association. 1927. 22: 209--212.

Cited By

Haunschild JHenkel MReuter C(2025)Breaking down barriers to warning technology adoption: usability and usefulness of a messenger app warning boti-com10.1515/icom-2024-0067Online publication date: 26-Feb-2025
https://doi.org/10.1515/icom-2024-0067
He ZHuang CDing CRohatgi SHuang T(2024)If in a Crowdsourced Data Annotation Pipeline, a GPT-4Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642834(1-25)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642834
Bhatnagar JIyer RAgarwal S(2024)Formulating Explainable UI/UX for Bank ATM and Analytic Performance of HCI2024 International Conference on Emerging Techniques in Computational Intelligence (ICETCI)10.1109/ICETCI62771.2024.10704149(338-344)Online publication date: 22-Aug-2024
https://doi.org/10.1109/ICETCI62771.2024.10704149
Show More Cited By

Index Terms

Utility of human-computer interactions: toward a science of preference measurement

Index terms have been assigned to the content through auto-classification.

Recommendations

Do imposters threaten data quality? An examination of worker misrepresentation and downstream consequences in Amazon's Mechanical Turk workforce

Amazon's Mechanical Turk (MTurk) is fast becoming the most popular online research platform, and as such, it is crucial for researchers to recognize its advantages and shortcomings. Here, we focused on the issue of worker deception and examined the ...
Human computer interaction paradigm for business process task crowdsourcing
APCHI '13: Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction

In conventional systems like the web-based systems, device-based systems, or desktop systems, the Human Computer Interaction (HCI) spectrum encompasses of user interaction methods, interaction modes, user interface design, etc. that strive to facilitate ...
Do I Look Like a Criminal? Examining how Race Presentation Impacts Human Judgement of Recidivism
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Understanding how racial information impacts human decision making in online systems is critical in today's world. Prior work revealed that race information of criminal defendants, when presented as a text field, had no significant impact on users' ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

May 2011

3530 pages

ISBN:9781450302289

DOI:10.1145/1978942

General Chair:
Desney Tan
Microsoft Research
,
Program Chairs:
Geraldine Fitzpatrick
Vienna University of Technology
,
Carl Gutwin
University of Saskatchewan
,
Bo Begole
PARC
,
Wendy A. Kellogg
IBM Research

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '11

Sponsor:

SIGCHI

CHI '11: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2011

BC, Vancouver, Canada

Acceptance Rates

CHI '11 Paper Acceptance Rate 410 of 1,532 submissions, 27%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
1,142
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Haunschild JHenkel MReuter C(2025)Breaking down barriers to warning technology adoption: usability and usefulness of a messenger app warning boti-com10.1515/icom-2024-0067Online publication date: 26-Feb-2025
He ZHuang CDing CRohatgi SHuang T(2024)If in a Crowdsourced Data Annotation Pipeline, a GPT-4Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642834(1-25)Online publication date: 11-May-2024
Bhatnagar JIyer RAgarwal S(2024)Formulating Explainable UI/UX for Bank ATM and Analytic Performance of HCI2024 International Conference on Emerging Techniques in Computational Intelligence (ICETCI)10.1109/ICETCI62771.2024.10704149(338-344)Online publication date: 22-Aug-2024
Chuang YChang H(2024)Analyzing Novice and Competent Programmers' Problem-Solving Behaviors Using an Automated Evaluation SystemScience of Computer Programming10.1016/j.scico.2024.103138(103138)Online publication date: May-2024
Vasconcelos HJörke MGrunde-McLaughlin MGerstenberg TBernstein MKrishna R(2023)Explanations Can Reduce Overreliance on AI Systems During Decision-MakingProceedings of the ACM on Human-Computer Interaction10.1145/35796057:CSCW1(1-38)Online publication date: 16-Apr-2023
Islam MChaudhry B(2023)A Framework to Enhance User Experience of Older Adults With Speech-Based Intelligent Personal AssistantsIEEE Access10.1109/ACCESS.2022.323015111(16683-16699)Online publication date: 2023
Kokil UHarwood T(2022)The interplay between perceived usability and quality in visual design for tablet game interfacesJournal of Usability Studies10.5555/3542834.354283617:3(89-116)Online publication date: 1-May-2022
Abbas TGadiraju UKhan VMarkopoulos P(2022)Understanding User Perceptions of Response Delays in Crowd-Powered Conversational SystemsProceedings of the ACM on Human-Computer Interaction10.1145/35557656:CSCW2(1-42)Online publication date: 11-Nov-2022
Li ZZhao MDas DZHAO HMa YLiu WBeaudouin-Lafon MWang FRamakrishnan IBi X(2022)Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on TouchscreensProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517472(1-15)Online publication date: 29-Apr-2022
Payne SHowes A(2022)Adaptive InteractionundefinedOnline publication date: 4-Mar-2022
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten