skip to main content
10.1145/1868914.1868959acmotherconferencesArticle/Chapter ViewAbstractPublication PagesnordichiConference Proceedingsconference-collections
research-article

Five design challenges for human computation

Published: 16 October 2010 Publication History

Abstract

Human computation systems, which draw upon human competencies in order to solve hard computational problems, represent a growing interest within HCI. Despite the numerous technical demonstrations of human computation systems, however, there are few design guidelines or frameworks for researchers or practitioners to draw upon when constructing such a system. Based upon findings from our own human computation system, and drawing upon those published within HCI, and from other scientific and engineering literatures, as well as systems deployed commercially, we offer a framework of five challenging issues of relevance to designers of systems with human computation elements: designing the motivation of participants in the human computation system and sustaining their engagement; orienting participants, framing and orienting participants; using situatedness as a driver for content generation; considering the organisation of human and machine roles in human computation systems; and reconsidering the way in which computational analogies are applied to the design space of human computation.

References

[1]
Bell, M., et al. EyeSpy: supporting navigation through play. In Proc. CHI, pp. 123--132, ACM Press, 2009.
[2]
Benford, S. et al. The frame of the game: Blurring the boundary between fiction and reality in mobile experiences. In Proc. CHI, pp. 427--436, ACM Press, 2006.
[3]
Bernstein, M. et al. Collabio: a game for annotating people within social networks. In Proc UIST, pp. 97--100, ACM Press, 2009.
[4]
Butterfield, D. S. et al. Interestingness ranking of media objects. U.S. Patent application 2006/0242139 A1, February 8, 2006.
[5]
Dawkins, R. The Blind Watchmaker. Longman, 1986.
[6]
Fails, J. A. and Olsen, D. R. 2003. Interactive machine learning. In Proc. IUI '03. ACM, New York, NY, 39--45.
[7]
FoldIt website, http://fold.it, verified 17/09/09.
[8]
Galaxy Zoo website, http://www.galaxyzoo.org, verified 26/02/10.
[9]
Games With A Purpose website, http://www.gwap.com, verified 26/02/10.
[10]
Garfinkel, H. Some rules of correct decision making that jurors respect. In Studies in Ethnomethodology, pp. 104--115, 1967, Prentice Hall.
[11]
Goldin, D. et al. Interactive Computation: The New Paradigm. Springer Verlag: 2006.
[12]
Grimes, A. et al. EatWell: sharing nutrition-related memories in a low-income community. In Proc. CSCW '08. ACM, New York, NY, 87--96, 2008.
[13]
Hacker, S. and von Ahn, L. Matchin: eliciting user preferences with an online game. In Proc. CHI '09. ACM, New York, NY, 1207--1216, 2009.
[14]
Howe, J. The Rise of Crowdsourcing, Wired, June 2006, http://www.wired.com/wired/archive/14.06/crowds.html, verified 26/02/10.
[15]
Koleva, B. et al. Orchestrating a mixed reality performance. In Proc. ACM CHI, pp. 38--45, 2001.
[16]
Kosorukoff, A. Human-based genetic algorithm, http://www.geocities.com/alex+kosorukoff/hbga/hbga.html, verified 26/02/10.
[17]
Law, E., von Ahn, L. Input-agreement: A New Mechanism for Data Collection using Human Computation Games. In Proc. CHI, pp. 1197--1206, 2009.
[18]
Lintott C. J., et al. Galaxy Zoo: 'Hanny's Voorwerp', a quasar light echo?, submitted to MNRAS.
[19]
Mechanical Turk website, http://www.mturk.com, verified 26/02/10.
[20]
Paulos, E., Honicky, R. and Hooker, B. Citizen science: Enabling participatory urbanism. In M. Foth, editor, Handbook of Research on Urban Informatics. IGI Global, 2008.
[21]
Robinson, M. Design for unanticipated use. In Proc. ECSCW. Kluwer, pp. 187--202, 1993.
[22]
Rowland, D. et al. Ubikequitous computing: designing interactive experiences for cyclists. In Proc. MobileHCI. ACM Press, pp. 1--11, 2009.
[23]
Stardust@home website, http://stardustathome.ssl.berkeley.edu, verified 26/02/10.
[24]
Suchman, L. Plans and situated actions: The problem of human-machine communication. Cambridge University Press, 1987
[25]
Surowiecki, J. Wisdom of Crowds, Random House, 2004.
[26]
Takagi, H. Interactive evolutionary computation: fusion of the capabilities of EC optimization and human evaluation, In Proc. of the IEEE, vol. 89, no. 9, pp. 1275--1296, Sep 2001.
[27]
TxtEagle, http://www.newscientist.com/article/mg20126956>600-translations-by-text.html, verified 26/02/10.
[28]
von Ahn, L. and Dabbish, L. Labeling images with a computer game. In Proc ACM CHI, pp. 319--326, 2004.
[29]
von Ahn, L. and Dabbish, L. Designing games with a purpose. Communications of the ACM, 51(8): 58--67, 2008.
[30]
von Ahn, L., Kedia, M., and Blum, M. Verbosity: a game for collecting common-sense facts. In Proc. CHI '06. ACM Press, pp. 75--78, 2006.
[31]
von Ahn, L., Liu, R., and Blum, M. Peekaboom: a game for locating objects in images. In Proc. CHI, pp. 55--64, ACM Press, 2006.
[32]
von Ahn, L., Maurer, B., McMillen, C., Abraham, D. and Blum, M. reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science, 321 (5895): 1465--1468, 2008.
[33]
Wegner, P. Why interaction is more powerful than algorithms, Communications of the ACM, 40(5), May 1997, pp. 80--91.
[34]
Zittrain, J. Ubiquitous human computing, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 366, No. 1881, pp. 3813--3821, 2008.

Cited By

View all
  • (2022)The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop FailuresProceedings of Mensch und Computer 202210.1145/3543758.3543761(158-168)Online publication date: 4-Sep-2022
  • (2019)Towards Hybrid Crowd-AI Centered Systems: Developing an Integrated Framework from an Empirical Perspective2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)10.1109/SMC.2019.8914075(4013-4018)Online publication date: Oct-2019
  • (2018)Intrinsic elicitationProceedings of the 13th International Conference on the Foundations of Digital Games10.1145/3235765.3235803(1-10)Online publication date: 7-Aug-2018
  • Show More Cited By

Index Terms

  1. Five design challenges for human computation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    NordiCHI '10: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries
    October 2010
    889 pages
    ISBN:9781605589343
    DOI:10.1145/1868914
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    • Reykjavik University
    • University of Iceland

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 16 October 2010

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. citizen science
    2. crowdsourcing
    3. design framework
    4. games with a purpose
    5. human computation

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    NordiCHI '10
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 379 of 1,572 submissions, 24%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop FailuresProceedings of Mensch und Computer 202210.1145/3543758.3543761(158-168)Online publication date: 4-Sep-2022
    • (2019)Towards Hybrid Crowd-AI Centered Systems: Developing an Integrated Framework from an Empirical Perspective2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)10.1109/SMC.2019.8914075(4013-4018)Online publication date: Oct-2019
    • (2018)Intrinsic elicitationProceedings of the 13th International Conference on the Foundations of Digital Games10.1145/3235765.3235803(1-10)Online publication date: 7-Aug-2018
    • (2016)Design of CQA Systems for Flexible and Scalable Deployment and EvaluationWeb Engineering10.1007/978-3-319-38791-8_30(439-447)Online publication date: 25-May-2016
    • (2014)Competing or aiming to be average?Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing10.1145/2531602.2531615(1222-1233)Online publication date: 15-Feb-2014
    • (2014)ConclusionThe Fundamentals of Human Factors Design for Volunteered Geographic Information10.1007/978-3-319-03503-1_7(121-134)Online publication date: 4-Jan-2014
    • (2013)Human-Computer Interaction Issues in Human ComputationHandbook of Human Computation10.1007/978-1-4614-8806-4_32(411-419)Online publication date: 20-Nov-2013
    • (2013)Pervasive Human ComputingHandbook of Human Computation10.1007/978-1-4614-8806-4_27(333-345)Online publication date: 20-Nov-2013
    • (2011)Understanding Location-Based Information Sharing in a Mobile Human Computation GameProceedings of the 2011 International Conference on Internet of Things and 4th International Conference on Cyber, Physical and Social Computing10.1109/iThings/CPSCom.2011.48(209-216)Online publication date: 19-Oct-2011

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media