Chien-Ju Ho
Kuan-Ta Chen
ABSTRACT
The introduction of the ESP Game and other Games With A Purpose (GWAP) has demonstrated the potential of human computation in solving AI-hard problems. In such systems, users are normally required to input answers for questions proposed by the system, e.g., descriptions about a picture or a song. Since users may bring up irrelevant inputs intentionally or carelessly, and often the system does not have "correct" answers, we have to rely on the users to verify answers from others. We call this kind of mutual verification of users' answers "social verification."
In this paper, we propose formal models for two fundamental social verification mechanisms, simultaneous verification and sequential verification, in human computation systems. By adopting a game-theoretic approach, we perform an equilibrium analysis which explains the effect of each verification mechanism on a system's outcome. Our analysis results show that sequential verification leads to a more diverse and descriptive set of outcomes than simultaneous verification, though the latter is stronger in ensuring the correctness of verified answers. Our experiments on Amazon Mechanical Turk, which asked users to input textual terms related to a word, confirmed our analysis results. We believe that our formal models for social verification mechanisms will provide a basis for the design of future human computation systems.
- Luis von Ahn. Games with a purpose. IEEE Computer Magazine, 39(6):92--94, 2006. Google Scholar
Digital Library
- Luis von Ahn and Laura Dabbish. Labeling images with a computer game. In CHI '04: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 319--326. ACM Press, 2004. Google ScholarDigital Library
- Ingmar Weber, Stephen Robertson, and Milan Vojnovic. Rethinking the ESP game. Technical report, Microsoft Research, September 2008.Google Scholar
- Luis von Ahn, Ruoran Liu, and Manuel Blum. Peekaboom: a game for locating objects in images. In CHI '06: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 55--64. ACM Press, 2006. Google ScholarDigital Library
- Luis von Ahn, Mihir Kedia, and Manuel Blum. Verbosity: a game for collecting common-sense facts. In CHI '06: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 75--78. ACM Press, 2006. Google ScholarDigital Library
- Edith L. M. Law, Luis von Ahn, Roger B. Dannenberg, and Mike Crawford. Tagatune: A game for music and sound annotation. In International Conference on Music Information Retrieval (ISMIR' 07), pages 361--364, 2003.Google Scholar
- Luis von Ahn and Laura Dabbish. Designing games with a purpose. Commun. ACM, 51(8):58--67, 2008. Google ScholarDigital Library
- Chien-Ju Ho, Tsung-Hsiang Chang, and Jane Yung-jen Hsu. Photoslap: A multi-player online game for semantic annotation. In Twenty-Second Conference on Artificial Intelligence (AAAI-07). AAAI Press, July 2007. Google ScholarDigital Library
- Shaili Jain and David C. Parkes. A game-theoretic analysis of games with a purpose. In Internet and Network Economics, 4th International Workshop (WINE 2008), pages 342--350, December 2008. Google ScholarDigital Library
- Martin J. Osborne and Ariel Rubinstein. A course in game theory. The MIT Press, Cambridge, Massachusetts, 1994.Google Scholar
- Thomas C. Schelling. The Strategy of Conflict. Harvard University Press, Cambridge, Massachusetts, 1960.Google Scholar
- Robert Sugden. A theory of focal points. The Economic Journal, 105(430):533--550, 1995.Google Scholar
- Severin Hacker and Luis von Ahn. Matchin: Eliciting user preferences with an online game. In CHI '09: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press, 2009. Google ScholarDigital Library
Index Terms
- On formal models for social verification
Recommendations
KissKissBan: a competitive human computation game for image annotation
HCOMP '09: Proceedings of the ACM SIGKDD Workshop on Human ComputationIn this paper, we propose a competitive human computation game, KissKissBan (KKB), for image annotation. KKB is different from other human computation games since it integrates both collaborative and competitive elements in the game design. In a KKB ...
Formal verification of ASMs using MDGs
We present a framework for the formal verification of abstract state machine (ASM) designs using the multiway decision graphs (MDG) tool. ASM is a state based language for describing transition systems. MDG provides symbolic representation of transition ...
Coverage metrics for formal verification
In formal verification, we verify that a system is correct with respect to a specification. Even when the system is proven to be correct, there is still a question of how complete the specification is and whether it really covers all the behaviors of ...
Comments