Christopher G. Harris
ABSTRACT
In this paper, we introduce VisualizIR, a game where players identify relevant document terms that match predefined categories. VisualizIR evaluates players on accuracy, recall, and precision against an established gold standard, a pooled consensus of judgments made by other players, or a weighted combination of the two. The annotated document can then viewed by any XML-compatible browser, allowing for quick identification of terms in the document related to each category. Here we describe some of the playability design tradeoffs made during the game's development, as well as our findings from two experiments conducted using VisualizIR output.
- Csikszentmihalyi, M. Flow: The psychology of optimal experience. Harper Perennial, New York, 1991.Google Scholar
- Eickhoff, C., Harris, C. G., de Vries, A. P., and Srinivasan, P. Quality through flow and immersion: gamifying crowdsourced relevance assessments. In Proc of the 35th Annual ACM Conference on Research and Development in Information Retrieval (SIGIR'12). ACM, New York. 2012. Google ScholarDigital Library
- Medlock, B. Investigating classification for natural language processing tasks. VDM Verlag, 2008.Google Scholar
- Prabowo, R. and Thelwall, M. Sentiment analysis: A combined approach. Journal of Informetrics, 3:2, 2009. pp 143--157Google ScholarCross Ref
- Snow, R., O'Connor, B., Jurafsky, D. and Ng, A. Y. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In Proc of the Conference on Empirical Methods in Natural Language Processing. ACL, Stroudsburg, PA, USA, 2008. Google ScholarDigital Library
- von Ahn, L. and Dabbish, L. Designing games with a purpose. Comm of the ACM, 51:8, 2008. pp 58--67. Google ScholarDigital Library
Index Terms
- VisualizIR: a game for identifying and categorizing relevant text in documents
Recommendations
Reconstructing the world in 3D: bringing games with a purpose outdoors
FDG '10: Proceedings of the Fifth International Conference on the Foundations of Digital GamesWe are interested in reconstructing real world locations as detailed 3D models, but to achieve this goal, we require a large quantity of photographic data. We designed a game to employ the efforts and digital cameras of everyday people to not only ...
Reconstructing the world in 3D: bringing games with a purpose outdoors
HCOMP '10: Proceedings of the ACM SIGKDD Workshop on Human ComputationWe are interested in reconstructing real world locations as detailed 3D models, but to achieve this goal, we require a large quantity of photographic data. We designed a game to employ the efforts and digital cameras of everyday people to not only ...
Matching Game Mechanics and Human Computation Tasks in Games with a Purpose
SeriousGames '14: Proceedings of the 2014 ACM International Workshop on Serious GamesThe massive amount of time that people spend in online gaming is being increasingly exploited by a particular kind of Serious Games, the Games with a Purpose (GWAP), used to solve complex problems as a byproduct of their gameplay. The design of the ...
Comments