Patrick Minder
Abraham Bernstein
ABSTRACT
In this paper we present the programming language and framework CrowdLang for engineering complex computation systems incorporating large numbers of networked humans and machines agents. We evaluate CrowdLang by developing a text translation program incorporating human and machine agents. The evaluation shows that we are able to simply explore a large design space of possible problem solving programs with the simple variation of the used abstractions. Furthermore, an experiment, involving 1918 different human actors, shows that the developed mixed human-machine translation program significantly outperforms a pure machine translation in terms of adequacy and fluency whilst translating more than 30 pages per hour and that the program approximates the professional translated gold-standard to 75% using the automatic evaluation metric METEOR. Last but not least, our evaluation illustrates that our new human computation pattern staged-contest with pruning outperforms all other refinements in the translation task.
- Ahmad, S., Battle, A., Malkani, Z., and Kamvar, S. The jabberwocky programming environment for structured social computing. In Proc. of the 24th annual ACM symposium on User interface software and technology (2011). Google Scholar
Digital Library
- Banerjee, S., and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (2005), 65.Google Scholar
- Bernstein, A. How can cooperative work tools support dynamic group process? bridging the specificity frontier. In Proc. of the ACM conference on Computer supported cooperative work (2000). Google ScholarDigital Library
- Bernstein, A., Klein, M., and Malone, T. The process recombinator: a tool for generating new business process ideas. In Proc. of the 20th international conference on Information Systems (1999). Google ScholarDigital Library
- Bernstein, A., Klein, M., and Malone, T. Programming the global brain. Communications of the ACM 55, 5 (2012), 1--4. Google ScholarDigital Library
- Bernstein, M., Little, G., Miller, R., Hartmann, B., Ackerman, M., Karger, D., Crowell, D., and Panovich, K. Soylent: a word processor with a crowd inside. In Proc. of the 23nd annual ACM symposium on User interface software and technology (2010). Google ScholarDigital Library
- Chase, R., Aquilano, N., and Jacobs, F. Operations management for competitive advantage. McGraw-Hill/Irwin New York, 2006.Google Scholar
- Chen, Y., Liem, B., and Zhang, H. An iterative dual pathway structure for speech-to-text transcription. In Human Computation: AAAI Workshop (2011).Google Scholar
- Dean, J., and Ghemawat, S. Mapreduce: Simplified data processing on large clusters. Communications of the ACM 51 (2008). Google ScholarDigital Library
- Kittur, A., Smus, B., Khamkar, S., and Kraut, R. Crowdforge: Crowdsourcing complex work. In Proc. of the 24th annual ACM symposium on User interface software and technology (2011). Google ScholarDigital Library
- Little, G., Chilton, L., Goldman, M., and Miller, R. Turkit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (2010). Google ScholarDigital Library
- Malone, T., and Crowston, K. The interdisciplinary study of coordination. ACM Computing Surveys 26 (1994). Google ScholarDigital Library
- Malone, T., Laubacher, R., and Dellarocas, C. The collective intelligence genome. MIT Sloan Management Review 51 (2010).Google Scholar
- Malone, T., Laubacher, R., and Johns, T. General management: The age of hyperspecialization. Harvard Business Review 89, 7--8 (2011), 56--65.Google Scholar
- Minder, P., and Bernstein, A. Crowdlang: Programming human computation systems - interweaving human and machine intelligence in a complex translation task. Tech. rep., University of Zurich, 2012.Google Scholar
- Noronha, J., Hysen, E., Zhang, H., and Gajos, K. Platemate: crowdsourcing nutritional analysis from food photographs. In Proc. of the 24th annual ACM symposium on User interface software and technology, ACM (2011), 1--12. Google ScholarDigital Library
- Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., and Horvitz, E. Human computation tasks with global constraints. CHI (2012). Google ScholarDigital Library
Index Terms
- How to translate a book within an hour: towards general purpose programmable human computers with CrowdLang
Recommendations
Crowdsourced Monolingual Translation
An enormous potential exists for solving certain classes of computational problems through rich collaboration among crowds of humans supported by computers. Solutions to these problems used to involve human professionals, who are expensive to hire or ...
Divide and translate: improving long distance reordering in statistical machine translation
WMT '10: Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATRThis paper proposes a novel method for long distance, clause-level reordering in statistical machine translation (SMT). The proposed method separately translates clauses in the source sentence and reconstructs the target sentence using the clause ...
CrowdLang: a programming language for the systematic exploration of human computation systems
SocInfo'12: Proceedings of the 4th international conference on Social InformaticsHuman computation systems are often the result of extensive lengthy trial-and-error refinements. What we lack is an approach to systematically engineer solutions based on past successful patterns.
In this paper we present the CrowdLang programming ...
Comments