Abstract
Recently, attempts have been to request translation services from anonymous crowds. Compared with platform-based language services, services performed by humans have the advantages of flexibility and quality. However, due to the nature of human services the results are not consistent and quality is not assured. This research tries to solve this problem by creating workflows that make collaboration among crowdsourcing workers far more effective and efficient. We model workers and tasks, and calculate the optimal workflow. To confirm the feasibility of this model, we conduct a computational experiment to calculate the best workflow under various parameters. The results are consistent with existing research so the model is useful in understanding crowdsourcing workflows. In addition, a system is developed for realizing crowdsourcing workflows for translation services. Finally, we develop a translation interface that demonstrates the feasibility of the proposed method.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Ambati, V., Vogel, S., Carbonell, J.G.: Active learning and crowd-sourcing for machine translation. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LRECā10), vol. 11, pp. 2169ā2174 (2010)
Aziz, W., Castilho, S., Specia, L.: Pet: a tool for post-editing and assessing machine translation. In: LREC, pp. 3982ā3987 (2012)
Aziz, W., Castilho, S., Specia, L.: Pet: a tool for post-editing and assessing machine translation. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LRECā12), pp. 3982ā3987 (2012)
Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. In: Proceedings of the 23rd Annual ACM symposium on User interface software and technology, pp. 313ā322. ACM (2010)
Dai, P., Lin, C.H., Weld, D.S.: Pomdp-based control of workflows for crowdsourcing. Artif. Intell. 202, 52ā85 (2013)
Green, S., Heer, J., Manning, C.D.: The efficacy of human post-editing for language translation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 439ā448. ACM (2013)
Ishida, T. (ed.): The Language Grid: Service-Oriented Collective Intelligence for Language Resource Interoperability. Springer Science & Business Media (2011)
Kamar, E., Hacker, S., Horvitz, E.: Combining human and machine intelligence in large-scale crowdsourcing. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent SystemsāVolume 1, AAMAS ā12, pp. 467ā474. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2012)
Kittur, A., Khamkar, S., AndrĆ©, P., Kraut, R.: Crowdweaver: Visually managing complex crowd work. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ā12, pp. 1033ā1036. ACM, New York, NY, USA (2012)
Kittur, A., Nickerson, J.V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., Horton, J.: The future of crowd work. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ā13, pp. 1301ā1318. ACM, New York, NY, USA (2013)
Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: crowdsourcing complex work. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 43ā52. ACM (2011)
Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ā12, pp. 1003ā1012. ACM, New York, NY, USA (2012)
Lin, D., Ishida, T.: Participatory service design based on user-centered QoS. In: 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), vol. 1, pp. 465ā472 (2013)
Lin, D., Murakami, Y., Ishida, T., Murakami, Y., Tanaka, M.: Composing human and machine translation services: language grid for improving localization processes. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LRECā10), vol. 10, pp. 500ā506 (2010)
Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Exploring iterative and parallel human computation processes. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ā10, pp. 68ā76. ACM, New York, NY, USA (2010)
Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology, UIST ā10, pp. 57ā66. ACM, New York, NY, USA (2010)
Luther, K., Hahn, N., Dow, S.P., Kittur, A.: Crowdlines: supporting synthesis of diverse information sources through crowdsourced outlines. In: Third AAAI Conference on Human Computation and Crowdsourcing, pp. 110ā119 (2015)
Morita, D., Ishida, T.: Designing protocols for collaborative translation. In: Proceedings of the 12th International Conference on Principles of Practice in Multi-Agent Systems (PRIMAā09), pp. 17ā32. Springer (2009)
Murakami, Y., Lin, D., Ishida, T.: Service-Oriented Architecture for Interoperability of Multilanguage Services, pp. 313ā328. Springer, Berlin (2014)
Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 614ā622. ACM (2008)
Yoon, K.P., Hwang, C.L.: Multiple Attribute Decision Making: An Introduction, vol. 104. Sage Publications (1995)
Zaidan, O., Callison-Burch, C.: Crowdsourcing translation: professional quality from non-professionals. In: ACL, pp. 1220ā1229 (2011)
Acknowledgements
This work was partially supported by a Grant-in-Aid for Scientific Research (S) (24220002, 2012-2016) from Japan Society for the Promotion of Science (JSPS), Scientific Research (A) (17H00759, 2017-2020) from Japan Society for the Promotion of Science (JSPS), and Service Science, Solutions and Foundation Integrated Research Program from JST RISTEX.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
Ā© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Goto, S., Ishida, T., Lin, D. (2018). Optimizing Crowdsourcing Workflow for Language Services. In: Murakami, Y., Lin, D., Ishida, T. (eds) Services Computing for Language Resources . Cognitive Technologies. Springer, Singapore. https://doi.org/10.1007/978-981-10-7793-7_5
Download citation
DOI: https://doi.org/10.1007/978-981-10-7793-7_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-7792-0
Online ISBN: 978-981-10-7793-7
eBook Packages: Computer ScienceComputer Science (R0)