Abstract
Recently, attempts have been to request translation services from anonymous crowds. Compared with platform-based language services, services performed by humans have the advantages of flexibility and quality. However, due to the nature of human services the results are not consistent and quality is not assured. This research tries to solve this problem by creating workflows that make collaboration among crowdsourcing workers far more effective and efficient. We model workers and tasks, and calculate the optimal workflow. To confirm the feasibility of this model, we conduct a computational experiment to calculate the best workflow under various parameters. The results are consistent with existing research so the model is useful in understanding crowdsourcing workflows. In addition, a system is developed for realizing crowdsourcing workflows for translation services. Finally, we develop a translation interface that demonstrates the feasibility of the proposed method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Ambati, V., Vogel, S., Carbonell, J.G.: Active learning and crowd-sourcing for machine translation. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10), vol. 11, pp. 2169–2174 (2010)
Aziz, W., Castilho, S., Specia, L.: Pet: a tool for post-editing and assessing machine translation. In: LREC, pp. 3982–3987 (2012)
Aziz, W., Castilho, S., Specia, L.: Pet: a tool for post-editing and assessing machine translation. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pp. 3982–3987 (2012)
Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. In: Proceedings of the 23rd Annual ACM symposium on User interface software and technology, pp. 313–322. ACM (2010)
Dai, P., Lin, C.H., Weld, D.S.: Pomdp-based control of workflows for crowdsourcing. Artif. Intell. 202, 52–85 (2013)
Green, S., Heer, J., Manning, C.D.: The efficacy of human post-editing for language translation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 439–448. ACM (2013)
Ishida, T. (ed.): The Language Grid: Service-Oriented Collective Intelligence for Language Resource Interoperability. Springer Science & Business Media (2011)
Kamar, E., Hacker, S., Horvitz, E.: Combining human and machine intelligence in large-scale crowdsourcing. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems—Volume 1, AAMAS ’12, pp. 467–474. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2012)
Kittur, A., Khamkar, S., André, P., Kraut, R.: Crowdweaver: Visually managing complex crowd work. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12, pp. 1033–1036. ACM, New York, NY, USA (2012)
Kittur, A., Nickerson, J.V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., Horton, J.: The future of crowd work. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13, pp. 1301–1318. ACM, New York, NY, USA (2013)
Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: crowdsourcing complex work. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 43–52. ACM (2011)
Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12, pp. 1003–1012. ACM, New York, NY, USA (2012)
Lin, D., Ishida, T.: Participatory service design based on user-centered QoS. In: 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), vol. 1, pp. 465–472 (2013)
Lin, D., Murakami, Y., Ishida, T., Murakami, Y., Tanaka, M.: Composing human and machine translation services: language grid for improving localization processes. In: Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC’10), vol. 10, pp. 500–506 (2010)
Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Exploring iterative and parallel human computation processes. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10, pp. 68–76. ACM, New York, NY, USA (2010)
Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: Turkit: human computation algorithms on mechanical turk. In: Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology, UIST ’10, pp. 57–66. ACM, New York, NY, USA (2010)
Luther, K., Hahn, N., Dow, S.P., Kittur, A.: Crowdlines: supporting synthesis of diverse information sources through crowdsourced outlines. In: Third AAAI Conference on Human Computation and Crowdsourcing, pp. 110–119 (2015)
Morita, D., Ishida, T.: Designing protocols for collaborative translation. In: Proceedings of the 12th International Conference on Principles of Practice in Multi-Agent Systems (PRIMA’09), pp. 17–32. Springer (2009)
Murakami, Y., Lin, D., Ishida, T.: Service-Oriented Architecture for Interoperability of Multilanguage Services, pp. 313–328. Springer, Berlin (2014)
Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 614–622. ACM (2008)
Yoon, K.P., Hwang, C.L.: Multiple Attribute Decision Making: An Introduction, vol. 104. Sage Publications (1995)
Zaidan, O., Callison-Burch, C.: Crowdsourcing translation: professional quality from non-professionals. In: ACL, pp. 1220–1229 (2011)
Acknowledgements
This work was partially supported by a Grant-in-Aid for Scientific Research (S) (24220002, 2012-2016) from Japan Society for the Promotion of Science (JSPS), Scientific Research (A) (17H00759, 2017-2020) from Japan Society for the Promotion of Science (JSPS), and Service Science, Solutions and Foundation Integrated Research Program from JST RISTEX.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Goto, S., Ishida, T., Lin, D. (2018). Optimizing Crowdsourcing Workflow for Language Services. In: Murakami, Y., Lin, D., Ishida, T. (eds) Services Computing for Language Resources . Cognitive Technologies. Springer, Singapore. https://doi.org/10.1007/978-981-10-7793-7_5
Download citation
DOI: https://doi.org/10.1007/978-981-10-7793-7_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-7792-0
Online ISBN: 978-981-10-7793-7
eBook Packages: Computer ScienceComputer Science (R0)