Abstract
Workflow quality is a key determinant of crowdsourcing complex work, but finding ways to task design and plan has proved illusive. Instead, we formulate it as an optimization problem with budget constraints and fewer decision variables to set. We propose a two-staged approach CrowDIY that can not only estimate task attributes based on previous tasks but also optimize them with budget constraints in order to publish tasks more wisely in a timely manner. Several experimental studies have been conducted, and the results show compelling evidence that, under different conditions, the proposed approach can effectively reduce the workload of workflow design and plan, while avoiding commonly encountered trial-and-error in crowdsourcing workflows and leading up to successful complex outcomes.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bernaschina, C., Catallo I., Fraternali P., Martinenghi, D., Tagliasacchi, M.: Champagne: a web tool for the execution of crowdsourcing campaigns. In: International Conference on World Wide Web (Companion), pp. 171–174. ACM, New York (2015)
Bigham, J.P., Bernstein, M.S., Adar, E.: Human-computer interaction and collective intelligence. In: Handbook of Collective Intelligence, pp. 57–84. MIT Press (2015)
Chen, R., Chen, S.-F., Zhang, X.-Y.: A two-staged task assignment algorithm for worker recommendation in a crowdsourcing environment. In: International Conference on Industrial Engineering and Engineering Management, Singapore, pp. 2034–2038 (2017)
Doan, A., Ramakrishnan, R., Halevy, A.Y.: Crowdsourcing systems on the world-wide web. Commun. ACM 54, 86–96 (2011)
Huang, Y.-T.: Design and implementation of a workflow system for crowdsourcing. Master thesis, Dalian Maritime University (2017). (in Chinese)
Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: CrowdForge: crowdsourcing complex work. In: Annual ACM Symposium on User Interface Software and Technology, pp. 43–52. ACM, New York (2011)
Kulkarni, A., Can, M., Hartmann, B.: Collaboratively crowdsourcing workflows with turkomatic. In: ACM Conference on Computer Supported Cooperative Work, pp. 1003–1012. ACM, New York (2012)
Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: TurKit: human computation algorithms on mechanical turk. In: Annual ACM Symposium on User Interface Software and Technology, pp. 57–66. ACM, New York (2010)
Retelny, D., Bernstein, M.S., Valentine, M.A.: No workflow can ever be enough: how crowdsourcing workflows constrain complex work. In: ACM Human-Computer Interaction, CSCW, vol. 1, Article 89, 23 p. ACM (2017)
JGraphT. https://jgrapht.org. Accessed 10 Jan 2019
Gadiraju, U., Kawase, R.: Improving reliability of crowdsourced results by detecting crowd workers with multiple identities. In: Cabot, J., De Virgilio, R., Torlone, R. (eds.) ICWE 2017. LNCS, vol. 10360, pp. 190–205. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60131-1_11
Catallo, I., Martinenghi, D.: The dimensions of crowdsourcing task design. In: Cabot, J., De Virgilio, R., Torlone, R. (eds.) ICWE 2017. LNCS, vol. 10360, pp. 394–402. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60131-1_25
Acknowledgments
This work is supported by the National Natural Science Foundation of China (No. 61672122, No. 61602077), the Natural Science Foundation of Liaoning Province of China (No. 2015020023), the Educational Commission of Liaoning Province of China (No. L2015060) and the Fundamental Research Funds for the Central Universities (NO. 3132016348).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, R., Li, B., Xing, H., Wang, Y. (2019). CrowDIY: How to Design and Adapt Collaborative Crowdsourcing Workflows Under Budget Constraints. In: Bakaev, M., Frasincar, F., Ko, IY. (eds) Web Engineering. ICWE 2019. Lecture Notes in Computer Science(), vol 11496. Springer, Cham. https://doi.org/10.1007/978-3-030-19274-7_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-19274-7_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-19273-0
Online ISBN: 978-3-030-19274-7
eBook Packages: Computer ScienceComputer Science (R0)