Wei Guo
ABSTRACT
Amazon Mechanical Turk (Mturk), as the world's largest micro-task crowdsourcing platform, provides task search services to thousands of users every day. However, the lack of navigation panel in the platform's existing user interface makes task searching difficult and tedious, resulting in users rarely finding acceptable tasks among the countless crowdsourced assignments. To lower the task search threshold of Mturk, this study creates an automated task navigation structure by combining the unsupervised task clustering algorithm and the topic recognition algorithm, which can be fine-tuned according to task characteristics and help users focus on the task type quickly and precisely. Since the navigation panel is a non-existent element of Mturk, this paper develops a well-designed questionnaire to investigate users' perceptions of the improved interface. The results show that users were dissatisfied with Mturk's current UI, preferring the navigation interface outlined in this study. In conclusion, this work provides theoretical guidance for building similar automated task navigation panels for other crowdsourcing platforms while addressing the practical problems of Mturk.
- Ying Zhen, Abdullah Khan, Shah Nazir, Zhao Huiqi, Abdullah Alharbi, and Sulaiman Khan,Aug 2021, "Crowdsourcing usage, task assignment methods, and crowdsourcing platforms: A systematic literature review," Journal of Software-Evolution and Process, vol. 33, no. 8, Art. no. e2368.Google Scholar
Digital Library
- Herman Aguinis, Isabel Villamor, and Ravi S. Ramani,Apr 2021, "MTurk Research: Review and Recommendations," Journal of Management, vol. 47, no. 4, pp. 823-837, Art. no. 0149206320969787.Google ScholarCross Ref
- Simon A Campo, V. J. Khan, K. Papangelis, and P. Markopoulos,2018, "Community heuristics for user interface evaluation of crowdsourcing platforms," Future Generation Computer Systems, vol. 95, no. JUN., pp. 775-789.Google Scholar
- Mohammad Allahbakhsh, Boualem Benatallah, Cinzia Cappiello, Florian Daniel, and Pavel Kucherbaev,2018, "Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions," ACM Computing Surveys, vol. 51, no. 1, pp. 1-40.Google ScholarDigital Library
- Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini,May 1 2021, "The Impact of Task Abandonment in Crowdsourcing," Ieee Transactions on Knowledge and Data Engineering, vol. 33, no. 5, pp. 2266-2279.Google Scholar
- Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase, Training Workers for Improving Performance in Crowdsourcing Microtasks. Springer International Publishing, 2015.Google ScholarDigital Library
- Lydia B. Chilton, John J. Horton, Robert C. Miller, and Shiri Azenkot,2019, "Task search in a human computation market," ACM.Google Scholar
- Ipeirotis and G. Panagiotis,2010, "Analyzing the Amazon Mechanical Turk marketplace," Xrds Crossroads the Acm Magazine for Students, vol. 17, no. 2, p. 16.Google ScholarDigital Library
- A. Hotaling and J. P. Bagrow,2020, "Efficient crowdsourcing of crowd-generated microtasks," PLOS ONE, vol. 15.Google Scholar
- Shan, Liu, Fan, Xia, Jinlong, Zhang, Wei, Pan, Yajun, and Zhang,2016, "Exploring the trends, characteristic antecedents, and performance consequences of crowdsourcing project risks," International Journal of Project Management.Google Scholar
- N. Xu, X. Shuai, F. Li, J. Zhang, and S. Li, "Task Classification: Towards Tasks Correlation for Combinatorial Auction Mechanism in Crowdsourcing," in Second International Conference on Mechanical, 2018.Google Scholar
- Anupam Bhattacharjee and Manish Agrawal,Jan 2021, "Process Design to Use Amazon MTurk for Cognitively Complex Tasks," It Professional, vol. 23, no. 1, pp. 56-61.Google ScholarCross Ref
- Thomas C Mcandrew, Elizaveta A Guseva, and James P Bagrow,2016, "Reply & Supply: Efficient crowdsourcing when workers do more than answer questions," PLoSO, vol. 12.Google Scholar
- L. B. Chilton, J. J. Horton, R. C. Miller, and S. Azenkot, "Task Search in a Human Computation Market," in HCOMP 2010;Human computation workshop, 2011.Google Scholar
- M. Stankovic, J. Jovanovic, and P. Laublet,2011, "Linked Data Metrics for Flexible Expert Search on the Open Web," Springer Berlin Heidelberg.Google Scholar
- Ayswarya R. Kurup, G. P. Sajeev, and J. Swaminathan,2021 2021, "Aggregating Reliable Submissions in Crowdsourcing Systems," Ieee Access, vol. 9, pp. 153058-153071.Google ScholarCross Ref
- Xiang Fang and Clyde W. Holsapple,Sep 2011, "Impacts of navigation structure, task complexity, and users' domain knowledge on Web site usability-an empirical study," Information Systems Frontiers, vol. 13, no. 4, pp. 453-469.Google ScholarDigital Library
- C. Min,2019, "Reducing Web Page Complexity to Facilitate Effective User Navigation," IEEE Transactions on Knowledge and Data Engineering, vol. PP, no. 99, pp. 1-1.Google ScholarDigital Library
- Min Chen and Younguk Ryu,2013, "Facilitating Effective User Navigation through Website Structure Improvement," IEEE Transactions on Knowledge and Data Engineering.Google ScholarDigital Library
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G. Ipeirotis, Philippe Cudre-Mauroux, and Acm, "The Dynamics of Micro-Task Crowdsourcing The Case of Amazon MTurk," in 24th International Conference on World Wide Web (WWW), Florence, ITALY, 2015, pp. 238-247, 2015.Google Scholar
- Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon, "Improving Worker Engagement Through Conversational Microtask Crowdsourcing," in CHI '20: CHI Conference on Human Factors in Computing Systems, 2020.Google Scholar
- Lian Jian, Sha Yang, Sulin Ba, Li Lu, and Li Crystal Jiang,Mar 2019, "MANAGING THE CROWDS: THE EFFECT OF PRIZE GUARANTEES AND IN-PROCESS FEEDBACK ON PARTICIPATION IN CROWDSOURCING CONTESTS," Mis Quarterly, vol. 43, no. 1, pp. 97-+.Google ScholarDigital Library
- Emmanuel W. Ayaburi,2018, "Understanding Characteristics of High Performers in Two-Sided Competitive Crowdsourcing," International Journal of Innovation and Technology Management, vol. 15, pp. S0219877018500414-.Google ScholarCross Ref
- C. Gizem Korpeoglu, Ersin Kkrpeoolu, and Ssddka Tunn,2017, "Optimal Duration of Innovation Contests," Social ence Electronic Publishing.Google Scholar
- Yongqiang Sun, Nan Wang, Chunxiao Yin, and Jacky Xi Zhang,2015, "Understanding the relationships between motivators and effort in crowdsourcing marketplaces: A nonlinear analysis," International Journal of Information Management, vol. 35, no. 3, pp. 267-276.Google ScholarDigital Library
- Kaoru Ota, Mianxiong Dong, Jinsong Gui, and Anfeng Liu,Mar-Apr 2018, "QUOIN: Incentive Mechanisms for Crowd Sensing Networks," Ieee Network, vol. 32, no. 2, pp. 114-119.Google ScholarCross Ref
- Lian Jian, Sha Yang, Sulin Ba, Li Lu, and Li Crystal Jiang,2019, "Managing the Crowds: The Effect of Prize Guarantees and In-Process Feedback on Participation in Crowdsourcing Contests," MIS quarterly, vol. 43, no. 1, pp. 97-112.Google ScholarDigital Library
- Richard D M Stevenson, Andrew G Siddall, Philip J F Turner, Keith A Stokes, and James L J Bilzon, "A Task Analysis for the Development of Minimum Physical Employment Standards for Physically Demanding Occupations," in International Conference on Physical Employment Standards, 2015.Google Scholar
- Michael W. Smith, Melissa A. Bentley, Antonio R. Fernandez, Gregory Gibson, Sharon B. Schweikhart, and David D. Woods,2013, "Performance of Experienced Versus Less Experienced Paramedics in Managing Challenging Scenarios: A Cognitive Task Analysis Study," Annals of Emergency Medicine, vol. 62, no. 4, pp. 367-379.Google ScholarCross Ref
- Z. Li and J. Huang,2018, "A Text Classification Algorithm Based on Improved Multidimensional–Multiresolution Topological Pattern Recognition," International Journal of Pattern Recognition and Artificial Intelligence.Google Scholar
- Hyunjoong Kim, Han Kyul Kim, and Sungzoon Cho,Jul 15 2020, "Improving spherical k-means for document clustering: Fast initialization, sparse centroid projection, and efficient cluster labeling," Expert Systems with Applications, vol. 150, Art. no. 113288.Google Scholar
- Sammoudarachid and El-Zaartali,2021, "An Optimized Approach for Prostate Image Segmentation Using K-Means Clustering Algorithm with Elbow Method," Computational Intelligence and Neuroscience.Google Scholar
- D. M. Blei, Andrew Y. Ng, and Michael I. Jordan,2001, "Latent Dirichlet Allocation," The Annals of Applied Statistics.Google Scholar
- Siwei Qiang, Yongkun Wang, and Yaohui Jin,2017, "A Local-Global LDA Model for Discovering Geographical Topics from Social Media," Springer, Cham.Google Scholar
- Siwei Qiang, Yongkun Wang, and Yaohui Jin, "A Local-Global LDA Model for Discovering Geographical Topics from Social Media," in Asia-pacific Web, 2017.Google Scholar
- S. X. Liu and L. P. Yang,2009, "Method, Device and System for Processing, Browsing and Searching an Electronic Documents."Google Scholar
- F Biolcati Rinaldi,2011, "Likert Scales," Encyclopedia of Consumer Culture.Google Scholar
- Liang, Lei, Yang, Tian, Zhang, Han, Zhen-Li, Xing, and Hao,2017, "Reliability and Validity Analyses Are Essential for Questionnaire Research," Hepatology: Official Journal of the American Association for the Study of Liver Diseases, vol. 66, no. 3, pp. 1009-1009.Google Scholar
- Lyndsey D. Ruiz, Anna M. Jones, and Rachel E. Scherr,2020, "Validity and Reliability of a Nutrition Knowledge Questionnaire for High School–Aged Adolescents," Journal of Nutrition Education and Behavior.Google Scholar
- M A Syakur, B K Khotimah, E M S Rochman, and B D Satoto,2018, "Integration K-Means Clustering Method and Elbow Method For Identification of The Best Customer Profile Cluster," IOP Conference Series: Materials Science and Engineering, vol. 336, pp. 012017-.Google Scholar
- J. Choi, T. Kim, and S. G. Lee,2019, "A Cross-Sentence Latent Variable Model for Semi-Supervised Text Sequence Matching."Google Scholar
- Mazzone and Marco,2014, "The Continuum Problem: Modified Occam's Razor and Conventionalisation of Meaning," International Review of Pragmatics, vol. 6, no. 1, pp. 29-58.Google ScholarCross Ref
Index Terms
- Task navigation panel for Amazon Mechanical Turk
Index terms have been assigned to the content through auto-classification.
Recommendations
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsA growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 ...
Mechanical turk as an ontology engineer?: using microtasks as a component of an ontology-engineering workflow
WebSci '13: Proceedings of the 5th Annual ACM Web Science ConferenceOntology evaluation has proven to be one of the more difficult problems in ontology engineering. Researchers proposed numerous methods to evaluate logical correctness of an ontology, its structure, or coverage of a domain represented by a corpus. ...
Investigating the Amazon Mechanical Turk Market Through Tool Design
We developed TurkBench to better understand the work of crowdworkers on the Amazon Mechanical Turk (AMT) marketplace. While we aimed to reduce the amount of invisible, unpaid work that these crowdworkers performed, we also probed the day-to-day ...
Comments