research-article

Context-aware in-process crowdworker recommendation

Authors:

Qing WangAuthors Info & Claims

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Pages 1535 - 1546

https://doi.org/10.1145/3377811.3380380

Published: 01 October 2020 Publication History

Abstract

Identifying and optimizing open participation is essential to the success of open software development. Existing studies highlighted the importance of worker recommendation for crowdtesting tasks in order to detect more bugs with fewer workers. However, these studies mainly focus on one-time recommendations with respect to the initial context at the beginning of a new task. This paper argues the need for in-process crowdtesting worker recommendation. We motivate this study through a pilot study, revealing the prevalence of long-sized non-yielding windows, i.e., no new bugs are revealed in consecutive test reports during the process of a crowdtesting task. This indicates the potential opportunity for accelerating crowdtesting by recommending appropriate workers in a dynamic manner, so that the non-yielding windows could be shortened.

To that end, this paper proposes a context-aware in-process crowdworker recommendation approach, iRec, to detect more bugs earlier and potentially shorten the non-yielding windows. It consists of three main components: 1) the modeling of dynamic testing context, 2) the learning-based ranking component, and 3) the diversity-based re-ranking component. The evaluation is conducted on 636 crowdtesting tasks from one of the largest crowdtesting platforms, and results show the potential of iRec in improving the cost-effectiveness of crowdtesting by saving the cost and shortening the testing process.

References

[1]

2019. https://www.topcoder.com/.

[2]

2019. https://www.applause.com/.

[3]

2019. https://www.testbird.com/.

[4]

Emad Aghajani. 2018. Context-Aware Software Documentation. In 2018 IEEE International Conference on Software Maintenance and Evolution, ICSME 2018, Madrid, Spain, September 23-29, 2018. 727--731.

[5]

Eiman Aldahari, Vivek Shandilya, and Sajjan G. Shiva. 2018. Crowdsourcing Multi-Objective Recommendation System. In Companion of the The Web Conference 2018 on The Web Conference 2018, WWW 2018, Lyon, France, April 23-27, 2018. 1371--1379.

[6]

John Anvik, Lyndon Hiew, and Gail C Murphy. 2006. Who should fix this bug?. In ICSE'06. 361--370.

Digital Library

[7]

Pamela Bhattacharya and Iulian Neamtiu. 2010. Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging. In ICSM'10. 1--10.

Digital Library

[8]

Gerardo Canfora, Massimiliano Di Penta, Rocco Oliveto, and Sebastiano Panichella. 2012. Who is going to mentor newcomers in open source projects?. In FSE'12. 44.

Digital Library

[9]

Zherui Cao, Yuan Tian, Tien-Duy B. Le, and David Lo. 2018. Rule-based specification mining leveraging learning to rank. Autom. Softw. Eng. 25, 3 (2018), 501--530.

Digital Library

[10]

Di Chen, Wei Fu, Rahul Krishna, and Tim Menzies. 2018. Applications of psychological science for actionable analytics. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT. 456--467.

Digital Library

[11]

Ning Chen and Sunghun Kim. 2012. Puzzle-based automatic testing: Bringing humans into the loop by solving puzzles. In ASE'12. 140--149.

Digital Library

[12]

N. Cliff. 2014. Ordinal methods for behavioral data analysis. Psychology Press.

[13]

Qiang Cui, Junjie Wang, Guowei Yang, Miao Xie, Qing Wang, and Mingshu Li. 2017. Who Should Be Selected to Perform a Task in Crowdsourced Testing?. In COMPSAC'17. 75--84.

[14]

Qiang Cui, Song Wang, Junjie Wang, Yuanzhe Hu, Qing Wang, and Mingshu Li. 2017. Multi-Objective Crowd Worker Selection in Crowdsourced Testing. In SEKE'17. 218--223.

[15]

Yuanrui Fan, Xin Xia, David Lo, and Shanping Li. 2018. Early prediction of merged code changes to prioritize reviewing tasks. Empirical Software Engineering 23, 6 (2018), 3346--3393.

Digital Library

[16]

Yang Feng, Zhenyu Chen, James A Jones, Chunrong Fang, and Baowen Xu. 2015. Test report prioritization to assist crowdsourced testing. In FSE'15. 225--236.

Digital Library

[17]

Yang Feng, James A Jones, Zhenyu Chen, and Chunrong Fang. 2016. Multi-objective test report prioritization using image understanding. In ASE'16. 202--213.

Digital Library

[18]

Ruizhi Gao, Yabin Wang, Yang Feng, Zhenyu Chen, and W Eric Wong. 2018. Successes, challenges, and rethinking-an industrial investigation on crowdsourced mobile application testing. Empirical Software Engineering (2018), 1--25.

[19]

Ruizhi Gao, Yabin Wang, Yang Feng, Zhenyu Chen, and W. Eric Wong. 2019. Successes, challenges, and rethinking - an industrial investigation on crowdsourced mobile application testing. Empirical Software Engineering 24, 2 (2019), 537--561.

Digital Library

[20]

Marko Gasparic, Gail C. Murphy, and Francesco Ricci. 2017. A context model for IDE-based recommendation systems. Journal of Systems and Software 128 (2017), 200--219.

Digital Library

[21]

María Gómez, Romain Rouvoy, Bram Adams, and Lionel Seinturier. 2016. Reproducing context-sensitive crashes of mobile apps using crowdsourced monitoring. In 2016 IEEE/ACM International Conference on Mobile Software Engineering and Systems. IEEE, 88--99.

Digital Library

[22]

Victor HM Gomide, Pedro A Valle, José O Ferreira, José RG Barbosa, Adson F Da Rocha, and TMGdA Barbosa. 2014. Affective crowdsourcing applied to usability testing. International Journal of Computer Scienceand Information Technologies 5, 1 (2014), 575--579.

[23]

Christoph Hannebauer, Michael Patalas, Sebastian Stünkel, and Volker Gruhn. 2016. Automatically Recommending Code Reviewers Based on Their Expertise: An Empirical Comparison. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). 99--110.

Digital Library

[24]

Rui Hao, Yang Feng, James Jones, Yuying Li, and Zhenyu Chen. 2019. CTRAS: Crowdsourced test report aggregation and summarization. In ICSE'2019. 921--932.

Digital Library

[25]

Kihong Heo, Hakjoo Oh, and Hongseok Yang. 2019. Resource-aware program analysis via online abstraction coarsening. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019. 94--104.

Digital Library

[26]

Qiao Huang, Emad Shihab, Xin Xia, David Lo, and Shanping Li. 2018. Identifying self-admitted technical debt in open source projects using text mining. Empirical Software Engineering 23, 1 (2018), 418--451.

Digital Library

[27]

Gaeul Jeong, Sunghun Kim, and Thomas Zimmermann. 2009. Improving bug triage with bug tossing graphs. In FSE'09. 111--120.

Digital Library

[28]

He Jiang, Xin Chen, Tieke He, Zhenyu Chen, and Xiaochen Li. 2018. Fuzzy Clustering of Crowdsourced Test Reports for Apps. ACM Transactions on Internet Technology 18, 2 (2018), 18.

Digital Library

[29]

Muhammad Rezaul Karim, Ye Yang, David Messinger, and Guenther Ruhe. 2018. Learn or Earn? - Intelligent Task Recommendation for Competitive Crowdsourced Software Development. In 51st Hawaii International Conference on System Sciences, HICSS 2018, Hilton Waikoloa Village, Hawaii, USA, January 3-6, 2018. 1--10.

[30]

Di Liu, Xiaofang Zhang, Yang Feng, and James A. Jones. 2018. Generating descriptions for screenshots to assist crowdsourced testing. In 25th International Conference on Software Analysis, Evolution and Reengineering, SANER. 492--496.

[31]

Zheng Liu and Lei Chen. 2017. Worker Recommendation for Crowdsourced Q&A Services: A Triple-Factor Aware Approach. PVLDB 11, 3 (2017), 380--392.

Digital Library

[32]

David Ma, David Schuler, Thomas Zimmermann, and Jonathan Sillito. 2009. Expert recommendation with usage expertise. In ICSM'09. 535--538.

[33]

Ke Mao, Ye Yang, Qing Wang, Yue Jia, and Mark Harman. 2015. Developer recommendation for crowdsourced software development tasks. In SOSE'15. 347--356.

Digital Library

[34]

Dominique Matter, Adrian Kuhn, and Oscar Nierstrasz. 2009. Assigning bug reports using a vocabulary-based expertise model of developers. In MSR'09. 131--140.

Digital Library

[35]

Gail C. Murphy. 2018. The Need for Context in Software Engineering (IEEE CS Harlan Mills Award Keynote). In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE 2018). 5--5.

Digital Library

[36]

Gail C. Murphy. 2019. Beyond integrated development environments: adding context to software development. In Proceedings of the 41st International Conference on Software Engineering: New Ideas and Emerging Results, ICSE (NIER) 2019, Montreal, QC, Canada, May 29-31, 2019. 73--76.

Digital Library

[37]

R. Musson, J. Richards, D. Fisher, C. Bird, B. Bussone, and S. Ganguly. 2013. Leveraging the Crowd: How 48,000 Users Helped Improve Lync Performance. IEEE Software 30, 4 (2013), 38--45.

[38]

Nadim Nachar et al. 2008. The Mann-Whitney U: A test for assessing whether two independent samples come from the same distribution. Tutorials in quantitative Methods for Psychology 4, 1 (2008), 13--20.

[39]

Hoda Naguib, Nitesh Narayan, Bernd Brügge, and Dina Helal. 2013. Bug report assignee recommendation using activity profiles. In MSR'13. 22--30.

[40]

Haoran Niu, Iman Keivanloo, and Ying Zou. 2017. Learning to rank code examples for code search engines. Empirical Software Engineering 22, 1 (2017), 259--291.

Digital Library

[41]

Thomas Oberlin and Rangasami L. Kashyap. 1973. Bayes Decision Rules Based on Objective Priors. IEEE Trans. Systems, Man, and Cybernetics 3, 4 (1973), 359--364.

[42]

Mejdl S. Safran and Dunren Che. 2019. Efficient Learning-Based Recommendation Algorithms for Top-N Tasks and Top-N Workers in Large-Scale Crowdsourcing Systems. ACM Trans. Inf. Syst. 37, 1 (2019), 2:1--2:46.

Digital Library

[43]

Gerard Salton and Michael McGill. 1984. Introduction to Modern Information Retrieval. McGraw-Hill Book Company.

Digital Library

[44]

Ahmed Tamrawi, Tung Thanh Nguyen, Jafar M Al-Kofahi, and Tien N Nguyen. 2011. Fuzzy set and cache-based approach for bug triaging. In FSE'11. 365--375.

[45]

Ming Tan, Lin Tan, Sashank Dara, and Caleb Mayeux. 2015. Online Defect Prediction for Imbalanced Data. In ICSE'15. 99--108.

[46]

Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed Hassan, and Kenichi Matsumoto. 2016. An empirical comparison of model validation techniques for defect prediction models. TSE'16 43 (2016), 1--18.

[47]

Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. 2019. Superion: grammar-aware greybox fuzzing. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019. 724--735.

Digital Library

[48]

Junjie Wang, Qiang Cui, Qing Wang, and Song Wang. 2016. Towards effectively test report classification to assist crowdsourced testing. In ESEM'16. 6.

Digital Library

[49]

Junjie Wang, Qiang Cui, Song Wang, and Qing Wang. 2017. Domain adaptation for test report classification in crowdsourced testing. In ICSE-SEIP'17. 83--92.

[50]

Junjie Wang, Mingyang Li, Song Wang, Tim Menzies, and Qing Wang. 2019. Images don't lie: Duplicate crowdtesting reports detection with screenshot information. Information & Software Technology 110 (2019), 139--155.

Digital Library

[51]

Junjie Wang, Song Wang, Jianfeng Chen, Tim Menzies, Qiang Cui, Miao Xie, and Qing Wang. 2019. Characterizing Crowds to Better Optimize Worker Recommendation in Crowdsourced Testing. IEEE Transactions on Software Engineering (2019).

[52]

Junjie Wang, Song Wang, Qiang Cui, and Qing Wang. 2016. Local-based active classification of test report to assist crowdsourced testing. In ASE'16. 190--201.

Digital Library

[53]

Junjie Wang, Ye Yang, Rahul Krishna, Tim Menzies, and Qing Wang. 2019. iSENSE: Completion-Aware Crowdtesting Management. In ICSE'2019. 932--943.

Digital Library

[54]

Song Wang, Wen Zhang, and Qing Wang. 2014. FixerCache: Unsupervised caching active developers for diverse bug triage. In ESEM'14. 25.

Digital Library

[55]

Song Wang, Wen Zhang, Ye Yang, and Qing Wang. 2013. DevNet: exploring developer collaboration in heterogeneous networks of bug repositories. In ESEM'13. 193--202.

[56]

Lili Wei, Yepang Liu, and Shing-Chi Cheung. 2019. Pivot: learning API-device correlations to facilitate Android compatibility issue detection. In Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019. 878--888.

Digital Library

[57]

Eric W Weisstein. 2004. Bonferroni correction. (2004).

[58]

Qiang Wu, Christopher J. C. Burges, Krysta M. Svore, and Jianfeng Gao. 2010. Adapting boosting for information retrieval measures. Information Retrieval 13, 3 (01 Jun 2010), 254--270.

[59]

Xin Xia, David Lo, Ying Ding, Jafar M. Al-Kofahi, Tien N. Nguyen, and Xinyu Wang. 2017. Improving Automated Bug Triaging with Specialized Topic Model. IEEE Trans. Software Eng. 43, 3 (2017), 272--297.

Digital Library

[60]

Miao Xie, Qing Wang, Guowei Yang, and Mingshu Li. 2017. COCOON: Crowd-sourced Testing Quality Maximization Under Context Coverage Constraint. In ISSRE'17. 316--327.

[61]

Jifeng Xuan, He Jiang, Zhilei Ren, and Weiqin Zou. 2012. Developer prioritization in bug repositories. In ICSE'12. 25--35.

[62]

Hui Yang, Xiaobing Sun, Bin Li, and Yucong Duan. 2016. DR_PSF: Enhancing developer recommendation by leveraging personalized source-code files. In COMPSAC'16, Vol. 1. 239--244.

[63]

Ye Yang, Muhammad Rezaul Karim, Razieh Saremi, and Guenther Ruhe. 2016. Who Should Take This Task?: Dynamic Decision Support for Crowd Workers. In ESEM'16. 8.

Digital Library

[64]

M. B. Zanjani, H. Kagdi, and C. Bird. 2016. Automatically Recommending Peer Reviewers in Modern Code Review. IEEE Transactions on Software Engineering 42, 6 (2016), 530--543.

Digital Library

[65]

Wen Zhang, Song Wang, Ye Yang, and Qing Wang. 2013. Heterogeneous network analysis of developer contribution in bug repositories. In CSC'13. 98--105.

Digital Library

[66]

Xiaofang Zhang, Yang Feng,Di Liu, Zhenyu Chen, and Baowen Xu. 2018. Research Progress of Crowdsourced Software Testing. Journal of Software 29(1) (2018), 69--88.

[67]

Xiaofang Zhang, Yang Feng,Di Liu, Zhenyu Chen, and Baowen Xu. 2018. Research Progress of Crowdsourced Software Testing. Journal of Software 29(1) (2018), 69--88.

[68]

Guoliang Zhao, Daniel Alencar da Costa, and Ying Zou. 2019. Improving the pull requests review process using learning-to-rank algorithms. Empirical Software Engineering 24, 4 (2019), 2140--2170.

Digital Library

[69]

M. Zhou and A. Mockus. 2012. What make long term contributors: Willingness and opportunity in OSS community. In ICSE'12. 518--528.

Cited By

Fang CYu SZhang QLi XLiu YChen Z(2025)Enhanced Crowdsourced Test Report Prioritization via Image-and-Text Semantic Understanding and Feature IntegrationIEEE Transactions on Software Engineering10.1109/TSE.2024.351637251:1(283-304)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TSE.2024.3516372
He WDi PMing MZhang CSu TLi SSui Y(2024)Finding and Understanding Defects in Static Analyzers by Constructing Automated OraclesProceedings of the ACM on Software Engineering10.1145/36607811:FSE(1656-1678)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660781
Yu SFang CZhang QDu MLiu JChen Z(2024)Semi-supervised Crowdsourced Test Report Clustering via Screenshot-Text Binding RulesProceedings of the ACM on Software Engineering10.1145/36607761:FSE(1540-1563)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660776
Show More Cited By

Recommendations

Context- and Fairness-Aware In-Process Crowdworker Recommendation
Identifying and optimizing open participation is essential to the success of open software development. Existing studies highlighted the importance of worker recommendation for crowdtesting tasks in order to improve bug detection efficiency, i.e., detect ...
Context-Aware Personalized Crowdtesting Task Recommendation
Crowdsourced software testing (short for crowdtesting) is a special type of crowdsourcing. It requires that crowdworkers master appropriate skill-sets and commit significant effort for completing a task. Abundant uncertainty may arise during a ...
Query-driven context aware recommendation
RecSys '13: Proceedings of the 7th ACM conference on Recommender systems

Context aware recommender systems go beyond the traditional personalized recommendation models by incorporating a form of situational awareness. They provide recommendations that not only correspond to a user's preference profile, but that are also ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

June 2020

1640 pages

ISBN:9781450371216

DOI:10.1145/3377811

General Chairs:
Gregg Rothermel
North Carolina State University
,
Doo-Hwan Bae
KAIST, South Korea

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
National Key Research and Development Program of China

Conference

ICSE '20

Sponsor:

SIGSOFT

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, South Korea

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
380
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fang CYu SZhang QLi XLiu YChen Z(2025)Enhanced Crowdsourced Test Report Prioritization via Image-and-Text Semantic Understanding and Feature IntegrationIEEE Transactions on Software Engineering10.1109/TSE.2024.351637251:1(283-304)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TSE.2024.3516372
He WDi PMing MZhang CSu TLi SSui Y(2024)Finding and Understanding Defects in Static Analyzers by Constructing Automated OraclesProceedings of the ACM on Software Engineering10.1145/36607811:FSE(1656-1678)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660781
Yu SFang CZhang QDu MLiu JChen Z(2024)Semi-supervised Crowdsourced Test Report Clustering via Screenshot-Text Binding RulesProceedings of the ACM on Software Engineering10.1145/36607761:FSE(1540-1563)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660776
Zhang HPei YLiang STan S(2024)Understanding and Detecting Annotation-Induced Faults of Static AnalyzersProceedings of the ACM on Software Engineering10.1145/36437591:FSE(722-744)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643759
Jerónimo AMoreno PCamacho JVega G(2024)Techniques of SAST Tools in the Early Stages of Secure Software Development: A Systematic Literature Review2024 IEEE International Conference on Engineering Veracruz (ICEV)10.1109/ICEV63254.2024.10766004(1-8)Online publication date: 21-Oct-2024
https://doi.org/10.1109/ICEV63254.2024.10766004
Zhang HPei YChen JTan SChandra SBlincoe KTonella P(2023)Statfier: Automated Testing of Static Analyzers via Semantic-Preserving Program TransformationsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616272(237-249)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616272
Liu ZChen CWang JHuang YHu JWang Q(2023)Nighthawk: Fully Automated Localizing UI Display Issues via Visual UnderstandingIEEE Transactions on Software Engineering10.1109/TSE.2022.315087649:1(403-418)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3150876
Xiao WLi JHe HQiu RZhou M(2023)Personalized First Issue Recommender for Newcomers in Open Source Projects2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00158(800-812)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00158
Di Martino SFasolino AStarace LTramontana P(2023)GUI testing of Android applications: Investigating the impact of the number of testers on different exploratory testing strategiesJournal of Software: Evolution and Process10.1002/smr.2640Online publication date: 11-Dec-2023
https://doi.org/10.1002/smr.2640
Wang JYang YWang SHu JWang Q(2022)Context- and Fairness-Aware In-Process Crowdworker RecommendationACM Transactions on Software Engineering and Methodology10.1145/348757131:3(1-31)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3487571
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten