Turki Alelyani
ABSTRACT
In software crowdsourcing, task price is one of the most important incentive to attract broad worker participation and contribution. Underestimating or overestimating a task's price may lead to task starvation or resource inefficiency. Nevertheless, few studies have addressed pricing support in software crowdsourcing. In this study, we propose Context-Centric Pricing approach to support software crowdsourcing pricing based on limited information available at early planning phase, i.e. textual task requirements. In the proposed approach, the global models include a list of 6 pricing factors and employ different natural language processing techniques for prediction; in addition, local models can be derived w.r.t. more relevant context, i.e. a set of similar tasks identified based on Topic modeling and we evaluate 7 predictive models. The proposed approach is evaluated on 450 software tasks extracted from TopCoder, the largest software crowdsourcing platform. The results show that: 1) the proposed models can be used at early crowdsourcing planning phase, when information on traditional metrics are not available; 2) the best model achieves 65% in accuracy measure; 3) the local model exhibits 27% increases superior to the global model. The proposed work can stimulate future research into crowdsourcing pricing estimation and inform ideas for crowdsourcing decision-makers.
- Singer, Yaron, and Manas Mittal. "Pricing Tasks in Online Labor Markets." Human computation. 2011.Google Scholar
- Singer, Yaron. "Budget feasible mechanisms." Foundations of Computer Science (FOCS), 2010 51st Annual IEEE Symposium on. IEEE, 2010. Google ScholarDigital Library
- Ke Mao, Ye Yang, Mingshu Li, Mark Harman. "Pricing Crowdsourcing-based Software Development Tasks". In Proceedings of the 35th International Conference on Software Engineering (ICSE'13). NIER Track, pp. 1221--1224, 2013.Google Scholar
- Howe, Jeff. "The rise of crowdsourcing." Wired magazine 14.6 (2006):1--4Google Scholar
- Gonen, Rica, et al. "Increased efficiency through pricing in online labor markets." Journal of Electronic Commerce Research 15.1 (2014): 58.Google Scholar
- Difallah, Djellel Eddine, et al. "Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement." Second AAAI Conference on Human Computation and Crowdsourcing. 2014.Google Scholar
- Wang, Jing, Panagiotis G. Ipeirotis, and Foster Provost. "Quality-based pricing for crowdsourced workers." (2013).Google Scholar
- Harwell, Richard, et al. "What is a Requirement?" INCOSE International Symposium. Vol. 3. No. 1. 1993.Google Scholar
- Humayun, Mamoona, and Cui Gang. "Estimating effort in global software development projects using machine learning techniques." International Journal of Information and Education Technology 2.3 (2012): 208.Google ScholarCross Ref
- El Bajta, Manal. "Analogy-based software development effort estimation in global software development." 2015 IEEE 10th International Conference on Global Software Engineering Workshops. IEEE, 2015 Google ScholarDigital Library
- Azzeh, Mohammad, and Ali Bou Nassif. "Fuzzy Model Tree For Early Effort Estimation." Machine Learning and Applications (ICMLA), 2013 12th International Conference on. Vol. 2. IEEE, 2013. Google ScholarDigital Library
- B. Baskeles, B. Turhan, and A. Bener, "Software effort estimation using machine learning methods," In Proc. of 22nd international symposium on computer and information sciences, 2007. Google ScholarCross Ref
- A. Heiat, "Comparison of artificial neural network and regression models for estimating software development effort," Information and Software Technology,pp 911--922, 2004.Google Scholar
- E. Jun. and J. Lee, "Quasi-optimal case-selective neural network model for software effort estimation," Expert Systems with Applications, pp. 1--14,2001. Google ScholarCross Ref
- A. Oliveira, "Estimation of software project effort with support vector regression," Neurocomputing, pp. 1749--1753, 2006. Google ScholarDigital Library
- H. Park. and S. Baek, "An empirical validation of a neural network model for software effort estimation," Expert Systems with Applications, vol. 35, no. 3, pp. 929--937, 2008. Google ScholarDigital Library
- Srivastava, Ashok N., and Mehran Sahami, eds. Text mining: Classification, clustering, and applications. CRC Press, 2009. Google ScholarCross Ref
- McCallum, Andrew Kachites. "MALLET: A Machine Learning for Language Toolkit." http://mallet.cs.umass.edu. 2002.Google Scholar
- Menzies, Tim, and Andrian Marcus. "Automated severity assessment of software defect reports." Software Maintenance, 2008. ICSM 2008. IEEE International Conference on. IEEE, 2008. Google ScholarCross Ref
- Quinlan, J. Ross. C4. 5: programs for machine learning. Elsevier, 2014.Google ScholarDigital Library
- Pedregosa, Fabian, et al. "Scikit-learn: Machine learning in Python." Journal of Machine Learning Research 12.Oct (2011): 2825--2830.Google ScholarDigital Library
- Forman, George. "An extensive empirical study of feature selection metrics for text classification." Journal of machine learning research 3.Mar (2003): 1289--1305.Google Scholar
- Yang, Yiming, and Xin Liu. "A re-examination of text categorization methods" Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 1999. Google ScholarDigital Library
- Joachims, Thorsten. "Text categorization with support vector machines:Learning with many relevant features." European conference on machine learning. Springer Berlin Heidelberg, 1998.Google ScholarDigital Library
- Alelyani, Turki, and Ye Yang. "Software crowdsourcing reliability: an empirical study on developers behavior." Proceedings of the 2nd International Workshop on Software Analytics. ACM, 2016. Google ScholarDigital Library
- Menzies et al. "Local vs. global models for effort estimation and defect prediction." Automated Software Engineering (ASE), 2011 26th IEEE/ACM International Conference on. IEEE, 2011. Google ScholarDigital Library
Index Terms
- Context-Centric Pricing: Early Pricing Models for Software Crowdsourcing Tasks
Recommendations
Envy-Free Pricing in Large Markets: Approximating Revenue and Welfare
We study the classic setting of envy-free pricing, in which a single seller chooses prices for its many items, with the goal of maximizing revenue once the items are allocated. Despite the large body of work addressing such settings, most versions of ...
Pricing commodities
How should a seller price her goods in a market where each buyer prefers a single good among his desired goods, and will buy the cheapest such good, as long as it is within his budget? We provide efficient algorithms that compute near-optimal prices for ...
Contingent Pricing to Reduce Price Risks
The price for a product may be set too low, causing the seller to leave money on the table, or too high, driving away potential buyers. Contingent pricing can be useful in mitigating these problems. In contingent pricing arrangements, price is ...
Comments