Article

Researching crowdsourcing software development: perspectives and concerns

Authors:

Klaas-Jan Stol,

Brian FitzgeraldAuthors Info & Claims

CSI-SE 2014: Proceedings of the 1st International Workshop on CrowdSourcing in Software Engineering

Pages 7 - 10

https://doi.org/10.1145/2593728.2593731

Published: 02 June 2014 Publication History

Abstract

Crowdsourcing is an emerging form of `outsourcing’ software development. While there has been considerable research in the area of crowdsourcing in general, very little research has focused specifically on how crowdsourcing works in a software development context, and as far as we know, there have been no published studies of crowdsourcing software development from a customer perspective. Based on a review of the literature, we identified a number of key concerns related to crowdsourcing that are of particular importance in a software development context. Furthermore, we observed a number of recurring key stakeholders, or actors, each of whom has a unique perspective on crowdsourcing. This paper presents a research framework that consists of the various combinations of stakeholders and key concerns. The framework can be used to guide future research on the use of crowdsourcing as a `sourcing’ strategy, as well as a means to review and synthesize research findings so as to be able to compare studies on crowdsourcing in a software development context.

References

[1]

P. Ågerfalk and B. Fitzgerald. Outsourcing to an unknown worforce: Exploring opensourcing as a global sourcing strategy. MIS Quarterly, 32(2), 2008.

Digital Library

[2]

A. Aurum, R. Jeffery, C. Wohlin, and M. Handzic. Managing Software Engineering Knowledge. Springer, 2003.

Digital Library

[3]

A. Begel, J. Bosch, and M. A. Storey. Social networking meets software development: Perspectives from github, msdn, stack exchange, and topcoder. IEEE Softw., 30(1), 2013.

Digital Library

[4]

A. Begel, J. D. Herbsleb, and M.-A. Storey. The future of collaborative software development. In CSCw, 2012.

Digital Library

[5]

E. Bonabeau. Decisions 2.0: The power of collective intelligence. MIT Sloan Manage Rev, 50(2):45–52, 2009.

[6]

A. Bozzon, M. Brambilla, S. Ceri, M. Silvestri, and G. Vesci. Choosing the right crowd: Expert finding in social networks. In Proc. EDBT/ICDT, 2013.

Digital Library

[7]

D. C. Brabham. Moving the crowd at istockphoto: The composition of the crowd and motivations for participation in crowdsourcing application. First Monday, 13(6), 2008.

[8]

D. C. Brabham. Crowdsourcing. MIT Press, 2013.

Digital Library

[9]

D. Chandler and A. Kapelner. Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior & Organization, 90:123––133, 2013.

[10]

L. B. Chilton, G. Little, D. Edge, D. S. Weld, and J. A. Landay. Cascade: Crowdsourcing taxonomy creation. In CHI. ACM, 2013.

Digital Library

[11]

D. DiPalantino and M. Vojnovic. Crowdsourcing and all-pay auctions. In 10th Conf. Electronic Commerce, 2009.

Digital Library

[12]

E. Dolstra, R. Vliegendhart, and J. Pouwelse. Crowdsourcing gui tests. In 6th International Conference on Software Testing, Verification and Validation, 2013.

Digital Library

[13]

L. B. Erickson, I. Petrick, and E. M. Trauth. Organizational uses of the crowd: Developing a framework for the study of crowdsourcing. In SIGMIS-CPR, 2012.

Digital Library

[14]

S. Faridani, B. Hartmann, and P. G. Ipeirotis. What’s the right price? pricing tasks for finishing on time. In AAAI Workshop on Human Computation, 2011.

[15]

L. Hetmank. Components and functions of crowdsourcing systems–a systematic literature review. In 11th Int’l Conf. Wirtchaftsinformatik, 2013.

[16]

J. J. Horton and L. B. Chilton. The labor economics of paid crowdsourcing. In Conference on Electronic Commerce, 2010.

Digital Library

[17]

P. G. Ipeirotis and P. K. Paritosh. Managing crowdsourced human computation. In WWW, 2011.

Digital Library

[18]

P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In ACM SIGKDD Workshop on Human Computation, pages 64–67, 2010.

Digital Library

[19]

A. Kittur, B. Smus, S. Khamkar, and R. E. Kraut. Crowdforge: Crowdsourcing complex work. In Proc. ACM Symposium on User Interface Software and Technology. ACM, 2011.

Digital Library

[20]

A. Kulkarni, M. Can, and B. Hartmann. Collaboratively crowdsourcing workflows with turkomatic. In CSCW, 2012.

Digital Library

[21]

T. D. LaToza, W. B. Towne, A. van der Hoek, and J. D. Herbsleb. Crowd development. In Proc. CHASE, 2013.

[22]

T. W. Malone and K. Crowston. The interdisciplinary study of coordination. ACM Comput Surv, 26(1), 1994.

Digital Library

[23]

R. Musson, J. Richards, D. Fisher, C. Bird, B. Bussone, and S. Ganguly. Leveraging the crowd: how 48,000 users helped improve lync performance. IEEE Softw., 30(4), 2013.

Digital Library

[24]

F. Pastore, L. Mariani, and F. G. Crowdoracles: Can the crowd solve the oracle problem? In 6th Int’l Conf. Software Testing, Verification and Validation, 2013.

Digital Library

[25]

A. C. Rouse. A preliminary taxonomy of crowdsourcing. In Australasian Conf. Information Systems, 2010.

[26]

E. Schenk and C. Guittard. Crowdsourcing: What can be outsourced to the crowd, and why?, 2009. HAL Working Papers.

[27]

E. Schenk and C. Guittard. Towards a characterization of crowdsourcing practices. J Innovation Economics, 1(7), 2011.

[28]

A. Schwarz, M. Mehta, N. Johnson, and W. Chin. Understanding frameworks and reviews: A commentary to assist us in moving our field forward by analyzing our past. Database Adv Inform Syst, 38(3), 2007.

Digital Library

[29]

K. Stol, P. Avgeriou, M. Babar, Y. Lucas, and B. Fitzgerald. Key factors for adopting inner source. ACM Trans Softw Engineer Methodol, Forthcoming, 2014.

Digital Library

[30]

K. Stol and B. Fitzgerald. Two’s company, three’s a crowd: A case study of crowdsourcing software development. In 36th International Conference on Software Engineering, 2014.

Digital Library

[31]

K. T. Stolee and S. Elbaum. Exploring the use of crowdsourcing to support empirical studies in software engineering. In Proc. ESEM, 2010.

Digital Library

[32]

Y. Zhao and Q. Zhu. Evaluation on crowdsourcing research: Current status and future direction. Inf Syst Front, April, 2012.

[33]

Y. Zhao and Q. Zhu. Exploring the motivation of participants in crowdsourcing contest. In ICIS, 2012.

Cited By

Alhagbani OAlyahya S(2024)A Coordination Approach to Support Crowdsourced Software-Design ProcessComputers10.3390/computers1312033113:12(331)Online publication date: 7-Dec-2024
https://doi.org/10.3390/computers13120331
Stol K(2024)Teaching Theorizing in Software Engineering ResearchHandbook on Teaching Empirical Software Engineering10.1007/978-3-031-71769-7_3(31-69)Online publication date: 25-Dec-2024
https://doi.org/10.1007/978-3-031-71769-7_3
Hasteer NSindhwani RBehl AVarshney ASharma A(2023)Exploring the inhibitors for competitive AI software development through cloud driven transformationAnnals of Operations Research10.1007/s10479-023-05619-5342:1(355-397)Online publication date: 18-Oct-2023
https://doi.org/10.1007/s10479-023-05619-5
Show More Cited By

Index Terms

Researching crowdsourcing software development: perspectives and concerns
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Social and professional topics
  1. Professional topics
    1. Computing and business
    2. Management of computing and information systems
      1. Software management

Recommendations

Two's company, three's a crowd: a case study of crowdsourcing software development
ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

Crowdsourcing is an emerging and promising approach which involves delegating a variety of tasks to an unknown workforce - the crowd. Crowdsourcing has been applied quite successfully in various contexts from basic tasks on Amazon Mechanical Turk to ...
Crowdsourcing Software Development

The earlier days of software development have witnessed the development through conventional development methods. With the aid of crowdsourcing, the tasks which are performed in a closed environment by limited persons can be distributed among the crowd. ...
Tasks Decomposition Approaches in Crowdsourcing Software Development
Human Interface and the Management of Information
Abstract
A main characteristic of crowdsourcing software development (CSD) is the complexity of tasks and skills required by workers to achieve successful software crowdsourcing. The tasks proposed to the crowd in CSD are checked to ensure they are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CSI-SE 2014: Proceedings of the 1st International Workshop on CrowdSourcing in Software Engineering

June 2014

18 pages

ISBN:9781450328579

DOI:10.1145/2593728

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 June 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

June 2, 2014

Hyderabad, India

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
805
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alhagbani OAlyahya S(2024)A Coordination Approach to Support Crowdsourced Software-Design ProcessComputers10.3390/computers1312033113:12(331)Online publication date: 7-Dec-2024
https://doi.org/10.3390/computers13120331
Stol K(2024)Teaching Theorizing in Software Engineering ResearchHandbook on Teaching Empirical Software Engineering10.1007/978-3-031-71769-7_3(31-69)Online publication date: 25-Dec-2024
https://doi.org/10.1007/978-3-031-71769-7_3
Hasteer NSindhwani RBehl AVarshney ASharma A(2023)Exploring the inhibitors for competitive AI software development through cloud driven transformationAnnals of Operations Research10.1007/s10479-023-05619-5342:1(355-397)Online publication date: 18-Oct-2023
https://doi.org/10.1007/s10479-023-05619-5
Pal K(2022)Markov Decision Theory-Based Crowdsourcing Software Process ModelResearch Anthology on Agile Software, Software Development, and Testing10.4018/978-1-6684-3702-5.ch010(194-211)Online publication date: 2022
https://doi.org/10.4018/978-1-6684-3702-5.ch010
Haqbani OAlyahya S(2022)Supporting Coordination among Participants in Crowdsourcing Software Design2022 IEEE/ACIS 20th International Conference on Software Engineering Research, Management and Applications (SERA)10.1109/SERA54885.2022.9806724(132-139)Online publication date: 25-May-2022
https://doi.org/10.1109/SERA54885.2022.9806724
Pal K(2020)Markov Decision Theory-Based Crowdsourcing Software Process ModelCrowdsourcing and Probabilistic Decision-Making in Software Engineering10.4018/978-1-5225-9659-2.ch001(1-22)Online publication date: 2020
https://doi.org/10.4018/978-1-5225-9659-2.ch001
Mejorado DSaremi RYang YRamirez-Marquez J(2020)Study on Patterns and Effect of Task Diversity in Software CrowdsourcingProceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3382494.3410689(1-10)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1145/3382494.3410689
Bibi SZozas IAmpatzoglou ASarigiannidis PKalampokis GStamelos I(2020)Crowdsourcing in Software Development: Empirical Support for Configuring ContestsIEEE Access10.1109/ACCESS.2020.29826198(58094-58117)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.2982619
Hasteer NBansal AMurthy B(2019)Barriers to crowdsource software developmentInternational Journal of Business Information Systems10.1504/ijbis.2019.10307032:2(127-144)Online publication date: 1-Jan-2019
https://dl.acm.org/doi/10.1504/ijbis.2019.103070
Illahi ILiu HUmer QZaidi S(2019)An Empirical Study on Competitive Crowdsource Software Development: Motivating and Inhibiting FactorsIEEE Access10.1109/ACCESS.2019.29156047(62042-62057)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2019.2915604
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten