skip to main content
10.1145/1985793.1985867acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Understanding broadcast based peer review on open source software projects

Published: 21 May 2011 Publication History

Abstract

Software peer review has proven to be a successful technique in open source software (OSS) development. In contrast to industry, where reviews are typically assigned to specific individuals, changes are broadcast to hundreds of potentially interested stakeholders. Despite concerns that reviews may be ignored, or that discussions will deadlock because too many uninformed stakeholders are involved, we find that this approach works well in practice. In this paper, we describe an empirical study to investigate the mechanisms and behaviours that developers use to find code changes they are competent to review. We also explore how stakeholders interact with one another during the review process. We manually examine hundreds of reviews across five high profile OSS projects. Our findings provide insights into the simple, community-wide techniques that developers use to effectively manage large quantities of reviews. The themes that emerge from our study are enriched and validated by interviewing long-serving core developers.

References

[1]
J. Asundi and R. Jayant. Patch Review Processes in Open Source Software Development Communities: A Comparative Case Study. HICSS, 0:166c, 2007.
[2]
C. Bird, A. Gourley, and P. Devanbu. Detecting Patch Submission and Acceptance in OSS Projects. In MSR '07, 2007.
[3]
C. Bird, A. Gourley, P. Devanbu, M. Gertz, and A. Swaminathan. Mining email social networks. pages 137--143, 2006.
[4]
S. Breu, R. Premraj, J. Sillito, and T. Zimmermann. Information needs in bug reports: improving cooperation between developers and users. In CSCW '10, pages 301--310, 2010.
[5]
M. Fagan. A history of software inspections. 2002.
[6]
K. Fogel. Producing Open Source Software. O'Reilly, 2005.
[7]
B. Glaser. Doing grounded theory: Issues and discussions. Sociology Press Mill Valley, CA, 1998.
[8]
C. Gutwin, R. Penner, and K. Schneider. Group awareness in distributed software development. In CSCW '04, pages 72--81, 2004.
[9]
P.-H. Kamp. A bike shed (any colour will do) on greener grass... FreeBSD mailing list archive http://www.webcitation.org/5ZZaDOxyW, 1999.
[10]
A. Mockus, R. T. Fielding, and J. Herbsleb. Two case studies of open source software development: Apache and Mozilla. ACM Transactions on Software Engineering and Methodology, 11(3):1--38, July 2002.
[11]
M. Nurolahzade, S. M. Nasehi, S. H. Khandkar, and S. Rawal. The role of patch review in software evolution: an analysis of the mozilla firefox. In IWPSE-Evol '09, pages 9--18, 2009.
[12]
A. Onwuegbuzie and N. Leech. Validity and Qualitative Research: An Oxymoron? Quality and quantity, 41(2):233--249, 2007.
[13]
C. N. Parkinson. Parkinson's Law: The Pursuit of Progress. John Murray, 1958.
[14]
E. S. Raymond. The Cathedral and the Bazaar. O'Reilly and Associates, 1999.
[15]
P. C. Rigby. Understanding Open Source Software Peer Review: Review Processes, Parameters and Statistical Models, and Underlying Behaviours and Mechanisms. thechiselgroup.org/rigby-dissertation.pdf, Dissertation, 2011.
[16]
P. C. Rigby, D. M. German, and M.-A. Storey. Open Source Software Peer Review Practices: A Case Study of the Apache Server. In ICSE '08, pages 541--550, 2008.
[17]
C. Sauer, D. R. Jeffery, L. Land, and P. Yetton. The Effectiveness of Software Development Technical Reviews: A Behaviorally Motivated Program of Research. IEEE Trans. Softw. Eng., 26(1):1--14, 2000.
[18]
S. Whittaker and C. Sidner. Email overload: exploring personal information management of email. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 276--283, 1996.
[19]
R. K. Yin. Case Study Research: Design and Methods, volume 5 of Applied Social Research Methods Series. Sage Publications Inc., 2 edition, 1994.

Cited By

View all
  • (2025)Code context-based reviewer recommendationFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-023-3256-919:1Online publication date: 1-Jan-2025
  • (2024)Factoring Expertise, Workload, and Turnover Into Code Review RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.336675350:4(884-899)Online publication date: 23-Feb-2024
  • (2024)An Empirical Study on Collaborative Uses of Communication Channels for Software Development and Management2024 34th International Conference on Collaborative Advances in Software and COmputiNg (CASCON)10.1109/CASCON62161.2024.10838181(1-10)Online publication date: 11-Nov-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSE '11: Proceedings of the 33rd International Conference on Software Engineering
May 2011
1258 pages
ISBN:9781450304450
DOI:10.1145/1985793
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 May 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. case studies
  2. grounded theory
  3. open source software
  4. peer review

Qualifiers

  • Research-article

Conference

ICSE11
Sponsor:
ICSE11: International Conference on Software Engineering
May 21 - 28, 2011
HI, Waikiki, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)41
  • Downloads (Last 6 weeks)5
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Code context-based reviewer recommendationFrontiers of Computer Science: Selected Publications from Chinese Universities10.1007/s11704-023-3256-919:1Online publication date: 1-Jan-2025
  • (2024)Factoring Expertise, Workload, and Turnover Into Code Review RecommendationIEEE Transactions on Software Engineering10.1109/TSE.2024.336675350:4(884-899)Online publication date: 23-Feb-2024
  • (2024)An Empirical Study on Collaborative Uses of Communication Channels for Software Development and Management2024 34th International Conference on Collaborative Advances in Software and COmputiNg (CASCON)10.1109/CASCON62161.2024.10838181(1-10)Online publication date: 11-Nov-2024
  • (2024)Quantifying and characterizing clones of self-admitted technical debt in build systemsEmpirical Software Engineering10.1007/s10664-024-10449-529:2Online publication date: 26-Feb-2024
  • (2024)The upper bound of information diffusion in code reviewEmpirical Software Engineering10.1007/s10664-024-10442-y30:1Online publication date: 17-Oct-2024
  • (2023)Quality Evaluation of Modern Code Reviews Through Intelligent Biometric Program ComprehensionIEEE Transactions on Software Engineering10.1109/TSE.2022.315854349:2(626-645)Online publication date: 1-Feb-2023
  • (2023)Integrating Visual Aids to Enhance the Code Reviewer Selection Process2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00037(293-305)Online publication date: 1-Oct-2023
  • (2023)Using Large-scale Heterogeneous Graph Representation Learning for Code Review Recommendations at Microsoft2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)10.1109/ICSE-SEIP58684.2023.00020(162-172)Online publication date: May-2023
  • (2023)What makes a code review useful to OpenDev developers? An empirical investigationEmpirical Software Engineering10.1007/s10664-023-10411-x29:1Online publication date: 22-Nov-2023
  • (2023)More than React: Investigating the Role of Emoji Reaction in GitHub Pull RequestsEmpirical Software Engineering10.1007/s10664-023-10336-528:5Online publication date: 18-Sep-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media